Flat-File - Transform Huge Files Faster
Smart data architects keep it simple
When data is fixed or limited File formats save or use, you may already know that IRI CoSort is still the fastest data transformation tool for flat-file data within Unix, Linux, and Windows file systems. In 1992, IRI built a popular 4GL for CoSort called SortCL, to several Data transformations to perform and create reports, all in the same job script and I/O piped through large files.
SortCL continues to be available for CoSort package users and is also the default data processing engine for flat files (and others Data sourcesin full IRI Voracity ETL package.
Why are flat files so popular?
The performance and efficiency of flat files in large-scale data environments explain why more and more companies are turning to this approach. IRI FACT and CoSort have proven themselves to be stable and reliable tools across many industries when it comes to processing large volumes of data quickly and securely. The extraction and transformation process is optimized to save both time and resources.
With the continuous development of IRI products, including Voracity, IRI remains a leading provider of data management solutions, focusing on speed and flexibility. Flat files play a central role in this, as they support a huge range of data processing scenarios despite their simplicity.
Voracity is a comprehensive data management platform based on Eclipse and powered by CoSort or Hadoop, consolidating data discovery, integration, migration, governance, and analytics. Data movements, manipulations, and governance activities will be best supported in this visual design and deployment environment for flat files.
When you run SortCL-based transformations in the CoSort wizards—or complete ETL workflows in the Voracity wizards or diagrams—you can reuse the automatically generated metadata from the data definition file (.DDF) for your sources. These files contain field names and attributes that serve as symbolic references in the underlying SortCL scripts, which define the source-to-destination mappings and the custom report layouts.
SortCL job scripts or XML workflows (containing these scripts) that perform ETL can be run from the command line, via batch scripts (shell scripts), or from IRI Workbench... either ad hoc or on a scheduled basis. Schedules. Voracity users can also Preview see the mapping results before the entire workflow is executed.
Whether you define ETL jobs using CoSort (.scl) or Voracity (.flow), you benefit from task consolidation and proven techniques for optimizing file system I/O, memory, and multicore processing. Eliminate the overhead of high-volume transformation from DB and BI layers and eliminate the need for additional hardware, in-memory databases, and even Hadoop.
- Input of one or more large sequential sources
- Perform multiple transformations (filter, sort, join, etc.)
- Compare files to capture changes and BI
- Remap, reformat, and align fields
- Create segmented, customized reports
- Converting Data Types and File Formats
- Masking or Unmasking PII at the Field Level
- Generating test data in one or more file formats
- Simultaneous output to multiple destinations, including custom-formatted reports
This Chart illustrates the capabilities of SortCL in the CoSort product for the transformation, conversion, protection, and reporting of flat files – all in a single pass.
This diagram illustrates the capabilities of SortCL within the Voracity platform. The same single-pass capabilities are available here, but within a complete data management environment (curation).
In Voracity, you can merge flat file data with data in relational databases, dark (document) data, legacy and „modern“ big data platforms (Hadoop, NoSQL, cloud, and SaaS). Additionally, it allows you to mask sensitive file data at the field level while processing and merging it for BI and analytics tools.