Transform Huge Files Faster


Smart Data Architects Keep it Simple

Flat, or sequential, files are a convenient, common format for data exchange. Because they carry little to no storage or access overhead, they are also one of the fastest source formats for data access, manipulation, loading, and reporting. That is one reason why the IRI FACT (Fast Extract) product exists - to produce flat files from big database tables.

 

If you store or use data in fixed or delimited file formats, you may already know that IRI CoSort continues been the fastest data transformation tool for flat files in Unix, Linux and Windows file systems. In 1992, IRI built a popular 4GL for CoSort called SortCL to perform multiple data transformations and generate reports, all in the same job script and I/O pass through large files.

 

SortCL is still available to CoSort package users, and is also the default data manipulation engine for flat files (and other data sources) in the full IRI Voracity ETL package.

Voracity is a total data management platform built on Eclipse -- and powered by CoSort or Hadoop -- that consolidates data discovery, integration, migration, governance and analytics. Flat-file data movement, manipulation, and management activities are best served in this visual design, and deployment environment.