Data remapping and reformatting
Change layouts and reports in data transformation
Challenges
Data transformation, data remapping, data reformatting, and reporting are often performed in slow, separate steps: e.g., sorting, then merging or aggregating, then handing off a flat file to a data quality or data processing tool. Then, the output of these processes is opened in a data mart, BI tool, etc. All of these I/O passes add up.
Complex languages like Perl or Python are sometimes used to transform data. Your data transformation technique might be hard to code or maintain over time and runs too slowly at volume.
Solutions
The data transformation program SortCL in the data manipulation product IRI CoSort or in the data management platform IRI Voracity (ETL) platform maps data using source columns (field names) as symbolic references for mapping data to the output. In combination with extensive, built-in data formatting features, this mapping pass also allows for reformatting, Replication, Reporting and even virtualization (Federationof data.
In other words: You can data into the same simple job script (and quick I/O pass) with ETLor other Data MigrationReformat and report on masking, sanitization, and obfuscation operations. Here Find all supported data sources (and targets) and this sample job in the IRI Workbench GUI, which is based on Eclipse:
Specifically, SortCL can re-map (i.e., reposition, rescale, align, trim, pad) and type-convert values when mapping fixed or variable position fields from input to output. Additional custom layout options include changing fixed-position layouts to variable (floating) and vice-versa.
Here are some things IRI software users can do simultaneously with the data they are remapping:
Transform: Filter, Sort, Join, Aggregate, Lookup, Cross-Calculate
Clean: clean, deduplicate, validate, enrich, standardize
Protect: Encrypt, redact, pseudonymize, or otherwise obscure, anonymize, or de-identify PII using masking rules
Parsing, removing, or rewriting header lines in output. Inserting special formatting characters and environment variables, including markup language commands for web-enabled reports.
Perform mathematical expressions (cross-calculations) between field data or on linked and/or aggregated values to derive and output new detail or summary report values. Create as many output targets and formats as you need within the same job script and I/O run.
Repurpose files from one type to another. For example, from a COBOL index file to CSV and vice versa. Append a „sequencer“ field to each sorted record, allowing you to cross-reference by index values across multiple tables or files. Populate targets directly via ODBC, pipes, or procedures, or feed them flat files for loading or further integration.