Text files

Convert and Process „Fixed“ and „Delimited“ Formats

Overview
CLF/ELF
CSV
Excel
JSON
LDIF
LS-RS-VS
MF-ISAM
MFVL
Text
Variable blocked
Vision
VSAM
XML

Challenges

Structured

Structured text files are fixed or variable, sequential (flat) files that can be as small as one record or contain billions of archived lines from database extracts, web logs, transaction feeds, mainframe datasets, and other applications.

You may have to:

  • Sort a large text file
  • Extract data or create a report from a text file
  • Convert between text and XML file formats
  • Convert a text file to another format
  • Encrypt or de-identify fields in a text file
  • Loading text data into a spreadsheet or database
  • Reformatting a text file from old or binary data

You may need to perform more than one of these functions simultaneously, with many extensive source and target files.

Unstructured

Unstructured text sources, including files and repositories in these formats:

  • ASN.1 TAP3
  • .DOC, .DOCX
  • .EML, .OST, .PST
  • .PDF, RTF
  • .PPT, .PPTX
  • .TXT, .XML
  • .XLS, .XLSX

can be converted, but the data within them cannot be easily extracted or used in the way structured data can.

Solutions

Use IRI NextForm, to convert structured text files into other formats (e.g., CSV, ODBC, XML, etc.) or from other formats into text. NextForm supports data type conversion at the field level and remapping of the record layout. The NextForm Edition „for Unstructured Data“ can extract data from unstructured text files for this page and analyze and structure the processes described in the entire IRI product stack.

NextForm file definitions also work in SortCL- Programs under IRI CoSort. Reuse metadata when upgrading to CoSort for fast data transformation and reporting.

NextForm includes a JSON file parser to automatically generate field layouts used in file conversion scripts. NextForm also supports field-level data type conversion and record layout remapping. NextForm job definitions also work in SortCL-compatible products like Voracity if you upgrade later.

There are initially known limitations in handling function calls and multiple array elements in unstructured JSON files. Provide the IRI with a sample for analysis if possible.

The program SortCL in CoSort can

  • Transform of data (e.g., sorting, merging, aggregating, cross-calculating, etc.) in text files
  • Text files Convert to other file formats and create text files from them
  • Report from text file sources

with a simple 4GL for layout and manipulation definitions or a powerful free Eclipse-based GUI.

Assign one or more input files in text format to and from other file formats. Create detail, Change Data Captures– and Slowly Changing Dimensions– Reports) from text files. Pass pre-sorted, filtered, and converted subsets to BI-Tools, Database Loaders or other applications.

Use IRI FieldShield, fields in structured text files with encryption, masking, etc. to protect.

Use IRI RowGen, when you need test data in text file formats. RowGen uses the same layout metadata as CoSort (SortCLand NextForm, so that you can easily switch between generating test data and transforming real data.

Product links