Substring Manipulation
Pattern Matching for Expressions and String Functions
Challenges
String-level expression logic is used to search, manipulate, and evaluate data according to specific patterns or rules. This functionality is often associated with text editors, SQL, and shell commands, but is rarely integrated into large-scale data processing operations.
In other words, string parsing, pattern matching, and other low-level data manipulations typically need to be performed in separate tools and I/O steps, increasing coding and processing overhead.
Some tools for ETL, data quality, and reporting also lack the kind of substring functionality required for special use cases, such as manipulating date values or replacing sensitive characters.
Solutions
The SortCL program in IRI CoSort and IRI Voracity supports Perl Compatible Regular Expression (PCRE) logic for pattern matching, as well as searching, replacing, and other manipulations at the string and substring level. SortCL also supports field padding and alignment, character checking, and field reordering.
These functions are also in connection with the Data determination, the Master data management and the improvement of Data quality useful.
More importantly, this complex data transformation can occur within the same job script and I/O pass as all of SortCL's other simultaneous functions, such as:
Data transformation (Sorting, Linking, Aggregation, Filtering, Remapping, etc.)
Data migration (of data types and file formats)
Data protection (Field-level encryption, de-identification, masking, etc.)
Reporting (Batch, Delta, Detail, and Summary BI)
Please use the form below to share your use cases for substring operations or other complex data manipulations.