Match and Join
Fast data synchronization and comparison reports
Challenges
Data reconciliation between large tables for querying, reporting, or virtualization purposes can be time-consuming. SQL join operations are typically inefficient for large-scale data integration (unification) operations. Custom programs intended to merge non-joined elements can also be slow or difficult to maintain.
Perhaps you also need a quick and easy way to compare two or more files across one or more fields. How can you do this and detect the changes (inserts, updates, deletes) between two files, especially when the data is in different file formats or in tables from different databases?
Solutions
The program Sort Control LanguageSortCLI'm IRI CoSort Data transformation package and the IRI Voracity Data management (ETL) platforms can simultaneously filter, sort, join, aggregate, and reformat multiple table and file sources at once.
SortCL uses simple and explicit 4GL- Text files to define data sources, targets, and transformations. Automatic scripting, cross-platform execution, modification, and management are available in the free Eclipse GUI., IRI Workbench, supported.
SortCL supports inner and outer join functionality to generate combined outputs and perform file comparisons based on specific conditions. With SortCL, you can:
- Input, merging, and output of one or more pre-sorted or unsorted tables and/or files
- Eliminate Inner Join Results from an Outer Join
- Eliminate and reformat null records
In the same simple job script and I/O pass with data linking, you can also:
- Cross-calculate and derive new values from the matching result
- Add field-level data masking functions for sensitive fields (PII)
- Define multiple targets for detail and summary reports individually
- Share selected information in various formats for data visualization tools.
The conclusion? The big data matching techniques supported in SortCL join operations allow you to merge matching data from different sources, exclude non-matches, and compare files and table data externally., capture modified data (CDC), to create business intelligence and reduce the effort for database queries and updates.
Did you know that IRI CoSort was the first data management product to merge flat files?
IRI introduced 1999 Joins in SortCL.