Select / Filter
Only processing and outputting the data you need
Challenges
You need to filter mass data or perform custom data selection to reduce the data volume or carry out selective data extraction for a variety of reasons:
Data Archiving, Migration, Replication, and Federation (Data Mesh)
Reporting, Business Intelligence, and Analytics, e.g., customer segmentation
Database subsetting (and masking) for providing a realistic, yet secure test schema
Responding to Data Subject Access Requests (DSARs) for GDPR or similar data privacy law compliance
Initiatives for data deduplication, data reduction, data forensics, and data quality
SQL SELECT statements are great for „normal“ database tables, but can take a long time with large amounts of data (and depending on key relationships). They can also hinder bulk unload and data subsetting tasks.
Data selection techniques outside the RDB realm also vary between different data sources and silos, necessitating different data filtering commands. This can be particularly cumbersome in heterogeneous data integration contexts, requiring complex ETL or coding efforts.
Solutions
The program SortCL in IRI CoSort and IRI Voracity) offers multiple horizontal (record/row level) and vertical (field/column level) selection options for any set of structured data sources. CoSort/SortCL spin-off products – including IRI FieldShield (for data masking), IRI NextForm (for data migration and replication) and IRI RowGen (for test data generation) – all have the same row and column selection (filtering) functions.
Manage record size, count, and flow before, during, and after data transformation. Eliminate, reduce, or pass through records for fast, SQL-oriented data filtering during input and output. Bulk data reduction increases the efficiency of SortCL transformations, reports, loads, and other downstream processes.
Use the direct /QUERY syntax in SQL in the input phase of a SortCL job or native SortCL syntax for conditional logic (e.g., if-then-else expressions) to include, omit, and reformat records in different stages of your script for data transformation, reporting, and/or protection. Filter records by data class or column names, or by record-specific conditions. For example, you can specify which value ranges are valid and output only records within or outside those ranges.
Remove duplicate records. Validate character forms and perform other data integrity checks to reject or isolate erroneous records before they are loaded into a database.
Base these specifications on your business logic and store them in text scripts managed in Eclipse and shared in Git so they can be easily modified and reused.