Data Cleaning
Cleaning and Transforming Big Data in a Single Pass
Challenges
Data cleansing can be complex, time-consuming, and expensive. The functions you write in 3GL, shell scripts, or SQL procedures can be complex and difficult to maintain. They may not satisfy all your business rules or accomplish all the work.
Custom data scrubbing techniques often run in separate batch steps or in a special „script transform component“ that you need to connect to your tool's data flow and execute in smaller chunks. This inefficiency is exacerbated with high input data volumes.
Data quality tools, on the other hand, can also handle a large part of this work. Unfortunately, they are not particularly efficient with large amounts of data and can be difficult to configure or change. They can also be functionally overloaded, not combined with related data operations, and/or cost too much.
Solutions
The SortCL program in IRI CoSort or IRI Voracity can data in more than 150 table and file sources find and clean. SortCL uses a simple 4GL– and Eclipse-GUI, you can define not only your data cleaning functions but also many other transformation, migration, and masking functions, as well as your target tables and files (including reports) in minute detail.
The integrated data quality operations that you can perform with IRI software or combine with these other activities include:
DeduplicationCharacter check
Data homogenization
· Find (scan) and replace values
Horizontal and conditional verticalSelection
Definition andEvaluation data structure (format)
· Detection and marking ofData changes and logic problems
SortCL also supports the definition of custom data formats via template definitions. This allows for scanning and Verify of formats.
For enhanced field-level data cleansing (based on complex business rules), you can plug in your own functions or functions from data quality vendor libraries. The CoSort documentation refers to examples from Trillium and the Melissa Data address standardization library. Declare a cleansing function for any field either in the pre-action plan or in the target phase of a job (i.e., up to two DQ routines per field, per job).
The bottom line? With CoSort SortCL – and perhaps specialized data quality libraries you add – you can cleanse your data in the same I/O pass where you filter, transform, back up, report on, or distribute it.
If you find PII like SSNs in your datasets and need to search for it, SortCL will do that as well as the standalone IRI FieldShield Data masking tool. If you need high-quality test data, visit IRI RowGen. RowGen uses SortCL metadata to create intelligent test data that adheres to your business rules, so you can test with realistic, synthetic yet safe data: good, bad, and null data.