Define Your Own Field-Level Manipulation Functions
Many ETL systems need to perform complex transformations. Unfortunately, their coding typically entails special script components, SQL procedures, or assemblies.
In addition to their learning and maintenance curves, these objects may not operate at the field level, or in the same I/O pass with other transformations, or leverage the APIs of best-of-breed software.
You may face data reengineering requirements that result from data profiling ... but then find that there is not enough time or expertise available. How will you write and maintain the custom scanning, cleansing, transformation, translation, or standardization routines that address your business requirements?
You may also need to apply special data protection(s) -- based on specific security circumstances -- that fall outside the scope of your application. Can you be sure the procedure you need to integrate will satisfy your business rules and comply with applicable data privacy laws?
The SortCL program in the IRI CoSort package and IRI Voracity platform integrates and transforms data in multiple sources. While it simultaneously manipulates, reports on, and secures data at the field level, the opportunity for custom transforms is also at hand.
Plug-in complex field-level transforms into SortCL jobs. Run them with your other data processing, presentation, and protection tasks.
Specify custom transforms on the same field twice in one job script, if necessary; i.e., in the pre-action layout or target phases. This allows two transforms per field within the same I/O pass.
Examples of custom transforms in SortCL include data quality routines written by Trillium and Melissa Data. Most of the data masking and encryption routines that gave rise to IRI FieldShield, and the national ID test data generation functions in IRI RowGen, are also custom transforms.
"Roll your own" transforms with a simple C interface. Or, use custom development services from IRI Professional Services. Our engineers can design, test, and deliver field-level transforms that meet your exact requirements.