User-defined transformations
Define Custom Data Manipulation Techniques
Challenges
Many data warehouse ETL and analytics environments need to perform custom data transformations to handle special business logic that goes beyond the scope of standard functions. Unfortunately, coding typically requires specialized scripting components, SQL procedures, or assemblies.
In addition to their learning and maintenance overhead, custom data manipulation techniques cannot operate at the field level, cannot be serialized as reusable rules, cannot be executed in the same I/O pass as other transformations, and cannot leverage the APIs of best-in-class software.
It's also possible that data profiling requires you to re-engineer the data, only to discover that there isn't enough time or expertise available. How are you supposed to write and maintain the custom scanning, cleaning, transformation, translation, or standardization routines that meet your business requirements?
You may also need to take special privacy measures that go beyond the scope of your application and are based on specific security conditions. Can you be sure that the procedure to be integrated complies with your business rules and adheres to applicable data protection laws?
Solutions
The SortCL program in the IRI CoSort package and in the IRI Voracity platform integrates and transforms data from several sources. While simultaneously manipulating, reporting, and backing up data at the field level, there is also the possibility for custom transformations.
The plug-in complex field layer is transformed into SortCL jobs. Run them with your other data processing, presentation, and protection tasks.
If needed, specify custom transformations twice for the same field within a job script, i.e., in the pre-action plan or the target phase. This allows for two transformations per field within the same I/O pass.
Examples of custom transformations in SortCL include data quality routines from Trillium and Melissa Data. Most of the data masking and encryption routines that go to IRI FieldShield and the national functions for generating ID test data in IRI RowGen have led to, are also customer-specific transformations.
Customizing data transformation in the IRI environment means you can create your own transformations using a simple C interface. This means you can write your own custom transformation tools in the form of functions that perform special lookup operations and/or manipulate, mask, blend, process, format, or synthesize data in any other way you deem appropriate.
Alternatively, you can also use the custom transformation services of IRI Professional Services utilize. Our engineers can design, test, deploy, and execute field-level transformations that precisely meet your requirements.