Structured Big Data Integrations–, cleaning and conversion tasks take too long or rely on complex transformations in:

• Elaborate ETL or sorting tools
• Shell or 3GL programs
• SQL Joins or Procedures
• Data quality or BI applications

IRI Voracity, the entire data management platform based on CoSort, Hadoop, and Eclipse.

The metadata definition language and the standard processing engine for data manipulation in Voracity and all IRI software is the Sort Control Language (SortCL), which was originally for IRI CoSort-developed by users. SortCL uses an open fourth-generation language (4GL) to define, combine, and optimize transformations and mappings within the same job script and I/O pass.

You can also use SortCL for data cleansing (data quality) and masking of PII data, as well as for reporting (and limited analytics), or data preparation for your BI platform.

Create SortCL Jobs automatic in the free Eclipse GUIIRI Workbench), and change them graphically and visually in the ETL palette. Plan and execute jobs inside or outside the user interface and share them within the team in repositories like EGit. Voracity big data users can also use SortCL transformation jobs seamlessly in Hadoop Control MapReduce 2, Spark, Spark Stream, Storm, and Tez. Users of legacy ETL tools can transform their jobs into Voracity via AnalytiX DS technology.

The standalone NextForm-IRI's SortCL spin-off is specifically designed for converting: data types, endianness, record layouts, file formats, and databases. It is Eclipse-based. IRI Workbench GUI also supports Voracity ETL and standalone SortCL jobs as well as NextForm data migration, FieldShieldData masking and RowGen-Test data operations.

Changed data capture (CDC) and other reports run in separate steps or products. With high volume, they are too slow.

Use the multi-source assistant to capture bulk/change data capture in Voracity and schedule its jobs in near real-time with a high-definition setting in the task scheduler for incremental data replication.

You can also create SortCL jobs that process data into custom detail and Overview Reports transform and reassign. The reports can be used for:

• Billing and Customer Data Integration
• Clickstream and Other Analytics (BI)
• Marking and editing insertions, updates, deletions, and no changes
• Slowly Changing Dimensions (SCD)
• Using Lookups and Fuzzy Logic Criteria for Trend Detection

SortCL also performs BIRT Reports in the memory within the Eclipse GUI, IRI Workbench, or Splunk with a Voracity add-on.

To comply with data protection regulations, you need Data security, but your development and outsourced processes need access to realistic data.

Most encryption methods protect all data in a database or much more than just the fields you need to protect. Conversely, most Data masking products too little.

Voracity or SortCL jobs that transform, migrate, and report on data (above) can simultaneously also protect sensitive data. Apply encryption to Field of vision, Canned and custom data masking, de-identification, hashing, randomization, substring, expression, pseudonymization, shifting, and custom functions. Determine re-identification risk with peer-approved statistical functions and scores. Also utilize key management, tokenization, and auditing.

Sensible Tools for Data Classification, Discovery, and Masking in the IRI Data Privacy Suite – IRI FieldShield, DarkShield, and IRI CellShield – support compliance with data privacy laws (CCPA, GDPR, HIPAA, PCI-DSS.).

Test data must look real, but using production data for testing is insecure, unavailable, or otherwise unacceptable. Other tools require subsetting and masking, and lack high data speed and versatility.

Voracity or SortCL jobs that transform, migrate, report, and back up data (above) can also be used to generate secure, intelligent test data without requiring real databases or files. The same 4GL behind SortCL data definitions and manipulations is behind the IRI RowGen Test data package (included in Voracity and also provides test data for ETL job previews!).

Easily switch between generating test data and processing and presenting real data. Use SortCL or RowGen in the IRI Workbench (Eclipse GUI) to create structurally and referentially correct test data for database tables as well as custom file and report formats with a mix of randomly selected/generated data.

Additional features include:

• All-Pairs and Connected (Valid) Pairs Test
• Canned and Custom Value Occurrences (Frequency) Distribution
• compound (or root) data type construction
• Automated parsing, generation, and DB population jobs that can run on any platform.

Master Data Management (MDM) hubs are too expensive and complex.

Use the MDM Assistant in IRI Voracity, in consolidation or registration style, to refine and reconcile disparate values and store (or propagate) the unified master values.

 

You can also simplify reference data and metadata, master data, and master data by: the common syntax of SortCL and its DDF (Data Definition File) format, the SortCL /TEMPLATE (composite value) function, and the future graphical IRI Workbench Hub for:

• Reference Data => Local and Remote Files, DB Tables, SET Files
• Reference metadata => DDF files
• Master Data => Cleanse, standardize, and create master data values
• Master Metadata => Master Data DDF and/or Fragment Files
• Transformation, protection, and test data generation rules

Very Large Database (VLDB) utility operations such as unload, load, migrate, query, and reorgs are too slow.

The IRI FACT (Fast Extract) tool for Oracle, DB2, Sybase, SQL Server, MySQL, Tibero, and Altibase unloads tables in parallel to flat files. FACT also creates SortCL DDF and loader control file metadata for optional ETL or offline reorganize (unload/sort/reload) operations. CoSort pre-sorts speed loads. This facilitates maintaining large fact tables in optimal query order, leading to faster lookup and response performance.

An old sorting tool is too slow, expensive, or inflexible.

IRI CoSort has Plug 'n' Play sorter presets, Parm conversion sort routines, and/or business incentives to leave your current sort.