Fast and affordable ETL solutions

With Voracity, you go beyond the old ETL tools of the past

Overview
Discover
Design
Expand
Provide
Support
Flat file ETL
ELT
How data is integrated
In ETL operations (extract, transform, load), data is extracted from various sources, transformed separately and loaded into a data warehouse (DW) database and possibly other destinations.

Data integration with ETL is often performed with traditional or cloud ETL tools and sometimes with custom in-house programs. In almost all cases, these solutions continue to suffer from poor performance on large volumes of data, inability to adapt to the growing variety, velocity and veracity of data sources, complexity of implementation, and high license and support costs.

These weaknesses are all the more glaring when compared with the data management platform IRI Voracity which has the best price-performance ratio of all ETL software products in the data warehousing industry.

 
Speed (and price) are crucial

Take a look at how Voracity performs the Extract, Transform and store can be optimized and combined in a unique way:

Big data: fast acquisition

IRI supports a variety of powerful extraction methods for static files and streaming data. Pump data in memory via pipes or procedures, web services, IoT devices, Kafka and more.

VLDB unloading: ODBC selection or „Fast extraction“.
Very large database tables (VLDB) require a powerful unloading (extraction) method for:
Data warehouse ETL and ELT operation
    Classic (offline) reorgs
    Archiving and storage
    Migration and replication
    Data exchange

IRI Voracity - and its component products IRI CoSort (for data transformation and reporting), IRI NextForm (for data and database migration) and IRI FieldShield (for PII classification and masking) - read data directly from relational and NoSQL DBs via ODBC or native protocols. Or you can swap large RDB tables in parallel to flat files or store your data with IRI FACT (Fast Extract) through pipes ... directly into the Voracity ETL workflow.

Extraction performance in DB and DW environments is limited by high data volumes and inefficient approaches. Read in this blog post about a remedy for large Oracle tables.

Surgical
The source-side selection in IRI CoSort via SQL and filter commands in CoSort SortCL program and Change Data Capture (CDC) scripts can improve collection performance by reducing the amount of data. SortCL supports any number of input tables, files, pipes, and procedures simultaneously and can apply specific filter criteria to each source independently or together via a virtual input record (inrec).

You can also extract values from semi-structured and unstructured sources into flat files based on literal string searches and Java regular expressions (patterns). Multiple Data recognition assistants, which also classify and profile the data in such sources, are included in the IRI Workbench GUI for Voracity to help you find, structure and use dark data.

Bulk
Used for VLDB data acquisition IRI FACT (Fast Extract) native drivers and parallel query methods to turn VLBD tables into flat files when bulk unloads are required. FACT enforces no database overhead or configuration changes. FACT also bypasses the need to set up log sniffers and complex CDCs in the database.

FACT uses SQL SELECT syntax in simple configuration files to unload data from: Oracle, DB2 UDB, Sybase, MS SQL Server, MySQL, Altibase and Tibero.

During extraction, FACT formats the data (e.g. separators) and converts it (e.g. date types) and the LOB fields of the silos. FACT also writes CoSort SortCL metadata for data transformation, conversion/replication, masking and reporting as well as metadata for load controller control files for the source databases. This facilitates troubleshooting and ETL in the same I/O pass.

The IRI Workbench GUI for FACT, CoSort, etc. supports the automatic creation of tables and load files for additional target databases including Teradata - in Eclipse.

Optimize every transformation

Voracity accelerates every essential data transformation process. It also optimizes ETL operations by combining sorts, joins and aggregations in a single job script, partition and I/O pass.

With Voracity, a decades-old CoSort engine (or seamlessly interchangeable Hadoop engines) can transform massive amounts of data without the need for a DB, a hand-coded Hadoop program, additional RAM, or an expensive ELT appliance.

Combine several transformations
Voracity ETL software allows you to use multiple CPUs and cores (threads), run multiple tasks in the same I/O pass, and dynamically allocate memory and disk usage. You can transform large amounts of data from many different tables and sources together. Simple text file metadata repositories that you can manage and share as a catalog in IRI Workbench workspaces allow you to discover, define, expose and leverage your data sources.

Beat the high costs and learning curves of other techniques with the easy-to-understand, shared, and modified metadata that Voracity uses for data and work definitions. Like the underlying CoSort SortCL program the Data transformation to optimize and consolidate your data, see the Data transformation section on the IRI website.

If the quickest way to Provision of a data warehouse database (DB) in the file system, what is the fastest way to load it? There are many ways to load DBs, including:

    Single-row or multi-row inserts
    Create or insert (with attachment note) from another table of your choice
    Conventional and direct path loads

Many DBAs do not know the fastest method and instead use proprietary export/import tools that tax their databases and do not serve heterogeneous data warehouse architectures.

The CoSort-software in the IRI Data Manager Suite or the IRI Voracity (ETL) platform can create and populate any database table directly using surgical or bulk methods. Voracity users can also load data into NoSQL DBs via files or connectors.

The CoSort data transformation phase of Voracity jobs can quickly sort flat files into index order, which RDB loaders can quickly pump into tables while bypassing slower, DB-related internal sorts. Having tables in order and removing the bulk transformation overhead from the DB layer both improve query performance.

You can also use CoSort with or without Voracity to sort large amounts of data. transform and to report, so that your DB does not have to do this. It's about freeing up your DB to do what it does best: Storing and querying.

Surgical
Use the built-in ODBC functions to create, insert, truncate, update and add within CoSort SortCL data maintenance and mapping jobs while you define your targets. Or use direct DB connectionen and SQL functions, which are integrated into the Eclipse IRI Workbench GUI and support CoSort, Voracity, FieldShield (data masking), NextForm (data/DB migration) etc.

Bulk
Voracity ETL and other new job creation wizards in IRI Workbench provide automatic table creation and load control file generation for Altibase Loader, DB2 UDB load, Oracle SQL*Loader, SQL Server and Sybase bcp, and Teradata Fast and Multiload. This allows you to use the fastest method for bulk loading relational DBs.... presorted files. See this article about High Speed DB Loading.

Because speed is critical, Voracity is ideal as a standalone ETL tool for any important Integration paradigm in big data environments. And because price matters, Voracity also gives you the option to customize your current ETL tool. acceleraten or to replace.Beyond ETL, Voracity also supports a range of related integration activities, from federation and masking to data quality and MDM. Click here, for more information about these related capabilities and implementation considerations in Voracity.


Combining speed and Voracity

Big Data Manipulation Podcast

Voracity Sample Live Demo

The advantages of Voracity

Click on the boxes below to learn why Voracity is a better alternative for ETL operations and beyond:

IRI FACT (Fast Extract) uses native drivers to extract large tables in parallel to flat files or pipes. to unload.

IRI CoSort takes over the output of FACT from a file or an in-memory stream (pipe) and takes over the Data transformationn, which Load presorting and reporting in the same job script and I/O pass.

The platform IRI Voracity for total data management combines FACT, CoSort and Bulk DB Load Utilities in a visualized, scheduled ETL workflow that does not need to be compiled or partitioned. It can even seamlessly run CoSort jobs in MapReduce, Spark, Storm or Tez.

Compare all this to slower, richer SQL and 3GL programs and to more expensive, complex ETL and ELT platforms.... not to mention the delays in onboarding disjoint Apache projects.

ETL metadata and job definition are stored in the IRI Workbench GUI for Voracity, which is based on Eclipse™. Data discovery and new job wizards and a range of visual ETL job design options, faster creation of reusable repositories and scripts without the need for training in new syntax.

Nevertheless, Voracity metadata is the easiest to learn and use in the IT industry. It uses the same human-readable 4GL of CoSort - called SortCL - which utilizes familiar data layout syntax, SQL manipulation concepts, and shared metadata repositories. Many users still prefer to code and optimize these simple scripts directly.

The Voracity ETL environment not only supports extremely fast extraction/loading and one-passData transformations, that do not require partitioning:

    Change data entry
    Search/extract/structure of dark data
    Database and file profiling
    Data masking, encryption, etc.
    Data migration and replication
    Creation, conversion and release of metadata
    Detailed and summary reports
    Master data management (basic)
    Metadata management & lineage
    Offline reorgs
    Slowly changing dimensions
    Test data generation
    Database subsetting
    Data quality (cleansing, deduplication check, unification, standardization)

 
Voracity supports these activities on a very wide range of structured, legacy, big data, cloud, and SaaS platforms.Data sources.

Create all E, T and L jobs in the IRI Workbench GUI for Voracity, which is based on Eclipse™. Edit the jobs or workflow in palettes, GUI dialogs, syntax-enabled script editors (or any text editor you prefer) or in erwin Mapping Manager (AnalytiX DS Mapping Manager). You have the ergonomic flexibility to edit the data definitions and manipulations visually or by scripting; everything done in one feeds the other.

Test or execute jobs individually or together in the GUI flow or later in a (scheduled) batch operation. You have this flexibility in execution because the job scripts are portable. You can run any of the parts or the entire project on any platform on which the engine(s) are licensed. Call them from the command line or any application.

The IRI Workbench GUI for Voracity provides the visual metadata creation, conversion, and discovery tools you need to generate, deploy, and manage job scripts, data definition files (DDF), and XML workflows that are common across IRI software products.

In the same place you can also Powershell, COBOL, C/C++, Hive, Impala, Java, Perl, Python, R, SQL and other programs supported by Eclipse, sometimes integrating them as steps in your Voracity workflow.

You can also use the SortCL program from CoSort in Voracity to Transformations for other ETL tools such as Informatica and DataStage to optimize.

Voracity is far more than an ETL tool, but the price is below most of them. Even if you don't use it for ETL, since its SortCL program can be merged across many sources and data in Flat files Voracity utilizes the CoSort tradition as one of the most cost-effective methods of Data capture of changes (CDC) For serious ETL architects, however, the consolidation and multi-processing within Voracity transformation jobs - in the file system or (seamlessly in) Hadoop - makes Voracity the most cost-effective Big Data processing alternative to DB/ELT appliances, Ab Initio, SyncSort, Teradata, and in-memory DBs.

After all Voracity with its cost-effective opex subscription leveln and its relative simplicity, it is the most affordable data management platform that can be commissioned and maintained.

Related solutions

Other resources