Frequently Asked Questions (FAQ)

Selected questions and answers about the IRI data management and data protection software that JET-Software supports in the German-speaking region.

BI & Analytics

IRI Voracity provides analytic functionality in five ways, with two more that were planned for 2019:

1) Embedded reporting and analysis – via CoSort SortCL programs that produce custom detail, summary, and trend reports in 2D formats, complete with cross-calculations and other built-in functions for data transformation, remapping, masking, and formatting. The reports can be descriptive or – through fuzzy logic and functions such as standard deviation plus Boost-powered statistical and BIRT-powered linear regression graphs – predictive.

2) Integration with BIRT in Eclipse – where, at report time, the BIRT charts you design are populated via ODA support with "IRI data sources" from Voracity/CoSort SortCL output. What's special: data preparation only runs when the report is requested, saving time and resources because the data is prepared outside the BI layer (with CoSort or Hadoop engines).

3) Data preparation ("franchising"), which accelerates time-to-visualization for 10 third-party BI/analytics platforms. Benchmarks show that SortCL alone is faster than BOBJ, Cognos, Microstrategy, QlikView, Splunk, Spotfire, R, and Tableau.

4) A direct Voracity add-on for Splunk, to pass data to Splunk for analysis.

5) Integration with a cloud dashboard from DWDigest for interactive BI, customizable and usable in any browser – including the internal browser of the IRI Workbench GUI.

6) Streaming analytics (pending) via JupiterOne – Voracity as a source of Kafka-fed data streams or as a target for live sentiment analysis.

7) Integration with the KNIME Analytics Platform in the IRI Workbench IDE, so that citizen data scientists can surpass themselves with machine learning, AI, neural networks, and unstructured data.

More on this at BI & Analytics.
It can be. CoSort offers a range of solutions for producing meaningful reports from huge volumes of data. You can use the SortCL program (4GL) as a standalone report generator or as a staging tool for processing and forwarding large data volumes. SortCL transforms and protects massive amounts of heterogeneous data from a wide variety of RDBMS sources, sequential or index files, and web and device logs (including ASN.1 TAP3 CDRs). It can also join, aggregate, calculate, and present this data in detail and summary reports – complete with special variables and tags for web pages. Via output to .CSV and .XML, SortCL directly populates spreadsheets like Excel, databases, and ETL and BI tools. See also BI & Analytics.
Essentially all tools that can import CSV, flat XML files, or RDBMS tables. The job of the IRI software is to centrally prepare ("wrangle") large, structured data. Users create SortCL programs in 4GL scripts – or via wizard in the free IRI Workbench GUI (Eclipse™) – that transform data from more than 125 sources (filter, sort, join, aggregate, mask/encrypt, pivot, pre-calculate, and so on). SortCL/Voracity directly populates: SAP Business Objects, IBM Cognos, DI-Diver, DWDigest, iDashboards, Microstrategy, Oracle DV/OBIEE, as well as newer platforms like Knime, Power BI, QlikView, R, Splunk, Spotfire, Tableau. Plus ODA driver support for BIRT, a Splunk add-on, the DWDigest cloud dashboard, and planned Kafka feeds to KNIME/JupiterOne. More at BI & Analytics.

CoSort & compatible IRI products – installation

Installing IRI Voracity and its component (or standalone) products – IRI CoSort, NextForm, RowGen, FieldShield, and DarkShield – requires a back-end installation of the CoSort package plus licensing of the executable SortCL program, which runs all these jobs on Windows, Linux, macOS, AIX, Solaris, or HP-UX systems. The graphical IRI Workbench (Eclipse-based) is an optional install for Windows, Linux, and macOS and ships with the Windows version of CoSort. Workbench also supports IRI Fast Extract (FACT); FACT requires a separate installation and license. The installation steps for all of these products – including the COBOL edition of NextForm – are documented in the installation guide. Separate installation instructions for the Voracity, CoSort, FieldShield, DarkShield, and RowGen APIs, as well as for FieldShield's proxy-based DDM option, are available on request (Contact us).
IRI can supply a CoSort build that works as either a fixed or a floating license. The standard version is a FIXED license, which does not expire, is tied to a specific host name, and requires host-name-specific registration (RegForm.txt form). Alternatively, we supply a different version with a special cosort.lic file for a FLOATING license, which must expire because it is not tied to a specific host name – typical (but not mandatory) for cloud installations. The decision is therefore not a question of on-premise vs. cloud, but of how frequently host names change. You can tell which version you have from the "build" tag: when you run $COSORT_HOME/bin/sortcl /v you will see R for fixed, or F for floating.
Practically all CoSort versions from the last 20+ years support RHEL on all 32- and 64-bit Intel-compatible systems (see Platforms & Prices). The current version, CoSort v10.5, is ported to RHEL 7 and 8 and also runs on 9 and future RHEL versions. Ports for z/Linux are available but, due to lower demand, may not be as current. For older, no-longer-supported versions such as 9.x, IRI uses the compatibility libraries that Red Hat ships with newer RHEL versions.
No. You only need a database login with sufficient rights to read or write the data and metadata for the desired operation. To read data from a table and write it to a file, read rights are enough. To create a new table and insert data, more rights are needed. To read data and then change the rows directly via UPDATE requires yet another set of permissions.
The IRI Workbench is the Eclipse interface (graphical job-design client) and runs on Windows, Linux, and macOS. The back-end SortCL program in the CoSort package / Voracity platform – the engine for data manipulation, mapping, preparation, masking, and mining – runs on all of these systems plus all Unix variants (AIX, Solaris, HP-UX, z/i/pSeries Linux). Because SortCL is the back end and Workbench the front end of several IRI products, this answer applies not only to CoSort (sorting) and Voracity (ETL, etc.), but also to FieldShield, RowGen, NextForm, and DarkShield.

The absolute minimum for pure CoSort/SortCL CLI operations is 40 MB RAM, but at least 512 MB–2 GB per CoSort user is recommended. Sort jobs that fit entirely in memory are usually faster – modern CoSort hosts are often configured with 64 GB–2 TB RAM to sort without the I/O overhead of work files. The Workbench needs at least 4 GB RAM and 10 GB free storage (after installing VMs, DBs, etc.); a JRE is included. For many database connections and parsing metadata, 6 GB and more is better; for schemas with hundreds of tables, up to 64 GB RAM can make sense.

We recommend running the licensed back end (SortCL) on the same system as your database source/target where possible, or in close network proximity – especially where network bottlenecks are known. With data masking (FieldShield, DarkShield) and test data generation (RowGen), the bottleneck is usually the network and I/O: masking, subsetting, or synthesizing takes about as long as simply reading/writing your data. This is another reason colocation on the same system, or an optimal I/O subsystem (Fibre Channel, SSD, multi-core), is advisable. As a rule of thumb: more RAM = better performance.
For an EC2 Linux instance that only runs the job scripts, a t2.small or t2.medium should suffice. For a Windows instance that handles both job design and runtime, a t2.large or even t2.xlarge is better. t3 instances of the same configuration are sometimes cheaper but often require more expensive storage.
Yes. This is no different from installing CoSort on a Unix server for multiple users. Installing Cygwin with SSH access for users would also work well. The crux of this approach is access to the input/output data streams – these would have to be network resources, which can slow runtimes considerably.
The command-line tools of the Voracity platform / CoSort package (e.g., sortcl-compatible and *2ddf programs) as well as the DarkShield web service run without issue in Docker or other containers. The configuration requirements are the same as with traditional use. The container needs a consistent host name for classic IRI licensing to work – otherwise other arrangements can be made. The required files, in particular a fixed or floating license file, must be present in $COSORT_HOME/etc and the environment variable must be exported. Since no network interface is available to run jobs, a mechanism to trigger the command-line call must be configured; alternatively, an SSH daemon (sshd) can run in the container. The IRI Workbench and certain calling programs (glue code) may need to run outside the container.
You can unpack the Workbench from the tarball with the following command:
tar -xzvf com.iri.pelican.workbench-linux.gtk.x86_64.tar.gz -C ${destination_directory}
Replace "${destination_directory}" with your desired target directory. Then start the Workbench by running the iri-workbench file in the target directory. Where the tarball is unpacked is up to the user or sysadmin policy. We recommend unpacking the package per user to $HOME/iri/workbench2/ A desktop or menu shortcut can be created for the executable.

CoSort – installation issues

The CoSort package with the executable sortcl must be installed on the system running the batch script. Because the batch script does not contain the full path to the sortclfile, this path must be in the PATH environment variable. Example for CoSort 10.5 on Windows: make sure the folder C:\IRI\cosort105 exists, the variable COSORT_HOME points to it, and the variable PATH contains the value %COSORT_HOME%\bin .
On Windows 10/11, make sure you have edit rights (not just write rights) for the file C:\IRI\CoSort105\etc\cosort.lic . Without them, the setup program cannot update the 3-part license key with the values you entered via the command line or the IRI Workbench setup ("Update License" option). You may have to enter the values by hand and replace the default values 0.0.0. On Windows 2003, first check whether the license keys you entered belong to the current (correct) private key in your RegForm.txt. If so, the registry was not updated correctly – open the installation directory and run global_config.reg and license.reg by double-clicking.
CoSort serial numbers are assigned to each new version build that you install for a specific operating system at a supported site, and may only be used on multiple host names that meet both criteria. So you can use the same install archive (with that S/N) on additional machines with the same version at the same site. These licenses are distinguished by the host name and the private-key value in each new RegForm.txt you submit for license keys.
Check the LOG file entry in $COSORT_HOME/etc/cosortrc. For v8.2.1, make sure the path does not contain a %s . You (and all CoSort users) need read and write rights for the LOG file (default: $COSORT_HOME/etc/cosort.log) and the overflow directories (WORK_AREA).

Data Governance – data/metadata forensics & RBAC

Yes, in several ways: access rights to job metadata, data sources (and targets), decryption keys, and via granular data-class function/rule assignments. See the other questions in this section; for help setting up your controls, contact us.
Since there is no dedicated "Voracity", "FieldShield", or "CoSort" server, there is currently no central place for user configuration. Users are identified by their login on the PC (running the IRI Workbench) or on the remote server (running the job/SortCL). In both cases, the operating system controls the user. When reading/writing files, the OS decides based on the user under which the job scripts run. For databases, credentials are stored in the JDBC/ODBC connection strings. Workbench artifacts in the workspace can additionally be protected when versioned via Git. The latest platform (OGS in CoSort 11) supports even more granular governance via RBAC for individual jobs, functions, and data classes; a future web UI will enable IAM and logging policies (integrable with Active Directory or LDAP).
Yes. With Voracity, CoSort, FACT, NextForm, and FieldShield and RowGen, data and metadata can be separated by role: via DBA-defined source/target access (in DSN files or the Workbench data-connection registry, protected by workspace/OS rights), via Active Directory (or Entra ID) or LDAP, and via repository-controlled access (e.g., Git) to files, workspaces, projects, metadata, executables, and logs, or to FieldShield/DarkShield keys (e.g., Azure Key Vault). The IRI Operational Governance System (OGS), introduced in 2026, initially focuses on the SortCL back-end engine and uses a central policy file to define users, groups, and roles for job-specific attributes. A browser-based front end with SSO follows in 2027.
Via access controls of the client machine or of Active Directory/LDAP and via file-system rights. In addition, the erwin (AnalytiX DS) governance platform or any Eclipse-compatible version-control system such as Git can lock specific projects, jobs, and metadata on a role basis.
Yes. Multiple roles/rights for FieldShield or Voracity metadata (DDF, job scripts, flows) can be assigned by having administrators bind policy-controlled Active Directory/LDAP objects to these assets. Further options are the erwin data governance platform (Premium) or Git. The IRI Test Data Hub in the ValueLabs TDM portal also supports administrator and tester roles. Rights are set very granularly at the individual or group level (DB login, execution, data access, audit-log query/report).
Yes. With Voracity (profiling, ETL, DQ, MDM, BI), CoSort (transformation), FieldShield (PII discovery/masking), NextForm (migration/replication), and RowGen (TDM, subsetting), access to sources and targets can be controlled down to the column level via DBA or file rights (DSN files, Workbench registry), as well as via field-level reveal permissions in secured job scripts and decryption keys. With DarkShield, access to relational/NoSQL databases is restricted via connection-specific configurations (JDBC drivers, API keys); file sources are subject to local OS or cloud API-key rights.
Both: access to data, metadata, and job scripts – plus execution rights – are linked to Active Directory objects, DBA logins, and/or file-system controls that are enforced on a policy basis for authentication. Calling applications can add further authentication layers.

Flat Files

Besides the many data warehouse architects who use CoSort's SortCL tool and find its flat-file approach faster than SQL procedures and ETL tool steps, experts also emphasize the efficiency of flat files when staging large data volumes: "The first system for which the data warehouse is responsible is the data staging area, where production data from many sources is brought in, cleaned, unified, combined, and finally delivered to the presentation systems … The two dominant data structures in the staging area are the flat file and the entity/relationship schema." – Ralph Kimball, "The Foundations for Modern Data Warehousing", Intelligent Enterprise Magazine.
By using flat files, the SortCL program – the main tool of IRI CoSort and the default engine of the IRI Voracity platform – avoids the usual overhead of DB connectivity and transformations. SortCL even flattens ODBC-connected tables and semi-structured sources such as MongoDB and JSON files. This lets the engine move sequentially and consistently through the data rows while using asynchronous I/O, advanced memory management, and multiple operating-system threads (without Java programs). The flat structure makes it possible to combine data transformation, reporting, protection, and prototyping in the same job and I/O pass: you can filter, cleanse, standardize, transform, mask, and report on huge data volumes simultaneously. So it's "one product, one place, one pass" instead of the complexity, cost, and time of competing approaches (multiple products, places, passes, and prices).
Through the SortCL program, users of IRI CoSort or IRI Voracity leverage the resources of their existing file systems – without the overhead and limitations of databases and SQL procedures, and without the cost of megavendor ETL tools, ELT appliances, in-memory DBs, or complex Apache projects. With SortCL, many tasks can run and combine simultaneously against sources of any size: data transformation (select, sort, merge, join, aggregation, re-map, pivot, cross-calculation), data cleansing, data governance (masking/managing master data, metadata, data quality), data migration, data replication, data federation, data masking (de-ID, encryption, hashing, pseudonymization, randomization, redaction, tokenization), data presentation (2D BI reports), data franchising (CSV/XML/ODBC for BI tools), data staging, and prototyping (safe test data). For processing, presenting, protecting, and prototyping big data, SortCL and flat files remain the fastest, most cost-efficient way. If your data resides in HDFS, many of these functions also run in MapReduce 2, Spark, Spark Stream, Storm, or Tez via Voracity's Hadoop gateway ("VGrid") – without reprogramming.

Job monitoring, recovery & logging

If you run Voracity jobs on Windows, Linux, or Unix with the standard CoSort/SortCL program, this executable can send event or status messages to the (CLI or GUI) console – at various verbosity levels. If you run Voracity jobs in Hadoop (MR2, Spark, Storm, Tez), the HDFS job inventory shows the status.
In Hadoop, we believe this happens automatically. Standard CoSort/SortCL executions only allow pause/resume when there is insufficient space for the overflow (temp) files.
Several logs are produced: SortCL app statistics, error logs, and a self-appending runtime performance file. Optionally, an XML audit log file is produced per run, showing the script content and environment details.

Master Data

The new data unification wizard in IRI Voracity follows the consolidation style of MDM and lets you compare, match, and group new master values via fuzzy matching. Alternatively, move the master data manually to a central location/metadata repository where it is easier to clean and manage. Define files and fields in SortCL-readable format and search for name/attribute discrepancies. Create SortCL jobs to integrate and standardize these files, and use field-level transformation functions to cleanse the data according to your business rules. You can then repopulate the master data tables as needed.
CoSort's SortCL tool can apply the necessary security filters to enforce your need-to-know rules and to encrypt and anonymize certain vulnerable file data before sharing. Mask your master data at the field level when you need to test it, outsource it, or make it available to different departments – while keeping less sensitive data accessible.
All CoSort (SortCL) scripts (whether for transformation, reporting, protection, or prototyping) are parsed and stored with runtime information in audit logs that you can secure and query with your preferred XML application or with SortCL itself.
The SortCL program in IRI CoSort can flatten certain hierarchical data – including mainframe index files – and prepare it for DB loads and BI tools. SortCL can join product data with customer transactions (and other files via common keys) and output custom reports, including aggregation and cross-calculation. SortCL can also perform field-level lookups and populate reformatted files.

User-defined transformations

At the field level, you cannot integrate custom transformations that need file-level information (e.g., statistics). However, SortCL already supports aggregation at the file and record level, plus cross-calculation for detail and summary reports.
SortCL passes the field data to the routine, which returns it in transformed state – formatted according to the remaining attributes of the output field description.
Write or license a library and specify it at the beginning of a SortCL job script (which can also be used in a thread-safe API call). In the /INREC or /OUTFILE phase of the script, define the library as a field attribute.

Support

As the German-speaking IRI partner, JET-Software supports you directly – send a detailed e-mail to info@jet-software.com, use the Contact form or call us at +49 6073 711 403.
A list of common technical questions about installation and configuration is available to customers with a maintenance contract in the protected support area. Beyond that, you can get many technical and how-to answers at any time via JET software.

Synthetic test data (RowGen)

Yes – all of that and more. Notably, RowGen can even produce all these targets simultaneously from the same source of synthetically generated (or masked) data, including detail and summary reports each with their own formatting.
Prices for RowGen v3 start well below $10,000 for perpetual use and rise proportionally with additional CPU cores you want to use. Maintenance is free in the first year and 15–20% of the license fee thereafter, depending on the upgrade scope. (For a quote in the DACH region, contact JET software.)
It reads common data model and file layout metadata to shape the test data rows into precise flat-file, report, and database table structures while preserving referential integrity. RowGen is a complete test data solution: generate large volumes in parallel simultaneously, apply your own rules/value ranges, use DB data models and metadata, preserve realism and relationships, transform/segment/report, format in custom file layouts, and audit jobs for compliance checks.

DW ETL operations

Voracity is a modern all-in-one platform for rapidly managing and using enterprise data. It is also a standalone ETL and data lifecycle management product that prepares, protects, and provisions many forms of big data in production plus test data for DevOps. Voracity saves costs on software, hardware, and consulting and extends your enterprise information management (EIM) – all from one Eclipse interface. More about Voracity.
All the functions listed below are supported by the IRI Voracity platform and the included IRI Data Manager and IRI Data Protector suites (except FACT, which is supported but as a premium option). See the Components. "GUI" refers to the IRI Workbench – a free Eclipse-based development environment (IDE) for designing and managing the jobs that integrate and transform data via the SortCL program in Voracity, IRI CoSort, and all other IRI software. "DTP" is the Data Tools Plugin (Data Source Explorer) in the Workbench, "DDF" are the Data Definition Files (metadata of the source/target data layouts).
Understood – and that is exactly what we have been doing for years by accelerating jobs in existing ETL tools (especially Informatica and DataStage transformations). Use IRI's scriptable and batchable transform engine alongside these platforms to accelerate ETL, BI/analytics tools, and DB operations. Supported tools include: ETL (ETI Solution, IBM DataStage, Informatica PowerCenter, Microsoft SSIS, Oracle Data Integrator, Pentaho, Talend), BI (BIRT, BOBJ, Cognos, Excel, MicroStrategy, QlikView, OBIEE), analytics (JupiterOne, R, SAS, SpotFire, Splunk, Tableau), and databases (DB2, Greenplum, MySQL, Oracle, SQL Server, Sybase, Teradata). Call SortCL jobs (in the CoSort package or in Voracity) from your tool's command line to prepare big data faster and populate the tables/file formats your tool reads directly. Use the same high-performance engines as Voracity: FACT (extraction), CoSort/Hadoop (transformation), NextForm (migration/replication), FieldShield (masking), and RowGen (test data).
Yes. Voracity is integrated via API with the AnalytiX DS metadata hub technology, so you can migrate from legacy ETL products largely automatically and cost-effectively. Ask about CATfx templates for Voracity from your current ETL tool, as well as LiteSpeed conversion services for more complex mappings. Whether you are switching platforms or getting started with data integration – Voracity shortens time-to-deployment and information delivery. (ETL replatforming)
Some things Voracity offers over legacy and open-source ETL (let alone ELT): built-in data profiling tools for flat files, databases, and dark data; high performance and scalability with or without Hadoop; a minimal learning curve thanks to simple, open text metadata; running jobs outside the GUI (command line, batch, via system/API call); a familiar open-source GUI (Eclipse); advanced aggregation (lead/lag, ranking, running, expressions); multiply nested conditions and derived fields with PCRE, fuzzy matching, and C functions; composite data value definition for format masking and test data; built-in migration, replication, masking (12 protection functions), DB subsetting, and referentially correct test data; CDC, slowly changing dimensions, plus detail, summary, trend (predictive), and clickstream reporting; seamless metadata integration with FACT, Hadoop, AnalytiX DS, and MIMB; excellent price-performance and fast US developer support. Put another way – with Voracity there is no need for: separate transformation stages (everything in one job/I/O pass), partitioning (automatic multithreading), manual metadata definition (automatic detection + CATfx/MIMB), separate BI tools (reports in the same pass), separate masking tools (includes FieldShield), separate test data tools (includes RowGen), long-term consulting (familiar Eclipse GUI), separate MDM/DQ hubs (MDM wizard), and a new team/version-control paradigm (works with Git, CVS, SVN, AnalytiX DS). Moreover: backed by IRI, a stable company with more than 40 international offices; available at affordable prices for perpetual or subscription use.
CoSort is a data transformation and thus ETL engine, but not a classic ETL package. The IRI Voracity platform, which uses CoSort for transformations, is. Voracity can use CoSort and Hadoop for transformations plus FACT for extraction and pre-sorted bulk loads. With Voracity/CoSort you benefit from powerful, I/O-consolidated, multithreaded transformations (filtering, sorting, joining, aggregating, converting). SortCL also handles slowly changing dimensions, fuzzy logic, lookup tables, pivoting, running/ranking/windowed aggregates, and bulk CDC. In Voracity, much of this is wrapped in graphical wizards, workflow, and transform-mapping diagrams, so you don't have to script these jobs. They can be checked via preview and run from the GUI or command line; metadata is versionable, shareable, and securable (e.g., EGit). (ETL, Voracity)
Yes, both. Since as far back as 1999, experts have regarded CoSort as an ETL engine for high-performance staging and integration. CoSort – specifically its SortCL program – takes on the heavy lifting of selection, transformation, reporting, and pre-load sorting against sequential files in an ODS, DW staging area, or on extracted tables. SortCL is a push-down optimization for Informatica PowerCenter and the sequential file stage of IBM DataStage (faster, combined single-pass operations for sort, join, aggregation). Besides proven integrations and plug-'n'-play sort replacements for DataStage and Informatica, CoSort also connects with Kalido, ETI, SAS, and TeraStream. SortCL programs can be called as executables from any tool – e.g., Ab Initio, Pentaho, JasperETL, Pervasive DataRush, and others.
Yes – either via the DataStage sequential file stage or a before-job subroutine. With or without the larger Voracity platform, use CoSort as an external transformation hub and combine large sort, join, aggregation, reformatting, protection, and cleansing functions in one job and I/O pass in the file system. Voracity adds the visual ETL design environment and Hadoop execution options. With AnalytiX DS technology, most DataStage jobs can be converted to Voracity automatically. (DataStage, ETL replatforming)
Yes. With or without the larger Voracity platform, use CoSort as an external transformation hub and call these jobs as command-line operations in existing Informatica flows. Voracity adds the visual ETL design environment and Hadoop options. With AnalytiX DS technology, most PowerCenter jobs can be converted to Voracity automatically. (Informatica, ETL replatforming)
Two compatible third-party technologies help: 1) The AnalytiX DS Mapping Manager or LiteSpeed conversion migrates legacy ETL jobs into equivalent Voracity jobs (powered by SortCL or interchangeable Hadoop engines), including source/target data layouts and most transform mappings; manual translation/testing fills gaps in complex mappings. 2) To convert only the data layouts from another tool into SortCL layouts, use the Meta Integration Model Bridge (MIMB), which converts Informatica .xml, DataStage .DSX, and other metadata into SortCL DDF.
To some extent, yes. Voracity's core functions (transformation, mapping, masking, embedded report formatting) all reside in the SortCL program, which you can already script and run with your CoSort license. And the GUI for CoSort (IRI Workbench, on Eclipse) is the same one used by Voracity and all sub-products of the IRI Data Manager (CoSort, FACT, NextForm) and IRI Data Protector (FieldShield, CellShield EE, RowGen) suites – with access to many Voracity features (data discovery, ETL flow diagrams, MDM wizards, FieldShield masking, RowGen test data). As a pure CoSort licensee, however, you are only entitled to IRI support for the functions documented in the CoSort manual and the GUI operations in the "CoSort" and "IRI" menus. Voracity subscribers receive support for all functions visible in the Workbench (profiling, masking, test data, ETL, MDM, CDC, SCD, metadata management, BIRT integration) plus compatible add-on options (AnalytiX DS Mapping Manager, FACT, SortCL in Hadoop, predictive reports with BIRT, Paques). For a full feature comparison, see the product feature matrix.
CoSort's SortCL program has built-in filter and selection logic to reduce, segment, and cleanse data volumes during or after processing. For more advanced data cleansing, integrate your own function libraries to perform field-level custom transformations before or after sort/join/merge/report. SortCL provides a sample template: a Melissa Data object for address standardization that cleanses the address field on output. (Data quality, Clean up)
CoSort SortCL, IRI Voracity, and other IRI programs (FACT, NextForm, FieldShield, RowGen) all run from the command line and can therefore be scheduled into batch streams via cron, Stonebranch Universal Controller (UAC), Cisco TES, CA Autosys, ASCI ActiveBatch, and the like. PoCs have been done with Oracle DBMS_Scheduler, UAC, and Full360 metaController. The IRI Workbench (the free Eclipse GUI) also has a built-in task launch scheduler.

IRI software – prices & licensing

The purchase price (CapEx, perpetual use) of IRI de-identification software like FieldShield is generally based only on the number of host names running static masking jobs (where the engine/SortCL program is installed). It does not vary by cores, Workbench users, data sources, rows, functions, and so on. The same applies to Voracity licenses (incl. FieldShield, CellShield EE, DarkShield, RowGen, CoSort). DarkShield licenses for database sources (RDB/NoSQL) with C/BLOBs are priced like FieldShield; DarkShield masking of files/documents without databases is priced volumetrically. FieldShield and DarkShield are available individually (or together at a discount) as a one-time license covering perpetual use, documentation, and the first year of support. There are discounts for multiple and runtime licenses; at volume, Voracity is cheaper. At least one engine license is required for masking; the base/first Voracity price tier includes five engine licenses. Recommendation: the number/distribution of licenses should consider performance, convenience, and budget – from a performance standpoint, ideally one engine per large DB source/target (avoids network/I/O contention); for convenience, often additional local dev/test licenses on Workbench PCs; budget-wise, additional production licenses procured at the same time are discounted 20–40%. Annual maintenance: 20% of the license fee (incl. upgrades); with Voracity subscriptions, support is included. Further notes: Voracity includes a CellShield EE license, the DarkShield wizards + API, and the same 5 SortCL engines; higher tiers (e.g., up to 15, 50, 100, 500 host names) lower the unit price. For dynamic masking via the FieldShield "SandKey" API (SDK) or DarkShield REST API, a more deeply tiered volume pricing structure applies (royalty tied to a primary static development license). For a DACH quote, contact JET software.
You buy additional host-name copies of a standalone product at the then-current terms (full price on the largest new system, additional ones needed at the same time at a discount). There is no refund for fewer licenses, but your annual maintenance base is reduced if you formally confirm decommissioning and removal from the host names. Since Voracity is licensed in tiers (host-name blocks), you only pay the difference for a higher tier if additional licenses push you into the next tier (e.g., from 5 to 6/7 or beyond 15/50/100/500). Conversely, decommissioning/non-renewal (after certification) enables a lower tier rate at the next renewal.
Prices per copy (incl. SortCL) are typically in the mid four- to five-figure USD range and depend on hardware (e.g., RAM on x86 PCs, model on Unix OEM servers) and the number of cores used. CoSort is also available by subscription within Voracity (then without regard to machine size). For perpetual use, there are discounts for follow-on licenses ordered at the same time, limited/expiring use, runtime integration, and GSA procurement. Final quotes are made under NDA. Quote for the DACH region: JET software.
The IRI Workbench (Eclipse IDE) is included for every Windows/Linux host name running a licensed IRI product; a macOS version is available free on request for licensees. IRI CoSort: perpetual use per host name based on installed RAM (x86 Windows/Linux) and number of licensed threads; on AIX/Solaris/HP-UX/zLinux by OEM model. IRI Voracity (incl. CoSort, NextForm, FieldShield, DarkShield, RowGen, Ripcurrent): usually subscription (1 or 5 years) based only on the number of SortCL-executing host names; Hadoop and other premium options cost extra; perpetual use on request. IRI FACT and IRI RowGen: per host name by licensed CPU cores. IRI NextForm: several editions/prices (Details). IRI FieldShield: only by executing host names (Linux and outside the USA more expensive). CellShield/DarkShield: by number of tables/documents to protect (DarkShield for DBs same as FieldShield). DMaaS: daily, hourly, or project pricing. For a DACH quote: JET software.
In the first year it is often included in the license fee; thereafter it is an annual option as a percentage of the license fee, depending on the desired support/upgrade level. 20% of the license base is common; that entitles you to product updates and limited license transfer rights. If support lapses, retroactive support is due and a major upgrade may be necessary (at a discount). 24/7 support is an additional premium.
The software does not expire. The standard business model allows unlimited use over time, with or without support. So you do not have to pay again at the end of a multi-year term – unless you explicitly requested a lease.

ISV/OEM integration

The CoSort package includes all tools, conversion utilities, and callable libraries plus complete PDF documentation for executables and APIs.
When a CoSort package is installed, default tuning parameters are automatically configured to use available RAM and CPU resources. Users can adjust the parameters (including assignment of overflow disks and audit logs) manually in a simple text file.
Usually via system-specific license keys. Call us to describe your situation (Contact us).
That depends on which components you deploy and the range of functions you need. For a simple sort, choosing the interface often takes as long as implementing it. Custom input/output or comparison routines, or your own field-level transformations (e.g., custom cleansing or statistics functions), take longer.
From $150 to $15,000 per copy or site when CoSort is embedded in your application and redistributed. It depends on the integrated part, the number/speed of distribution, plus the (average) end-user hardware and the price of your application. We aim for fair licensing within your business model.
It can be any of these (or any combination).
You can invisibly embed, re-license, or reference a complete or partial CoSort package. Partial options include: standalone interfaces (sorti, sortcl); third-party sort plug-ins (e.g., Unix sort, SAS); runtime libraries (cosort_r(), sortcl_routine()).
That depends on your current methodology, hardware, kernel and CoSort tuning, data volumes, and job specification. You can measure it in a free, confidential test.
If so, less than for production. Contact us to discuss your situation (Contact us).

Creating/converting large XML files

Yes, provided your XML files describe structured, sequential records. The output can be another flat XML file or any other file type supported by SortCL.
Yes, if the XML file contains structured, sequential records and the output process type is documented in SortCL.
Yes, if these process types are documented in SortCL. Valid output targets can also be custom-formatted and protected for reporting, handoffs, and outsourcing.
Yes, within the format limits mentioned above.
No – unless you run out of disk space.

Oracle acceleration

It helps to spool the data without qualifiers into a flat file or pipe it to CoSort. So leave ORDER BY, GROUP BY, DISTINCT, and JOIN to CoSort – it does all of these (simultaneously) faster in the file system and delivers formatted reports in the same I/O.
SortCL supports all SQL aggregate functions (sum, average, count, max, min), but is more efficient because it can sort on multiple keys and produce aggregate results for one or more output files in the same (offline) pass.
On an ia64 HP server rx5670 (four 1-GHz Itanium2 CPUs, 32 GB RAM), Oracle 9i (SQL*Plus) joined two 1-GB tables in 48 minutes. Unloading the same tables with IRI Fast Extract (FACT), piping into flat sort/join operations in SortCL, and then piping into SQL*Loader produced the same joined table in 18 minutes (about 1/3 the time of the online method).
Direct-path loads with pre-sorted data are the fastest way to create new tables, since this method bypasses the overhead of Oracle index sorting. For bulk loads, first sort the data with CoSort on the primary index key; the CREATE INDEX then skips the sort step. For regular insert loads, sort on the clustered index as the key.
Via database-specific APIs and parallel unload techniques that produce portable flat files. For details, brochure, white paper, webinar, or a test, see FACT.
CoSort (or Voracity) SortCL scripts do many of the same tasks considerably faster and with far less code. SortCL runs outside the database on flat files and uses the same relational logic: SELECT/WHERE, DISTINCT, ENCRYPT, ORDER BY, GROUP BY, JOIN. For example: SortCL uses conditional /INCLUDE and /OMIT statements for selection; DISTINCT corresponds to /NODUPLICATES, ORDER BY to a /KEY, GROUP BY to a /SUM, /AVERAGE, and so on.
Like FACT, the AnalytiX DS Mapping Manager or MITI's Meta Integration Model Bridge (MIMB) can automatically convert the file layout metadata from your relational or ETL tool into the CoSort/SortCL Data Definition File (DDF) format.
See BI & analytics > reporting functions.

CoSort – sorting applications & performance tuning

CoSort is a robust, commercial-grade software package for efficiently manipulating and managing large volumes of data. More specifically: a package for sorting, data transformation, migration, and reporting that addresses a very broad range of challenges in data integration, data masking, business intelligence, and adjacent disciplines. "CoSort" stands for co-routine sort and was first released commercially: CP/M 1978, DOS 1980, Unix 1985, Linux 1990, Windows 1995, IBM i/Z 2000. CoSort uses parallel processing, advanced memory management, I/O techniques, task consolidation, and superior algorithms to optimize data movement and manipulation in existing file systems – without a paradigm shift to DB engines, NoSQL, Hadoop, or appliances. Sometimes a little more RAM, but that usually suffices.
Very fast. Performance varies by source size/format, data and job orientation, hardware configuration, resources, concurrent activity, and tuning. The best benchmarks (e.g., 1 GB in 12 seconds, 50 GB in 2 minutes) run in memory on fast multi-CPU Unix servers.
As soon as you perceive a bottleneck – depending on your hardware, from about 500,000 to 50 million rows. CoSort routinely sorts in the terabyte range and scales linearly without Hadoop; input files in the tens to hundreds of gigabytes are common. Any number of input/output files and structured formats are supported simultaneously (line/record/variable sequential, blocked, CSV, I-SAM, LDIF, flat XML, Vision). CoSort is also the default engine of the IRI Voracity platform, which can run many CoSort transformation and masking jobs (in SortCL 4GL or graphically in the Workbench) seamlessly in Hadoop – so the question is rather at what volume you would run such jobs via a Hadoop engine instead.
Usually via a CoSort Resource Control (cosortrc) text file, which can be global, user- and/or job-specific (on Windows, registry settings set defaults that can be overridden by an rc file). You set ceilings/floors for CPU/core threads and memory, I/O buffers, and disk space for sort overflow. The most important factor for sort performance is usually the speed of the I/O channels. For small (in-memory) files, optimize reading/writing of the source/target files; for large files, also the throughput to the work-area files. The local work areas (temporary merge files) can often be optimized – multiple fast SSDs on separate controllers make overflow sorts almost as fast as in-memory. After that, the memory settings matter most; CoSort 10 has powerful self-tuning techniques. As the only memory setting, we recommend MEMORY_MAX AUTO in $COSORT_HOME/etc/cosort.rc. Example tuning file:
THREAD_MAX 6
MIN_THREAD 1
MEMORY_MAX AUTO
WORK_AREAS /mnt/ssd1/work
WORK_AREAS /mnt/ssd2/work
MONITOR_LEVEL 0
MINIMUM_YEAR 70
ON_EMPTY_INPUT PROCESS_WITH_ZEROS
OUTPUT_TERMINATOR INFILE
Set THREAD_MAX up to the number of cores your license allows. For further questions, contact us.
An external sort is too big for memory and uses temporary work files (LWF). One physical work file is created per thread/work-area combination. The memory needed while READING the input is approximately: THREAD_MAX × AIO_BUFFERS × BLOCKSIZE × number of WORK_AREAS. In the merge phase (WRITING), considerably more memory is needed, since each physical file can contain multiple logical work files. If the memory available via MEMORY_MAX is insufficient, the "insufficient merge memory" error (error 2) occurs. Remedy: increase MEMORY_MAX or decrease one of the other settings involved in the calculation.
CoSort uses multiple threads as soon as the sort volume is at least twice the BLOCKSIZE specified in the cosortrc (typically 1–2 MB with auto-tuning). CoSort does not distinguish between physical CPUs, cores, or hyper-threading – the operating system schedules the threads. CoSort just creates the sort threads up to the maximum specified in the tuning file (never above the license limit). Cores are usually the best indicator of peak performance before the point of diminishing returns (resource contention, Amdahl's law). Since every sortcl process is independent (no inter-process communication), only 1–2 max threads are often more efficient in concurrent multi-job environments. Memory tunes itself with MEMORY_MAX AUTO. For testing, request a temporary license key for the total number of physical cores via the normal registration process and then vary THREAD_MAX. Runtime results are logged and can be analyzed offline later.
No – which CPU cores the threads use is decided solely by the operating system. If your OS supports it, you can try system tuning to assign sortcl processes to specific chips/cores.
CoSort has no inter-process communication; each instance stands alone. So with MEMORY_MAX = 10% and two simultaneous jobs, you use 20% of system memory.
Yes. A higher BLOCKSIZE improves read/write performance if source and work files are on the same device – otherwise a lower BLOCKSIZE is better. Test values from 100K up to 16M. On large external sorts, too big a BLOCKSIZE can lead to insufficient merge memory.
AIO_BUFFERS is the number of buffers for reading/writing. CoSort uses overlapped I/O (while one buffer is processed, another is read or written). More buffers bring a slight speedup but need more merge memory. Use values above the default with caution, test first with the largest input that will occur, and check whether the gain is measurable.
CoSort users can display runtime information before, during, and after execution: via optional on-screen display levels, self-appending/replacing log files, application-specific statistics files, and a full audit trail for compliance and forensic requirements.
More than 120 – and counting. These include single- and multi-byte character sets, Unicode, C, COBOL, and mainframe numerics. If you are unsure what you have, JET software can help with the definition. CoSort also supports the (simultaneous) collation, conversion, and creation of more than two dozen file formats.
There is no specific mvs2scl option for this, but you can use grep when running the translation. Run on the command line:
mvs2scl job1.mvs | grep -v '$SORT' > job1.scl
Here, job1.mvs is the MVS script to translate and job1.scl is the SortCL script without the $SORTIN/$SORTOUT lines. The -v option outputs only lines that do not contain the expression $SORT.
SAS documents the CoSort option in the v7/8 systems for Unix (in the SAS and CoSort manuals). Using CoSort accelerates native SAS PROC performance off the mainframe significantly and affordably. In SAS, CoSort is tuned automatically or via a resource control file set up by the administrator (modifiable at any time at the global, user, or job level). For SAS 9 and later, please contact SAS, as they have not yet updated their "sort appendage" for CoSort.
They differ and are based on collaboration and feedback with partners and customers who own their data and job definition metadata.

IRI (the company)

The company is IRI, Inc. – IRI stands for Innovative Routines International. IRI was founded in New York in 1978 as Information Resources, Inc. and renamed itself during the relocation to Florida in 1995 (the name Information Resources was already taken there). CoSort is IRI's first product and made the company known. We are not affiliated with the other "IRI" – the Chicago market research firm Information Resources, Inc. (which, ironically, is itself an IRI CoSort licensee).
CoSort is IRI's best-known product for data management and manipulation. The central tool in the CoSort package is the Sort Control Language (SortCL) program – SortCL refers to both the executable and the 4GL syntax for job scripts (data definition + manipulation). SortCL has given rise to several IRI spin-off products that use the same definition syntax but cover a lower-cost, targeted functional subset: NextForm (migration), FieldShield (masking of sensitive data), and RowGen (test data). Voracity is IRI's "total data management" platform containing CoSort and all SortCL spin-off functions – plus data discovery/profiling, visual job design, Hadoop engine options, cloud/big data connectors, data quality and MDM wizards, and multiple analytics frameworks. The IRI Workbench is the free, common Eclipse GUI for all of these jobs.
Until recently, IRI's focus was on back-end systems that few people talk about. As a "silent partner" doing the heavy lifting for other people's products and operations (including CGI, CSC, Cincom, Epsilon, NTT Data, Sabre, Sungard), IRI was less visible than its software stack justifies. In fact, IRI is also a prominent enterprise software vendor known within many large companies worldwide (including American Airlines, Bank of America, Comcast, Disney, EDS, Fidelity) and by the consultancies that serve them (Accenture, Atos, Cognizant, DXC, HPE, IBM Global Services). IRI is not venture-funded and invests far more in R&D than in marketing – to the benefit of its customers. Gartner follows IRI software in the areas of data integration, data masking, legacy migration, test data, and business intelligence.
Yes and yes.
As the German-speaking IRI partner, JET-Software supports you before, during, and after your evaluation and directs you to the right resource. Since it is in our interest that your business and technical goals are met with the software, we work with you on successful solutions. Contact us.
JET-Software – as the German-speaking IRI partner – plus more than 40 international IRI support offices and numerous consultancies familiar with IRI software. Above all, you will find how easy it is to help yourself: all data- and job-related metadata is open, self-documenting, and easy to adapt, and the familiar Eclipse GUI automates script creation, integration, execution, and management.
Because they are lower. Being part of so many other applications, IRI did not need its competitors' marketing overhead; moreover, IRI is not a publicly traded company and serves no external investors or debt. We pass these savings on to customers.
In the German-speaking region, directly through JET-Software as the authorized IRI partner, which also services the software. Request a quote (Contact us) and, after a successful evaluation, specify the products to license. Prices depend on what you deploy and where (see the pricing FAQ).
Yes, 24/7 support is available. For worldwide 24/7 support, there is usually a surcharge on top of standard annual support.
Maintenance is free in the first year after licensing. For users under maintenance, minor releases are free on request or as needed for support. Major releases are optional but usually chargeable upgrades; the cost depends on your maintenance level.
Because you buy a perpetual-use license once. Support is optional, and you are not forced to upgrade (even though it makes sense eventually). If you use Voracity on a subscription basis, you can lock in license and support costs for five years via a discounted up-front payment.

IRI partnerships

Yes – but not in the onerous way of megavendor programs. We rely on customized, reciprocal relationships in these categories: expert consultants (specialized in industries and/or IT disciplines who recommend, refer, resell, and/or implement IRI software); industry alliances with hardware/software manufacturers (platform compatibility, reference architectures, benchmarks) plus analysts/media; international resellers who proactively market, sell, implement, and/or service IRI software; and ISVs, ASPs, MSPs, VARs, and system integrators who embed, bundle, or refer IRI software.
No – apart from the time you invest in learning the software and any sales/marketing activities. These are not required or quantified unless territorial exclusivity is desired.
That depends on the nature of the partnership. You can become an informed analyst or referral partner after half a day of orientation; signing the paperwork takes a few days. For resellers and embedding, the process can take days to weeks depending on onboarding and business opportunities.
Working with a vendor that has been stable and profitable since 1978 – without pressure from outside investors, high turnover, or delayed support; relatively high margins with no fees or quotas; de facto exclusivity in several geographic, industry, or technical domains; a well-integrated technology stack (common metadata, Eclipse GUI) for easy onboarding and cross-selling; fast, flexible, and fair commercial responses.
You should bring sufficient technical competence, commercial viability, relevant experience/opportunities, and a good reputation. A specific headcount, corporate history, or certifications are not reliable indicators of success. The most successful partners often have core teams of just 2–3 people for demos, support, sales/marketing, and administration; as opportunities grow, more staff should be added.
That depends on the relationship, e.g.: vendor-neutral consultancy/analyst → NDA or MNDA; referral only → reciprocal finder's-fee MoU; resale → reseller agreement; OEM integration → royalty agreement.
Yes, there are several training paths and certification levels. Generally, there are no extra costs beyond our time and yours for orientation and the path to self-learning. There is a fee for the certificate; however, we do not require certification of our partners. Certified partners do have a clear advantage.
That depends on the product, the nature of the relationship, competitive/customer circumstances, and factors such as features, versions, special cases, and volume discounts. Typically, finder's-fee partners receive a 10% discount, resellers 25–50%. Public price ranges can be found on the website; specific prices are in the reseller guide.
Yes – and that is required when a permanent development copy with support is needed for creating/maintaining a runtime integration or solution bundle, or when production licenses are used in managed-service scenarios. A free but expiring (long-term) license is available for a disaster recovery system and for actively marketing resellers (training, demo, support).
We expressly encourage the use of our current logo and current descriptions. We only ask for pre-approval of the online messaging or branded materials you want to reuse or newly create. We are also happy to document use cases and joint solutions and to co-promote them.
Depending on the relationship, no formal renewal is often needed until an event such as a corporate or mission change occurs. Referral agreements usually auto-renew as long as the relationship is successful, but can be amended/terminated in the event of material technical, commercial, or legal circumstances. Other agreements (exclusive resale, runtime integration, subscription/support) may expire annually or be subject to further conditions.

Data security – classifying, searching, masking PII (IRI *Shield tools)

Both – IRI FieldShield (structured data, via ODBC and JDBC) and IRI DarkShield (structured data plus C/LOB columns, JDBC only) – can find and mask sensitive data in relational databases and are priced the same for them. The use case decides: DarkShield can search and mask separately or simultaneously, FieldShield separates the two; FieldShield can map masked results more easily to different targets (other DB, file, and report targets, even all at once, ETL-style), DarkShield only to the same RDB or a flat file; only DarkShield also handles semi- and unstructured sources (EDI, free text, documents, images, NoSQL, Parquet, etc.); with purely structured sources, FieldShield jobs are metadata-compatible with subsetting, incremental masking, test data synthesis, cleansing, ETL, and reporting thanks to SortCL; and FieldShield supports input filtering (SortCL include/omit or SQL) plus complex target-field logic (combined masking/string transformation, joins, conditional rules, reformatting). Both run on-premise by default but also in the cloud (LAN, SharePoint, Azure, S3, GCP), use the same data classes and masking functions for structural and referential integrity, share the IRI Workbench IDE, have callable APIs, can be integrated into DevOps pipelines and run in CLI jobs, and are components of the IRI Voracity platform. Both products are often licensed in a bundle or as a Voracity transaction. Describe your requirements to us and request an online meeting or a live demo: contact JET-Software.
That varies by use case. With RDB sources/targets, DBA skills and knowledge of data structures are an advantage. CDO/CISO or data governance/security stakeholders should be involved in the definition (classification) – which data types are sensitive and which masking rules apply. Data scientists help with ML/AI aspects (DarkShield NER models). BI/analytics architects are useful for evaluating anonymized outputs, PII search reports, and logs. For production users, knowledge of Eclipse, Git, 4GL/3GL (API), and relevant cloud connections is helpful. TDM architects assist with defining/configuring and provisioning masked, subsetted, or synthesized data.
That depends on your sources, targets, and the functionality required. For DB subsetting and test data, you need IRI RowGen or the IRI Voracity platform (includes subsetting, RowGen synthesis, and all *Shield products). We are happy to advise you on the right choice: Contact us.
By automatically and globally applying the same deterministic masking function to the same plaintext every time. This happens via rules tied to pattern-matched column names or, more reliably, via integrated data classes. Classified data is discovered/validated via RegEx patterns (with adjustable accuracy thresholds), lookup value matching, fuzzy matching, named-entity and facial recognition models, or JSON/XML/CSV/DB path filters. All Shield products – FieldShield, DarkShield, CellShield EE – share the same data classes and deterministic functions, preserving data and referential integrity across structured, semi- and unstructured sources. IRI's integrated data classification also removes the need for formally defined primary/foreign keys – referential integrity is supported even without constraints (in DBs as in files, documents, images). Where constraints are needed for automatically generated RDB test data, they reside in the DB subsetting and test data synthesis wizards; without defined constraints, subsetting/synthesis is possible but requires more manual intervention.
Right – there are more use cases and invocation methods than we can keep current. Several AI models are already in the product, discovering sensitive data via machine learning across structured, semi- and unstructured sources; IRI keeps introducing more. We are happy to demonstrate concrete solutions for your use case in a live demo: Contact us.
Yes – via an external command-line (CLI) call and via an RPC API that can be addressed through web requests from your own calling program.
Privacy-law groups and sensitivity classifications are included out of the box; the remaining default masking rules are not sector-specific but will likely already meet your requirements (e.g., FPE for card numbers, pseudonyms for names, redaction for tax IDs). Data classes, data class groups, and their associated rules are easy to modify – usually only needed once. There is a wide selection of masking rules.
Whatever meets your requirements. In IRI FieldShield (or the SortCL program in IRI CoSort), you can apply per field/column: encryption, masking, hashing, pseudonymization, randomization, de-ID, expression (calculation) logic, substring/byte shifting, data type conversion, or a custom function. Decision criteria per datum: security (how strong/"uncrackable"), reversibility (must what is concealed be revealed later), performance (computational overhead), and appearance (must the original format be preserved). You can protect one or more fields with the same or different functions, or entire records ("wholerec"); conditions and target/layout parameters are customizable and can be combined with transformation and reporting in the same job. Via multi-table wizards or global data classification, DBAs and data stewards can apply these protections consistently as rules database- or enterprise-wide. We are happy to help with the selection: Contact us.
Both! FieldShield and CoSort's SortCL program protect both source types simultaneously with one or more field-level security functions – in bulk (static masking) or surgically (dynamic masking, via filter command/stored procedure). Some DBs have built-in column encryption, but that approach can be cumbersome or limiting: you may need to protect multiple DBs, other sources, or data in motion (flat files) that a single method cannot cover; built-in DB crypto libraries may be too slow, costly, or complex; they are limited to a single method that may not meet security or appearance requirements; or you may need to leave data unchanged in the DB but protect it entering/leaving (this is where flat files come in). Other crypto products encrypt entire files, DBs, disks, or networks – that can take a long time and block access to non-sensitive data. FieldShield and SortCL encrypt (or otherwise protect) only the fields/columns that need it – in the same job script and I/O pass along with transformation, migration, and reporting.
For MongoDB, FieldShield or DarkShield can find and mask sensitive data in different ways depending on the use case (DarkShield covers both cases). For purely structured data in the collections: 1. FieldShield with CSV export/import; 2. FieldShield with CData O/JDBC drivers; 3. FieldShield with the IRI BSON driver. If the collections also contain semi-structured (JSON) or unstructured data (document, image, free text): 4. DarkShield GUI; 5. DarkShield API.
1) TDE offers only AES/3DES and only for MS SQL RDBs. IRI includes more encryption plus many other masking functions and covers multiple sources (other RDBs on-prem/cloud, legacy/COBOL files, free text, JSON/XML, MS Office/PDF documents, NoSQL, images). 2) TDE is not column-specific; IRI allows encryption/masking ad hoc at the field/row level, conditionally at the value level, and consistently at the column level across multiple tables/schemas as a data-class rule (incl. format-preserving encryption) to preserve referential integrity. 3) TDE is tied to SQL and thus vulnerable to SQL injection, which can reverse its entire application; IRI masks specific columns statically with potentially different functions – SQL cannot reverse the data, and even with the executable, the script, and one key, only one column would be exposed. 4) TDE's encryption/decryption is computationally expensive (every DB page); IRI field jobs are only I/O-bound (ODBC read/write speed, further reducible via incremental masking). 5) TDE requires Azure Key Vault (EKM); IRI supports Azure Key Vault too but does not require it (keys locally, as environment variables, or via Townsend Alliance Key Manager). 6) TDE does not interoperate directly with other metadata systems; FieldShield is part of SortCL/4GL – masking in the same job with ETL, CDC, DB subsetting, reorgs, cleansing, migration, reporting; metadata is shared with RowGen and supported by MIMB/erwin. 7) TDE is not integrated with PII classification/search, re-ID risk scoring, audit trails, or SIEM tools (Splunk ES) – all of which is included with FieldShield/DarkShield in the IRI Workbench. 8) TDE is not integrated into test data provisioning systems for DevOps; FieldShield and RowGen integrate directly into test data hubs, web services, or virtualized DB cloning tools (Actifio, Commvault, Windocks).
Automatic discovery of sensitive data & relationships: Oracle discovers columns with sensitive info via built-in patterns (national IDs, credit cards, PII) plus parent-child relationships. The Eclipse-based IRI Workbench IDE covers all discovery functions – DB profiling (statistics, integrity checking, value search), schema-wide PII search/classification, and ER diagrams for Oracle and other RDBs via FieldShield; plus ML/NLP-based named-entity recognition (already in DarkShield for unstructured data). Extensive masking format library & templates: Oracle offers predefined formats and downloadable templates for selected Oracle applications. IRI offers the same and more: selectable and customizable patterns for NIDs, credit cards, phone numbers, etc., plus test data functions for computationally valid NIDs; without Oracle-specific download templates, but with custom formats and 14 categories of masking functions for every Oracle version. Comprehensive masking transformations: Oracle covers conditional masking, consistent outputs, and more. FieldShield offers more functions and preserves consistency/referential integrity via ad hoc or stored rules (pattern-matched column names or pattern/value-based data classes); IRI Workbench additionally offers more job design/management options. Multi-factor subsetting: Oracle produces subsets by size, row percentage, or conditions. A DB subsetting wizard is included in IRI Voracity subscriptions and automatically produces referentially correct, masked/unmasked subsets of any connected schema; robust test data generation from scratch with realistic conditions can also be defined. Fast, secure, heterogeneous: Oracle masks/subsets on clones or during export. IRI does the same and more: masked/subsetted targets go to alternative tables (creatable in IRI Workbench), cloning/masking possible via Commvault/Actifio snapshots; Voracity users additionally perform transformation, migration, and cleansing in the masking job. Beyond that, IRI FieldShield/Voracity offers 15 additional masking and related capabilities available only there.
After you find and classify PII in the free, Eclipse-based IRI Workbench, declare field-level protection functions ad hoc and/or by rule in FieldShield or other SortCL-compatible jobs. These static masking functions provide field-protected views of sensitive data (SSNs, phone numbers, salaries, medical codes) in ODBC-connected DB tables and sequential files via some 14 categories of techniques: field filtering (removal), string manipulation/masking (redaction), quasi-identifier generalization, secure encryption/decryption, reversible and non-reversible pseudonymization, ASCII de-/re-ID, encoding, and byte shifting. The consistent application of these functions (based on data classes or column-name rules) preserves referential integrity. Other paths in Voracity: the built-in DB subsetting wizard (also with masking) or structurally/referentially correct synthetic test data via the RowGen DB test data wizard.
Yes. With FieldShield you can remove (redact, omit, delete) or randomize (random generation/selection as replacement) PII at the column or row level, instead of just obfuscating it (e.g., via encryption/blurring). The target can be a new table of the same structure in another schema that you create, build, and load in IRI Workbench – which is also a cross-platform DB administration environment.
Both, though the latter is more common. For in-place, simply declare the target to be the source. We recommend doing that only after testing the output (e.g., a small test file or stdout) to check format/appearance and functionality (such as reversibility via decryption) if you have no backup.
Failures are logged with reasons. Adjust the rule to be format-preserving or schema-compliant, then run "Mask Only" with the previous annotations to remediate without re-scanning.
In IRI masking products (FieldShield, CellShield, DarkShield), pseudonymization means replacing one identity with another – the values can be consistent and reproducible, some reversible/recoverable, others random. All pseudonymization techniques require a set file with replacement values. Consistent replacement requires two tab-separated columns (a "lookup set" or your crosswalk): every possible value of the unprotected column must appear on the left, followed by a tab and the unique replacement value; the file must be sorted on the left column. In some cases, the application can create the set files itself based on existing data (and optionally a list of replacement values) – but not when pseudonymization is applied via a non-data-specific rule. Replacement values can be supplied via a file or obtained by shuffling the original values; for small data sets a separate set file is advisable, for large ones (names, streets, cities) a shuffled version of the original values often suffices.
Yes, simultaneously. IRI CoSort (via SortCL) or IRI Voracity (via SortCL or interchangeable Hadoop engines) can enforce field-level security in the course of integration, data quality, and reporting jobs. So in the same product, program, and I/O pass, you can mask/redact, encrypt, pseudonymize, or otherwise de-identify PII while transforming, cleansing, and reshaping data from heterogeneous sources. Classic ETL/BI tools cannot do this as efficiently or affordably. In Voracity, you can process (integrate, cleanse), protect (mask), and present (report/analyze) or prepare data all at once. Alternatively, IRI masking programs protect static sources (or you call the API dynamically) to secure only certain fields, which your existing platform then transforms or visualizes – so you keep using your code, protect only the fields that need it, and keep both protected and unprotected data available.
There are several; start with the most recent method, "Masking PII in MongoDB and Cassandra with DarkShield" (4th IRI method). We are happy to advise you: Contact us.
The invocation options are flexible. Storage can become relevant if masked targets are written back to limited SharePoint storage – but that is not required (targets can be local or in other cloud stores like Azure Blob). So you can schedule the configured jobs or run them ad hoc from the GUI, CLI, or via API call.
It depends on the Excel version (not the operating system). The CellShield COM plug-ins must match the Excel version. To check your Office version: start an Office program → "File" tab → "Help" on the left (or "Account" from Office 2013) → "About Microsoft [Program]" section → check the version info at the top (e.g., "… 32-bit"). Note: even on 64-bit Windows you are not necessarily using 64-bit Office.
IRI will provide a web application to supplement the current thick client (IRI Workbench), but will probably never provide software on infrastructure controlled by IRI. Reason: IRI customers want control over their data, and IRI wants neither the liability of storing it nor the costs of SOC2/FedRAMP certifications. Technically, IRI software can already be run as rudimentary SaaS today – via VM/RDP in your cloud (incl. the DarkShield API in a container) and commercially via a pay-go subscription. In addition, DarkShield V7 supports more granular, volume-based license fees for all source types.
The DarkShield service parameters themselves do not determine the resources – it depends on the setup, especially where you host the API. DarkShield scales well and linearly in volume for files and supports load balancing across multiple nodes. General minimum resource recommendations for IRI software can be found in this FAQ.
No data is transmitted from the CellShield plugin. CellShield and the other IRI products for PII discovery and static masking make no connection and need none.
FieldShield, CellShield, and DarkShield (as well as CoSort), and thus Voracity, ship with multiple 128- and 256-bit encryption libraries using proven, compliant algorithms (3DES, AES, GPG, OpenSSL). For each PII element or substring, use the same or a different built-in routine, or link your own library as a custom field-level function; the same algorithm with a different key per field is also possible. Key management is via passphrases in the job script, secure files, and environment variables, plus third-party vaults (Azure Key Vault, Townsend Alliance Key Manager).
Who has access to masking jobs is usually a matter of software/job distribution (e.g., control over IRI projects/jobs in Git). Admin or end-user masking can be done from the local desktop GUI, via a prepared CLI command, or from a third-party program via CLI/API – access control is thus a system administration function (who gets GUI/API access). In distributed environments, the DarkShield API can reside on the same machine as IRI Workbench, on another Windows/Linux system, or in a Linux container (on-prem/cloud), enabling role separation as a network administration function. The future web app roadmap envisions finer role separation/job governance in a more self-contained portal.
All FieldShield and CoSort/SortCL job scripts and field-level functions can be recorded in XML audit logs that you can secure and query with your preferred XML reporting tool (also via SortCL scripts against the logs; samples are provided). In CoSort 11, an even more robust JSON audit log can be generated per a governance policy and evaluated via a data wrangling tool. IRI DarkShield produces multiple logs for search and masking operations.
Yes, e.g., to Splunk – in several ways. See articles such as "Revealing Data Profiling Secrets in Splunk", "Automatically Forward Target or Log Data into Splunk", and "Shedding Light on Dark Data with Splunk ES".