Test data generation for databases
Create a realistic, referentially correct schematic
Challenges
DBAs and data architects attempting to populate test databases often rely on unsafe and insufficient production data. There are a number of issues with this approach, including:
In addition to the realism and anonymity of the test data, availability is also a key consideration. Another key challenge for database testing solutions is how the test data is made available for use. It must be accessible in the required formats and silos (or DevOps pipelines) and remain consistent or recoverable in the event of changes.
Solutions
There are several tools on the market for generating database test data. The most flexible, fastest, and most robust test data solutions are provided by the IRI RowGenSoftware created.
With RowGen, you can approach database testing management from whatever angle best suits your use case. The primary method for generating test data is data synthesis, meaning RowGen generates synthetic yet highly realistic database test data. Alternatively, users of the IRI RowGen, FieldShield, or Voracity platforms can use their license for database subsetting (with or without data masking). Regardless of your strategy, the Eclipse user interface, the job metadata, and the engine behind it all remain the same.
The software works across databases using the same easy-to-define and reusable rules for test data generation. Your rules for generating (or masking) test data generally apply consistently to all data classes (e.g., columns) and use either the generation of random values or the selection of random values from other datasets. Once you have approved the rules, you can apply them with automate an assistant, which creates your database test data request.
Before you run the wizard, you should consider some of the key issues to keep in mind when defining your test data requirements. Read the series of articles on managing database test data and provisioning test data on the IRI blog, which begins here. Then you'll learn why the RowGen test data product—or the IRI Voracity data management platform that includes it—has been the best test data generation tool is necessary to achieve these goals.
The IRI RowGen-The software generates accurate and reliable test data that reflects the table formats, sizes, value ranges, and constraints of the production database—without using production data. RowGen uses your DDL information to quickly and automatically create and load large test tables that are structurally and referentially correct. For more information, see:
Blog > Test Data > RowGen automates the generation of test data for the database
Blog > Test Data > How to Generate Database Test Data
Video > Populating a database from scratch with realistic test data
Valid test tables must include the full range of values, data types, row layouts, and primary foreign key relationships on which the database application relies. For information on how to RowGen maintains data realism; see:
By generating test data for each phase of development, you can create phases simultaneously and independently of other phases. For example:
Step 1 – Read a personnel table and link the health insurance claim data to a status file.
Step 2 – Read the status file and create a list of doctors.
Step 3 – Create a web-enabled billing summary sorted by patient.
RowGen can generate the input table for Step 1, the status file for Step 2, and the web report in Step 3—all without requiring actual data or data from the other steps.
RowGen allows you to generate secure test data of any type, size, layout, and volume, enabling you to create and deploy a more secure development environment.
Alternatively, you can use IRI FieldShield Use real data that is protected by field-level functions (e.g., encryption). For more information, see: Solutions > Data masking
Both tools exist side-by-side in the IRI Workbench GUI, which also includes an automatic Database Subsetting with data masking. Either way, with secure data, there is no need to certify or lock in your outsourced application developers.
RowGen shows you what the actual data might look like and what the transformation and reporting application might look like. This is because both RowGen and the program SortCL in IRI CoSort and Voracity use the same metadata to define data manipulations and table layouts.
This means RowGen can also perform the same data transformations and reporting functions you would normally perform in the same I/O pass and job script that performs test data generation. It also means the same layouts (and even transformations) created for data synthesis jobs are immediately ready for data integration, masking, migration, reporting, and other data processing jobs when and as soon as real data becomes available.
If you only use RowGen, you can easily upgrade to CoSort or Voracity to transform and report on your real data (when available). Using the same RowGen job script that defines and transforms your test data, you can transform your real data in the same format.
IRI Test Data Generators can provide testers with test data in various ways, including:
- Ad hoc tables (or files in various formats) in persistent or federated (view) schemas
- Golden copies of test data in existing lower environments as well as the ability to create new tables
- During the cloning of databases in tools like Windocks, Actifio (Google), and Commvault
- Within DevOps pipelines like Amazon CodePipeline, Azure DevOps, Jenkins, GitLab, etc.
- Directly over Test Data Management (TDM) hubs from Cigniti or ValueLabs
On the pages DevOps and Virtual test data In this section, you'll learn more and find links to further information about these options. Compare these methods and costs to more expensive and proprietary solutions that require special drivers or the setup of ephemeral databases.