Talend Turbocharging
Accelerating Talend Jobs
Challenges
Even after consultation and coordination, large amounts of data (i.e., more than one million rows) can only be transformed slowly, especially without an expensive hardware or version upgrade from Talend.
Major data bottlenecks are large sorts, joins, aggregations, loads, and sometimes even spills. Parallelizing or optimizing at other layers or tools can be cumbersome, if not expensive, and impact performance for other users.
From a security perspective, Talend may not offer the data detection, classification, or masking features, or data testing capabilities that data stewards and application developers require.
Solutions
Sort acceleration, aggregation, and merging of Talend ETL flows with a tSystem call to programs CoSort Sort Control LanguageSortCL). Perform large data transformations multiple times time-efficiently, without crashing due to lack of disk space and without burdening other jobs in Talend, your DB, or BI tool. Also specify file format and data type conversions, field-level masking and cleansing functions, custom report layouts, and pre-sorted load files.
In this article Find benchmarks and advice on performance improvement through system calls, CoSort, or Voracity-Available to users. Accelerate and combine transformations and other large data staging jobs in much less time and storage space.
Data residing in tables and flat files within Talend can be sensitive and contain personal data subject to confidentiality restrictions and data privacy laws. Both IRI Voracity as well as IRI FieldShield Fields in structured datasets can be protected in any ODBC-connected database or flat file format.
Your business rules determine the function you choose for each column, i.e., format-preserving AES-256, FIPS-compliant OpenSSL, 3DES and/orGPG encryption, lookup value substitution (pseudonymization), character masking, hashing, redaction, custom expression logic, substring, or user field function.