CLF/ELF Weblog Conversion, Processing & Protection

General & Extended Weblogs

Convert, Transform, Protect, and Report on Clickstream Data

Challenges

NCSA Common and W3C Extended Log Formats (CLF and ELF) are two popular structures for logging clickstream activities on websites. The recorded visitor information can include the IP address, timestamp, page URL, entry and exit pages, etc.

As these logs grow larger, processing can take a very long time. They may not be in formats that your applications recognize or easily support, and they can contain Personally Identifiable Information (PII) that needs to be masked or encrypted for compliance or other reasons.

Solutions

IRI has technologies for converting web log file formats, as well as for transformation, conversion, protection, and reporting of Web log data for clickstream analysis and data webhouse operation.

The IRI software can address data in CLF, ELF, and other structured (sequential) web log formats, but has done additional work to facilitate the handling of CLF and ELF files. For clickstream data in unstructured text file formats, IRI offers a data restructuring feature to extract sought-after strings and incorporate them into structured data stores for the same activities as described below.

The pure conversion solution lies in the free (lite) or cost-effective (database) version of IRI NextForm. NextForm can convert CLF and ELF files to CSV and other flat file formats for free.

The upgrade edition that connects to databases populates weblog data directly into relational tables. NextForm allows you to convert files in LDIF format to other formats like CSV, XML, Text, etc., and vice versa. NextForm also supports field-level data type conversion and record layout remapping.

NextForm file definitions also work in SortCL, the popular 4GL and the data transformation and reporting program in IRI CoSort.

Beyond format conversion, CoSort can SortCL-Program these huge log files into CSV, LDIF, XML, text, index, and other structured file formats and DB tables, process (filter, sort, join, aggregate, scrub, reformat, etc.), secure, and report from these huge log files.

With a simple 4GL to define the layout and manipulate your log files, you can run and combine data:

    Transformation (Scrubbing, Sorting, Merging, Grouping, Calculating, Reordering, etc.)
    Conversion (Data types, record layouts, files)
    Masking (Field-level encryption, de-ID, etc.)
    Reporting (Custom details, delta, and summaries)

plus validation, find/replace, custom transformations, etc. in the same job script and I/O pass. With SortCL, you can map one or more sources in one or more formats to one or more detail or summary reports, and/or pass filtered subsets to specialized clickstream analysis tools. See this blog article for an example.

To protect PII in weblog files, use SortCL or IRI FieldShield. FieldShield offers the industry's most comprehensive suite of field-level security features for data at rest. Mask, encrypt, pseudonymize, randomize, or otherwise obfuscate and deidentify email addresses, IP addresses, and other data privacy elements. See this blog article for an example.

Do you need test data in log file formats? IRI RowGen supports random generation and selection of field-level data to create custom files. RowGen uses the same layout metadata as NextForm, CoSort (SortCLand FieldShield, so you can switch seamlessly between test data generation and real data processing.