R is a free programming language and software environment used by statisticians and data miners for analyses and predictions, and has become known as a visualization tool for large datasets. However, because R holds all its objects in memory, it cannot work effectively with very large datasets.

The SortCL program in the IRI Voracity Big Data Management Platform or the standalone IRI CoSort The `paket` package is a fast, simple, and cost-effective way to efficiently prepare large data for R, in terms of both job design and runtime performance.

When we ran a SortCL sort, join, and aggregation job before (and instead of) R, time-to-visualizations in tools like ggplot or qplot was halved:

R only works with several small data blocks and requires multiple code files to achieve the same result as a single SortCL job. Hadoop is another way to quickly prepare large datasets for R, and Voracity users can take most SortCL jobs seamless run in Map Reduce 2, Spark, Spark Streaming, Storm, or Tez without additional programming.

Further information on the benchmark and how SortCL can process data in the same Eclipse environment (via the StatET for R Plug-in for IRI Workbench), find here.