Spark runix code#

Typically, caches have a warmup penalty during the first access. Zero overhead cache: Cache reduces the latency of the workloads when the required data has already been accessed and present in the cluster. With help of consistent hashing, each node in the cluster is aware of which data belongs to which node, thus removing the necessity of a centralized system.įile format-agnostic cache: Although RubiX is specially built for columnar file formats such as Parquet and ORC, the same features of the cache can be availed by any big data file formats such as JSON, Avro, Text, etc. RubiX Cache and Its Featuresĭecentralized Architecture: As a distributed cache, RubiX decides the ownership of files among the cluster nodes by using consistent hashing. RubiX is a file system-level cache that enables users to store data in the cluster nodes, reducing the time to read the data from the remote object store for subsequent queries. Where Spark’s internal cache has its drawbacks, RubiX comes to the rescue. Not all users are comfortable making the necessary changes, and going through multiple iterations of fine-tuning what to cache also increases development effort.

Spark runix code#

The application code must be changed to cache the required data explicitly as well. Any other application accessing the same data won’t be able to take advantage of the cached data. This cache is bound to a particular Spark application.If the result size is bigger, Spark application performance will suffer as the memory available to store shuffle data or hash tables decreases. By default, the cache uses its main memory to store the result.

However, this feature also has a number of shortcomings. Users can use Spark internal caching to cache required data, which can reduce the latency of reading data from a remote object store by caching the result of a table scan operation. In this blog, we will focus on the features of RubiX cache, how it differs from Spark’s internal cache, and its performance improvement for Spark workloads. Our internal customers have experienced even greater gains for their I/O heavy workloads. With the use of RubiX, we have seen a significant performance gain - up to 4x for TPC-DS workloads. RubiX has been working as a solution in ad hoc query engines such as Presto. We at Qubole are excited to announce support for RubiX caching in the QDS platform. The architecture of RubiX is a result of efforts to improve query performance in a shared autoscaling Presto, Hive, and Spark cluster that process petabytes of data stored in columnar formats. RubiX, an open-source project from Qubole, was developed to eliminate this I/O latency by caching data in local compute nodes and reducing multiple round trip calls to object-store. For interactive queries, reading the same data over the network adds latency to the overall query time.

With the increased usage of public cloud storage, intelligent management of frequently accessed data has become more important.