Open Source Community Projects

  • Home >
  • Open Source Community Projects

We are committed to making open source data engines more accessible to all developers. Qubole sponsors new open source efforts that improve the utility, performance, and overall experience of managing the cost and complexity of operation.

OPEN SOURCE

Qubole Sparklens

Sparklens is an open source framework to improve the performance of Spark applications. Sparklens can inspect a Spark application and recommend the optimal configuration, such as the number of executors or cores to use.

OPEN SOURCE

RubiX

RubiX is a lightweight data caching framework that can be used by big data engines such as Hadoop. Rubix uses local disks to provide the best I/O bandwidth to the big data engines. Rubix is useful in shared storage architectures where the data execution engine is separate from storage. For example, on public clouds like AWS or Microsoft Azure, data is stored in a cloud store and the engine accesses the data over a network. Similarly, in data centers Presto runs on a separate cluster from HDFS and accesses data over the network.

RubiX can be extended to support any engine that accesses data using the Hadoop FileSystem interface via plugins. There are plugins to access data on AWS S3, Microsoft Azure Blob Store, and HDFS. RubiX can be extended for use with any other storage systems including other cloud stores.