A Technical Overview of Quantum by Qubole: An Interactive Serverless SQL Engine

June 10, 2019 by , , , , and Updated August 1st, 2019

Qubole is excited to announce the general availability of Quantum, an interactive serverless query engine available on Qubole Data Platform that offers direct standards-compliant SQL access to your object stores or data lakes. Quantum does not require any infrastructure setup. Users only pay for the queries they run.

Why Serverless?

Distributed SQL query engines like Presto are designed with in-memory pipelined architectures to reduce query latency while sacrificing little on query reliability. Typically, enterprises run large Presto clusters for data analysts and data scientists to run queries at scale. This approach is necessary for many use cases, particularly for complex and large reporting or analysis tasks. However, this owned-cluster approach  requires administration and configuration tasks, such as:

  • Setting appropriate cluster configuration parameters that may accommodate all types of workloads within the organization
  • Choosing the most cost-effective and performant cluster node types
  • Monitoring cluster health and adjusting the aforementioned configurations when user demands change

Qubole customers comprise a broad spectrum ranging from administrators who have gained expertise in tuning their clusters and deploy extensive monitoring of workloads and data pipelines, to data analysts or data scientists  who do not want to spend any time or effort in tuning or monitoring their clusters, but only need fast access to their data.

By providing a serverless SQL query engine in addition to Presto, Qubole offers customers multiple options to choose the best approach to fit their organization’s business requirements.

Quantum combines our Presto expertise with enterprise-grade infrastructure security and our experience in infrastructure operations and monitoring, along with value-add features such as custom metastore support, richer standards-compliant SQL statement support, service level assurances, and a unified experience for accessing object stores or data lakes.

Custom Metastore Support

Existing serverless SQL service offerings do not provide users with the ability to use a  metastore of their choice. With Quantum, data teams can use their own custom metastore and start using Quantum without recreating schemas or table metadata.

Most existing Qubole customers already use a custom metastore in the cloud. So there’s virtually no ramp up time to reap the benefits of Quantum.

Richer SQL Statement Support

Quantum supports both Hive DDL and Presto SQL*. Moreover, Hive Authorization and Presto Permission–related DDLs are also supported along with additional DDLs like ‘Alter table … Add Columns…

It also supports multiple SQL statements in a single query record, which can help users  streamline their SQL scripts and workflow.

* Presto SQL is compliant with ISO 9075 standard for SQL/Framework

Consistent Query Performance & SLAs

Quantum has been designed with a usage quota model according to which each account (or customer) is assigned a usage quota in terms of maximum concurrent queries and concurrent peak memory consumption. When a user submits a query, Quantum predicts the peak memory requirements for the given query using table statistics and its knowledge of the dependency graph of various subtasks in the query.

Upon receiving the resource estimate of a given query, Quantum compares the existing resource usage of that account with the account-level usage quota—preconfigured based on the service level agreement (SLA)—to determine how best to execute the query in the current circumstances.

Queries within the quota get preferred scheduling order and are never preempted. Queries that are out of the quota may experience deferred scheduling and may also get preempted to make space for other in-quota queries. In each case, Quantum proactively autoscales to accommodate queries as per their account-level SLA.

Enterprise-Grade Security, Encryption, and Protection

Quantum implements several infrastructure enhancements that provide enhanced security and protection:

  • Quantum runs in its own Virtual Private Cloud (VPC), which provides complete isolation and separation from the VPC that hosts the Qubole Data Platform control plane.
  • All customer credentials from the Qubole Data Platform control plane to Quantum and thereafter within Quantum components are encrypted and/or transferred over secure transport protocols such as SSL or over SSH.
  • Each Quantum component runs within its own AWS security group that has specific permissions for inbound and outbound access to other Quantum and Qubole Data Platform control plane components. This provides Quantum with strict isolation both in its own components as well as from components within the Qubole Data Platform control plane, and reduces the overall attack surface.
  • Quantum clusters have been augmented with security forensic and diagnostic tools such as OSQuery and cmd.com, which allows Qubole to react to any potential suspicious  activity in real time. 

Infrastructure Monitoring and Operations

Qubole provides pervasive infrastructure monitoring and operational support for Quantum. Building on the excellent Datadog monitoring support embedded in Presto on Qubole, plus more useful metrics and alerts, Qubole can detect potential performance or availability problems very efficiently. Moreover, backed by Qubole’s 24×7 DevOps support and 3-tier technical support, Quantum delivers the enterprise-grade assurance required for today’s data operations.

Unified Experience:

Quantum provides the same experience as with any other Qubole offering:

  • Unified Query Experience:
    Supported through UI, REST API, SDK and ODBC/JDBC (including third-party BI tools, such as Tableau or Looker): Users can seamlessly switch between Quantum and other engines on the native user interface, or through the REST APIs or SDKs or ODBC/JDBC via a single parameter change.
  • Unified Billing Experience:
    Quantum pricing is based on the amount of data read/scanned in a query, but  all charges are unified under Qubole Compute Units (QCU), as with any other charge on Qubole.

Conclusion

Quantum delivers a serverless interactive query experience  for fast access to object stores or data lakes. It builds on Qubole’s cloud-native expertise offering Presto on Qubole since 2014. Contact your Qubole account representative to try it today!

  • Blog Subscription

    Get the latest updates on all things big data.
  • Recent Posts

  • Categories

  • Events

    No upcoming events at this time.