Today, data is creating the most sustainable competitive advantage. However, the ability to use data is limited to data-driven organizations who have the resources to deal with the data volume and processing complexity.
Our vision at Qubole is to radically simplify how organizations leverage data for analytics and machine learning while giving them the choice of cloud, infrastructure, processing capability, and operational control.
We are already in a multi-cloud world, where it’s commonplace for organizations to use different cloud service providers for different use cases. Simplification and choice in this context mean that an organization should not only be able to choose their cloud service provider but also seamlessly migrate from one cloud to the other for efficiency and better economics.
A recent Gartner survey on cloud adoption revealed that 81% of respondents using the public cloud are using more than one cloud service provider (CSP).
Infrastructure simplification is driving the evolution from on-premises to virtualized data centers and finally to the serverless environment for both stateless and stateful services. Again, an organization should have the choice of infrastructure based on their needs.
Also, for data processing, there has been a rapid growth of technologies such as Apache Spark, Presto, Kafka, Flink, and others, each making it easier and more efficient for specific types of use cases. Once again, a developer should have the choice of the right engine for their use case.
So how do we achieve this vision of simplification without compromising the need for choice? The only way to do this is to unify best-of-breed data processing engines and tools in a single platform while providing the deployment flexibility across multiple clouds in a clustered or serverless mode.
This approach is in stark contrast to the fragmented approach seen in the industry today that force-fits either a single cloud, a single processing engine, or a fixed deployment model. Such an approach merely creates silos, increasing complexity.
Qubole’s Next Phase
Today, Qubole is the only data platform that radically simplifies data processing while offering the choice of powerful data processing engines on any cloud. Once configured by administrators, Qubole Data Platform provides a serverless experience for data engineers, data analysts, and data scientists.
Qubole’s customers on AWS, Microsoft Azure, GCP, and Oracle Cloud are processing over an exabyte of data per month using multiple engines such as Apache Spark, Presto, and Hive.
As customers become data-driven, they are adding more data, users, and use cases. However, this also increases the complexity of the infrastructure, which further burdens the platform administrators.
To cope with this growing need, our customers tell us that they would like a real serverless deployment option that eliminates any administrator intervention to manage the environment — but without comprising the fine-grained operational control should they need it for specific use cases.
Introducing Quantum, a New Serverless SQL Engine Available on Qubole Data Platform
Today, we are launching Quantum, a high-performance serverless SQL engine, available on Qubole Data Platform, that simplifies SQL access by offering a true serverless deployment option to enable data analysts to query petabyte-scale volumes of data using ANSI-SQL.
Quantum allows teams to realize value from their data much more quickly, and because of its serverless nature, users pay only for queries they run. Data analysts can query object stores on AWS, Azure, Google Cloud, and Oracle Cloud in seconds to achieve faster time to market with far less IT management overhead.
In addition to the proven benefits of Qubole’s Data Platform for multi-cloud environments, Quantum offers the following features:
- Intelligent Query Management – Reliable query execution through a smart, patent-pending algorithm to estimate and allocate resources.
- Support for Custom Metastores – Use Qubole’s built-in metastore or your preferred cloud metastore. No need to recreate schemas.
- Unified Query Experience – Seamlessly switch between managed and serverless mode as needed.
- Autoscaling – Quantum scales up, down, or out to match your workload and SLA requirements.
- Flexibility of access – Quantum UI, REST API, Scheduler or ODBC/JDBC access.
With the introduction of Quantum, users can use either a serverless SQL engine or a managed Presto cluster for finer-grained control.
With the release of Quantum, Qubole Data Platform once again becomes the industry’s first data platform to offer both clustered and serverless engines for SQL access in a single data platform, thereby simplifying access while preserving choice.
Ashish Thusoo is the Co-Founder and CEO of Qubole Inc.