Big Data Analytics: Microsoft Azure Data Lake Store and Qubole

Start Free Trial
September 27, 2017 by Updated June 9th, 2021

Microsoft’s Azure Data Lake Store (ADLS) integration with Qubole Data Service (QDS) is a significant milestone in the journey we started when we launched QDS for Azure Blob Storage in 2017. With this integration, it will now be possible to run rich queries and derive deeper insights from your data in ADLS as well.

ADLS is an enterprise-grade hyper-scale repository for big data workloads. It enables you to capture and process data of any size, type, and ingestion speed in one single place. ADLS supports any application that uses the open-source Apache Hadoop Distributed File System (HDFS) standard. With its HDFS support, you can easily migrate your existing Hadoop and Spark datasets to the cloud without recreating your HDFS directory structure.

ADLS Vs Azure Blob Store

ADLS is storage optimized for big data workloads of all kinds – batch, interactive, and streaming. On the other hand, Azure Blob Store is a general-purpose object store that works well for various use cases and is not specially tuned for read/write accesses of big data workloads. With ADLS, there are no limits on the amount of data you can store, and it is optimized for high-throughput and Input/Output Operations Per Second (IOPS). ADLS also enforces HTTPS protocol for data transfer to and from the store, thereby enforcing better security.

For more details, click here.

Highlights Of ADLS And QDS Integration

  • Configure QDS accounts with ADLS credentials for seamless and transparent access to ADLS on all (Hadoop, Spark, etc.) clusters in your account.
  • Run Apache Hive, Hadoop, and Spark queries through the QDS platform, which can now access data in your ADLS.
  • Migrate data from on-premise storage to ADLS using built-in native tools (in QDS) from a diverse set of storage solutions such as Azure SQL Service, Azure SQL Data Warehouse, Microsoft SQL Server, MySQL, and more.
  • Migrate data from cloud object stores using distributed Hadoop (MapReduce) jobs from Azure Blob Storage to ADLS.

Getting Started

ADLS

Sign up for the Azure portal and create an ADLS account.

Click here for detailed steps.

Free QDS Business Edition on Azure

Visit https://azure.qubole.com and Sign up for free* QDS Business Edition on Azure

For detailed steps, click here.

Using ADLS with QDS

Read the Quick Start Guide here

*Qubole offers Qubole Data Service (QDS) Business Edition at no cost, but Qubole limits usage compute hours (AVMU for Azure) per month, which is approximately a $1000/month value. You must provide your Azure cloud account, and you are responsible for the infrastructure costs managed by Qubole on your behalf.

Start Free Trial
  • Blog Subscription

    Get the latest updates on all things big data.
  • Recent Posts

  • Categories

  • Events

    Data Lake & Data Warehouse – A Modern Data Strategy Discussion

    Oct. 22, 2021 | North America

    Get Technical With Qubole Solution Architects & Engineers

    Oct. 27, 2021 | Online

    Get Technical With Qubole Solution Architects & Engineers

    Nov. 10, 2021 | Online

    The Future of Data Science and Machine Learning at Enterprise Scale

    Nov. 12, 2021 | North America

    Open Data Science Conference

    Nov. 16, 2021 | North America - West

    Data Lake Vs Data Warehouse

    Nov. 17, 2021 | Middle East
  • Read Hive and Presto Clusters with Jupyter on AWS, Azure, and Oracle