Data Lakes are becoming increasingly central to the analytical operations of organizations.  This brings in many more ‘transactional’ requirements on the pipeline architecture and the… The post...

In Part 1 of this series, we briefly touched upon the various design considerations to be made when architecting the Data Lake. We saw how… The post Part 2: Tuning the Data Ingestion process...

Increase data security and simplify the infrastructure with Qubole About Qubole Open Data Lake Platform Qubole is an open and secure data lake platform for… The post Enhanced Network Security with...

Data Lakes are a core pillar in an organization’s data strategy. Data lakes make organizational data from different sources, accessible to various end-users like business… The post Part 1:...

For decades our desks were covered in trophies, certificates, and medals demonstrating our accomplishments, achievements, and competencies. Over the time, these methods of recognition have… The...

We are pleased to announce the 0.6.0 release of ACID Data source for Apache Spark. This release should further empower Data lake users in enterprises… The post Enabling Spark SQL MERGE via...

We are pleased to announce the availability of Apache Spark 3.0 in the Qubole environment. Spark 3.0 release comes with a lot of exciting new… The post Introducing Apache Spark 3.0 on Qubole...

In our last blog, we covered all the basic concepts of Apache Airflow. In this blog, we will cover some of the advanced concepts and… The post Apache Airflow Concepts – DAG Scheduling and...

AWS Spot instances reduce cloud costs by up to 90% but can be interrupted by AWS at any given time causing running workloads to fail.… The post Introducing Capacity Reservation for Application...

It is common knowledge that data lakes offer the right architecture to support multiple use cases and tools, but can be operationally complex to implement… The post Data Discovery Tools – Qubole...

Now that you have read about how different components of Airflow work and how to run Apache Airflow locally, it’s time to start writing our… The post Apache Airflow Tutorial – DAGs, Tasks,...

In the past 2-3 years, Presto has set the bar for fast analytical processing in modern cloud data lake architectures. Qubole has offered a Presto… The post Presto on Qubole is 2.6x faster than...

Image credits: https://science.howstuffworks.com/terraforming.htm The Qubole Open Data Lake Platform Qubole is the open data lake company that provides a simple and secure data lake platform… The...

Running Large distributed Apache Spark clusters in the public cloud, that handle exponential increase in volumes of data to fuel analytics and Machine Learning (ML)… The post Logan: A Data-Driven...

Introduction Ad-hoc analytics and data exploration require compute resources that can process incoming jobs instantaneously and keep the response time low. Apache Spark is a… The post Cost and...

Columnar data formats have become the standard in data lake storage for fast analytics workloads as opposed to row formats. Columnar formats significantly reduce the… The post Columnar Format in...

Qubole is excited to announce the general availability of Managed Spot Block instances that provides up to 40% cost savings over On-Demand Ec2 Instances. Managed… The post Introducing Managed Spot...

As a general-purpose programming language, Python is universal. It’s quick and easy, but yet powerful with plenty of capabilities. It gives you an opportunity to… The post Boosting Parallelism for...