Benchmarking Auto-scaling Spark Clusters

Start Free Trial
August 8, 2016 by Updated March 7th, 2021


Have you ever had trouble deciding how large to make a cluster? Do you sometimes feel like you’re wasting money when a cluster isn’t being fully utilized? Or do you feel like your analysts’ time is being wasted, waiting for a query to return? At Qubole, we developed auto-scaling in order to help combat these problems.

Auto-scaling removes and adds cluster nodes depending on the workload. You simply select a minimum and maximum number of nodes and the auto-scaling algorithm does the rest. Workloads can be unpredictable day-to-day or hour-to-hour; auto-scaling lets you sit back and not have to worry about mis-sizing a cluster. Qubole was the first Big Data platform to introduce auto-scaling back in 2012, and we have been improving it ever since.

We wanted to show that auto-scaling can help with a number of challenges, such as:

  • Cluster underutilization which causes wasted node hours
  • Cluster overload which leads to slower execution

To demonstrate the utility of auto-scaling, we decided to simulate a real customer’s workload on three clusters:

  1. A static cluster with 5 slave nodes
  2. A static cluster with 10 slave nodes
  3. A cluster that auto-scales between 1 and 10 slave nodes

Our analysis resulted in two key takeaways. First, the 5 node cluster was almost twice as slow as the auto-scaling cluster, but it was only incrementally cheaper. Second, the 10 node cluster was a little bit faster than the auto-scaling cluster, but it was more expensive by a third, which can really add up quickly for compute costs.


5 static is slower

than Auto-scaling

by 90%

5 static is less expensive

than Auto-scaling

by 10%

10 static is faster

than Auto-scaling

by 13%

10 static is more expensive

than Auto-scaling

by 32%



ClusterTotal Number of Node HoursTotal Runtime (in Hours)
5 Static90*146
10 Static13267
1-10 Auto-scaling10077

Auto-scaling vs. 5 Nodes

The 5 node cluster took 8,757 minutes total, while the scaling cluster took only 4,644. This means that the 5 node cluster was 90% slower than the auto-scaling cluster; auto-scaling was almost twice as fast!

If we compare in terms of node hours, we can see that the auto-scaling cluster used 100 node hours, while the 5 node cluster used 90 node hours. This means that the 5 node cluster used only 10% more node hours than the auto-scaling cluster. Those 10 additional node hours buys you almost 70 hours your analyst could spend deriving essential business insights rather than waiting for a query to return.

Auto-scaling vs. 10 Nodes

The 10 node cluster took 3,999 minutes and the scaling cluster took 4,644. This means that the 10 node cluster ran about 13% faster than the auto-scaling cluster. This makes sense because the static cluster always had at least as many nodes active as the auto-scaling cluster. While 13% isn’t nothing, it isn’t significant when compared to the cost saving in terms of node hours.

If we compare the node hours, we see that the 10 static cluster used 132 node hours, while the auto-scaling cluster used only 100 node hours. This means that the 10 static cluster cost 32% more than the auto-scaling cluster. This dwarfs the 13% advantage that the static 10 node cluster has in terms of performance.

Commands per hour plotted against the number of nodes in each hour; the commands per hour is based on a real customer’s workload


For this particular benchmark analysis, we decided to focus on the benefits of auto-scaling for data analysts (as opposed to data scientists or data engineers). Data analysts tend to issue queries from the UI and their queries tend to be be highly specific. Importantly, the timing of their queries tends to be ad hoc, which compounds their unpredictability. In future benchmarks, we will look at more varied use cases with more job types and concurrency.

Commands Per Hour

The number of commands by hour; made using a Zeppelin notebook

We selected a typical day for a real customer cluster which ran jobs only from the Qubole’s ‘Analyze UI’ (used mostly by data analysts) and counted the number of commands in every hour (visualized in the line chart above). To improve reproducibility, we scaled down the workload by dividing the number of commands by two and rounding the result. We decided to focus on the 10 hours with the highest numbers of commands: 7AM to 5PM. This resulted in the following number of commands per hour:

Number of Commands Run67841010121711212

The number of commands per hour in this cluster varies a lot throughout the day. This high amount of variation is typical and follows the trend we would expect to see: during the work day (7AM-5PM) there is much more activity than during the evening, night, and early morning–there are actually no commands run between midnight and 7AM.

This big variation in number of commands per hour is partially explained by the difference in the number of users active in each hour. The line graph below shows how many users were active in each hour and the huge variation in how many commands each user issued. Tommen (user name obfuscated to protect customer privacy) ran just 3 queries total; Arya by contrast ran 16 queries in just one hour. These variations account for how unpredictable the load on a cluster may be in any given hour.

The number of commands by hour by user; made using a Zeppelin notebook

Query Selection

We chose TP-CDS (the standard benchmark for Big Data systems) Query 42, which typifies the query an analyst would use. Query 42 has a lot of specificity and is clearly an ad hoc query. We had to superficially modify the query to make it compatible with Spark SQL. You can find a copy of our modified query here.

We then scheduled the number of commands for each hour (6 commands for 7AM, 7 commands for 8AM, etc) for each one of the clusters. The queries were run concurrently and kicked off at the top of the hour.

Environment Details

The clusters had the following configurations:

  • Spark version 1.6.1
  • Master node type and slave node type: r3.xlarge – 4 cores, 30.5GiB mem
  • Cluster auto-termination disabled*
  • On-demand instances
  • Default Spark configurations
    • spark.executor.cores 2
    • spark.executor.memory 5120M
    • 0.5

Raw Results

Total Time Taken

HourNumber of Commands5 Static10 Static1-10 Auto-scaling
7AM6 Commands269 Minutes209 Minutes298 Minutes
8AM7 Commands320 Minutes247 Minutes288 Minutes
9AM8 Commands396 Minutes265 Minutes389 Minutes
10AM4 Commands152 Minutes118 Minutes193 Minutes
11AM10 Commands678 Minutes423 Minutes433 Minutes
12PM10 Commands781 Minutes325 Minutes472 Minutes
1PM12 Commands1,117 Minutes396 Minutes488 Minutes
2PM17 Commands1,977 Minutes608 Minutes709 Minutes
3PM11 Commands604 Minutes412 Minutes405 Minutes
4PM21 Commands274 Minutes733 Minutes704 Minutes
5PM2 Commands2,189 Minutes263 Minutes265 Minutes

Node Count (excludes master node)

Hour5 Static10 Static1-10 Auto-scaling
7AM5 Nodes10 Nodes5 Nodes
8AM5 Nodes10 Nodes6 Nodes
9AM5 Nodes10 Nodes7 Nodes
10AM5 Nodes10 Nodes5 Nodes
11AM5 Nodes10 Nodes8 Nodes
12PM5 Nodes10 Nodes8 Nodes
1PM5 Nodes10 Nodes10 Nodes
2PM5 Nodes10 Nodes10 Nodes
3PM5 Nodes10 Nodes9 Nodes
4PM5 Nodes10 Nodes10 Nodes
5PM5 Nodes10 Nodes8 Nodes
6PM5 Nodes0 Nodes0 Nodes
7PM5 Nodes0 Nodes0 Nodes

Looking Ahead

This is the first in a series of blog posts exploring auto-scaling. For the sake of simplicity, in this benchmarking analysis we decided to only look at the benefits of auto-scaling to data analysts using Qubole. But Qubole has a diverse range of user types, including not just analysts but data engineers and data scientists as well. In our next benchmarking, we are going to use a more diverse set of queries to capture the diversity of use cases.


* The number of node hours in the static 5 node cluster is not exactly half of the number of node hours in the static 10 node cluster for two reasons. First, we need to count the master node in addition to the slave nodes. So, the “5 node cluster” actually ran 6 nodes every hour (including the master node), while the “10 node cluster” actually ran 11. Additionally, the static 5 node cluster ran longer than either of the other clusters because of the time that it took for its queries to finish executing.

** Cluster auto-termination is a feature Qubole implemented to help customers avoid accidentally leaving an inactive cluster running. It causes a cluster to terminate when there has been no activity on the cluster from a job created through the UI or API. Since our jobs were created through the scheduler, the clusters could shut down prematurely and start up again at the start of the next hour. To avoid this situation, we disabled auto-termination for this benchmark. For more information about auto-termination, check out “Shutting Down an idle cluster” on this page.

Start Free Trial
  • Blog Subscription

    Get the latest updates on all things big data.
  • Recent Posts

  • Categories

  • Events

    Data Lakes vs. Data Warehouses

    Aug. 10, 2021 | Eastern NA

    Data Lakes vs. Data Warehouses

    Aug. 11, 2021 | India

    Data Lakes vs. Data Warehouses

    Aug. 12, 2021 | Central North America

    Data Lakes vs. Data Warehouses

    Aug. 18, 2021 | West North America
  • Read Qubole’s Notebook Integration with Github is Generally Available