How AWS Spot Blocks Can Reserve Spot Instances in QDS

Start Free Trial
September 16, 2021 by

As of today, Qubole Data Service (QDS) now supports a new way to reserve AWS Spot instances, and Qubole customers will be able to configure Master and Worker nodes to be Spot blocks.

Spot blocks are instance types that can be reserved for a predefined duration–between 1 to 6 hours. Since Qubole customers’ average cluster life cycle is under 6 hours, this gives them an option to guaranteed uninterrupted workload execution with relatively cheaper price, but higher stability compared to pure Spot nodes. Compared to Spot nodes, Spot blocks provide continuous computation time especially for applications that are not stateless and have no checkpoints generated in its execution.

For detailed documentation on using AWS Spot instances and Spot blocks in QDS, click here.

In QDS, you have the flexibility of choosing different cluster compositions for both Master and Auto-scaling nodes.


Original Cluster Compositions
  Master / Min Worker  Auto-scaling Worker
  #1  On-Demand  On-Demand
  #2  On-Demand  Spot
  #3  Spot  Spot
New Cluster Compositions Including Spot Blocks
  Master / Min Worker  Auto-scaling Worker
  #4  Spot Blocks  On-Demand
  #5  Spot Blocks  Spot
  #6  Spot Blocks  Spot Blocks


In the above:

  • If you know the cluster’s life cycle is within 6 hours and have an estimate about the duration of your workloads, you may choose #4 over #1 (same for #5 over #2) to leverage Spot blocks to achieve the same stability that On-Demand instances have as well as benefit from the lower cost of Spot blocks.
  • #6 is the only composition we would recommend to use Spot blocks for auto-scaling. The reason being when auto-scaling kicks in, we need to set a duration for Spot blocks request. If you choose Spot blocks for Master, Qubole will know the time remaining and use that time for autoscaling requests. For example, let’s say you choose #6 as your cluster configuration and duration for Master Spot blocks is set to 6 hours. In this case, after running for a couple of hours, let’s assume an autoscaling event occurs–when this happens, QDS will determine that the cluster life cycle only has 4 hours left (6-2=4). Therefore, when requesting Spot blocks for autoscaling, it will put in request for 4.


Spot blocks pricing is based on the requested duration and the available capacity; it’s typically cheaper than On-Demand nodes, but usually cost more than Spot nodes. Spot blocks and Spot instances are priced separately; view current spot pricing on AWS and select “Defined Duration for Linux” tab to learn more.

Note: Spot blocks work very similar to Spot nodes where users specify bidding parameters including max price. QDS uses 100% On-Demand price to bid for Spot blocks to make sure that total cost doesn’t exceed what you might pay for On-Demand nodes in the worst scenario.

In Summary

Spot Blocks provide a balance between cost and reliability for users who have the need to keep the cluster running for a definite period of time while keeping the costs at the minimum.

If you’re interested in QDS, sign up today for a free trial! To get an overview of QDS, click here to register for a live demo.

Start Free Trial
  • Blog Subscription

    Get the latest updates on all things big data.
  • Recent Posts

  • Categories

  • Events

    Data Lake & Data Warehouse – A Modern Data Strategy Discussion

    Oct. 22, 2021 | North America

    Get Technical With Qubole Solution Architects & Engineers

    Oct. 27, 2021 | Online

    Get Technical With Qubole Solution Architects & Engineers

    Nov. 10, 2021 | Online

    The Future of Data Science and Machine Learning at Enterprise Scale

    Nov. 12, 2021 | North America

    Open Data Science Conference

    Nov. 16, 2021 | North America - West

    Data Lake Vs Data Warehouse

    Nov. 17, 2021 | Middle East
  • Read Data Lake and Data Warehouse: Can the Platforms Work in Synergy?