How AWS Spot Blocks Can Reserve Spot Instances in QDS

Start Free Trial
September 16, 2021 by Updated October 15th, 2022

As of today, Qubole Data Service (QDS) now supports a new way to reserve AWS Spot instances, and Qubole customers will be able to configure Master and Worker nodes to be Spot blocks.

Spot blocks are instance types that can be reserved for a predefined duration–between 1 to 6 hours. Since Qubole customers’ average cluster life cycle is under 6 hours, this gives them an option to guarantee uninterrupted workload execution with a relatively cheaper price, but higher stability compared to pure Spot nodes. Compared to Spot nodes, Spot blocks provide continuous computation time, especially for applications that are not stateless and have no checkpoints generated in their execution.

For detailed documentation on using AWS Spot instances and Spot blocks in QDS, click here.

In QDS, you have the flexibility of choosing different cluster compositions for both Master and Auto-scaling nodes.

cluster_composition

Original Cluster Compositions
  Master / Min Worker  Auto-scaling Worker
  #1  On-Demand  On-Demand
  #2  On-Demand  Spot
  #3  Spot  Spot
New Cluster Compositions Including Spot Blocks

 

  Master / Min Worker  Auto-scaling Worker
  #4  Spot Blocks  On-Demand
  #5  Spot Blocks  Spot
  #6  Spot Blocks  Spot Blocks

 

In the above:

  • If you know the cluster’s life cycle is within 6 hours and have an estimate about the duration of your workloads, you may choose #4 over #1 (same for #5 over #2) to leverage Spot blocks to achieve the same stability that On-Demand instances have as well as benefit from the lower cost of Spot blocks.
  • #6 is the only composition we would recommend using Spot blocks for auto-scaling. The reason being when auto-scaling kicks in, we need to set a duration for the Spot blocks request. If you choose Spot blocks for Master, Qubole will know the time remaining and use that time for autoscaling requests. For example, let’s say you choose #6 as your cluster configuration, and the duration for Master Spot blocks is set to 6 hours. In this case, after running for a couple of hours, let’s assume an autoscaling event occurs–when this happens, QDS will determine that the cluster life cycle only has 4 hours left (6-2=4). Therefore, when requesting Spot blocks for autoscaling, it will put in a request for 4.

Pricing

Spot block pricing is based on the requested duration and the available capacity; it’s typically cheaper than On-Demand nodes, but usually costs more than Spot nodes. Spot blocks and Spot instances are priced separately; view current spot pricing on AWS and select the “Defined Duration for Linux” tab to learn more.

Note: Spot blocks work very similarly to Spot nodes where users specify bidding parameters including max price. QDS uses 100% On-Demand price to bid for Spot blocks to make sure that the total cost doesn’t exceed what you might pay for On-Demand nodes in the worst scenario.

In Summary

Spot Blocks provide a balance between cost and reliability for users who have the need to keep the cluster running for a definite period of time while keeping the costs at a minimum.

If you’re interested in QDS, sign up today for a free trial!

Start Free Trial
Read How Can Patents Be Beneficial For Your Organization?