Blog

×

AWS Spot Blocks – New Way to Reserve Spot Instances in QDS

By April 13, 2017

Co-authored by Goden Yao, Product Manager, Qubole and Somya Kumar, Member of the Technical Staff, Qubole

As of today, Qubole Data Service (QDS) now supports a new way to reserve AWS Spot instances, and Qubole customers will be able to configure Master and Slaver nodes to be Spot blocks.

Spot blocks are instance types that can be reserved for a predefined duration–between 1 to 6 hours. Since Qubole customers’ average cluster life cycle is under 6 hours, this gives them an option to guaranteed uninterrupted workload execution with relatively cheaper price, but higher stability compared to pure Spot nodes. Compared to Spot nodes, Spot blocks provide continuous computation time especially for applications that are not stateless and have no checkpoints generated in its execution.

For detailed documentation on using AWS Spot instances and Spot blocks in QDS, click here.

In QDS, you have the flexibility of choosing different cluster compositions for both Master and Auto-scaling nodes.

cluster_composition

Original Cluster Compositions
  Master / Min Slave  Auto-scaling Slave
  #1  On-Demand  On-Demand
  #2  On-Demand  Spot
  #3  Spot  Spot
New Cluster Compositions Including Spot Blocks
  Master / Min Slave  Auto-scaling Slave
  #4  Spot Blocks  On-Demand
  #5  Spot Blocks  Spot
  #6  Spot Blocks  Spot Blocks

 

In the above:

  • If you know the cluster’s life cycle is within 6 hours and have an estimate about the duration of your workloads, you may choose #4 over #1 (same for #5 over #2) to leverage Spot blocks to achieve the same stability that On-Demand instances have as well as benefit from the lower cost of Spot blocks.
  • #6 is the only composition we would recommend to use Spot blocks for auto-scaling. The reason being when auto-scaling kicks in, we need to set a duration for Spot blocks request. If you choose Spot blocks for Master, Qubole will know the time remaining and use that time for autoscaling requests. For example, let’s say you choose #6 as your cluster configuration and duration for Master Spot blocks is set to 6 hours. In this case, after running for a couple of hours, let’s assume an autoscaling event occurs–when this happens, QDS will determine that the cluster life cycle only has 4 hours left (6-2=4). Therefore, when requesting Spot blocks for autoscaling, it will put in request for 4.

Pricing

Spot blocks pricing is based on the requested duration and the available capacity; it’s typically cheaper than On-Demand nodes, but usually cost more than Spot nodes. Spot blocks and Spot instances are priced separately; view current spot pricing on AWS and select “Defined Duration for Linux” tab to learn more.

Note: Spot blocks work very similar to Spot nodes where users specify bidding parameters including max price. QDS uses 100% On-Demand price to bid for Spot blocks to make sure that total cost doesn’t exceed what you might pay for On-Demand nodes in the worst scenario.

In Summary

Spot Blocks provide a balance between cost and reliability for users who have the need to keep the cluster running for a definite period of time while keeping the costs at the minimum.

If you’re interested in QDS, sign up today for a free trial! To get an overview of QDS, click here to register for a live demo.

Share our Post

Comments are closed.