Policy for Supported Apache Spark Versions in Qubole

April 19, 2017 by Updated March 1st, 2019

At Qubole, we’ve made significant progress on our adoption of Spark on Qubole with new features and scalability. And to accommodate growing demands and leverage technological advancements made by the Apache Spark community, we at Qubole continue to support new versions as soon as possible while still supporting a trailing list of stable versions for production workloads. By focusing on the latest major versions, we can concentrate our operational efforts into testing and quality to ensure the best production experience possible.

In an effort to align our pace of development with the project, we did a careful analysis of which versions of Spark were being used by our customers. Based on our conclusions, we have made some difficult decisions to discontinue support for Spark versions older than 1.6.2. This will not affect you immediately, and we are putting a timeline in place for the deprecation of these versions.

Most of these changes will happen over the next few months, but today we are giving you advance notice. As always, we will strive to keep changes to a minimum and announce them in a timely manner.

Spark 1.x — Supported Versions Schedule

We are introducing two new stages to Spark version support—Deprecation and Sunset. In the first stage, “Deprecation”, the version will be marked as “Deprecated” in the cluster settings when selecting a Spark version. In the second stage, “Sunset”, which happens 90 days from “Deprecation” date, the Spark version will be removed altogether from the cluster settings, and the version will no longer be supported by the Qubole  support team. We will exercise due diligence in ensuring we do not break such unsupported versions. However, a feature in a new release might be backward compatible with an older revision, thus rendering it unusable. While this is not our intent, we must be realistic about the future of our development.

The deprecation policy for Spark 1.x versions will take effect on April 19, 2017 with the first set of versions being deprecated on July 31, 2017 — which will include: 1.3.0, 1.3.1, 1.4.0, 1.4.1, 1.5.0, 1.5.1, 1.6.0, and 1.6.1.

Our Commitment To Customers

Our change in policy is designed to bring you the best in technology as you’ve come to expect from Qubole. Our team remains committed to advancing the leading big data platform in the cloud that meets your needs and enables you to become a successful data-driven organization.

If you’re interested in trying out Qubole, sign up today for a free trial! To get an overview, click here to register for a live demo.

  • Blog Subscription

    Get the latest updates on all things big data.
  • Recent Posts

  • Categories

  • Events

    Building A Modern Data Lake On AWS

    Apr. 8, 2020 | Virtual Event

    Building A Modern Data Lake on AWS

    Apr. 16, 2020 | Virtual Event

    The Open Data Lake Talks: Optimizing Costs in A Changing World

    Apr. 22, 2020 | Virtual Event

    Building A Modern Data Lake On AWS

    Apr. 24, 2020 | Virtual Event

    Building A Modern Data Lake on Google Cloud

    Apr. 30, 2020 | Virtual Event

    Building A Modern Data Lake On AWS

    May. 8, 2020 | Virtual Event