Declarative Pipelines & Intelligent Orchestration

Dive into the cutting-edge world of data pipelines with Sean Knapp, co-founder and CEO of, as he unveils the design and creation of intelligent orchestration for data pipelines. This session, part of the Qubole Data Lake Summit 2020, offers deep insights into making data pipelines smarter, more efficient, and capable of handling the complexities of modern data engineering with minimal code.

What You’ll Learn:

  1. Introduction to Intelligent Orchestration:
    • Understand the evolution of data pipeline orchestration and the shift towards declarative engineering to minimize coding effort while maximizing efficiency.
  2. The Foundation of Smart Data Pipelines:
    • Discover how tracking every aspect of data, from its origin to transformations, enables smarter pipeline decisions and optimizations.
  3. Autonomy in Orchestration:
    • Learn how an autonomous control system can adapt to code and data changes, ensuring your data pipelines are always up-to-date without manual intervention.
  4. Efficient Change Detection and Scalability:
    • Explore techniques like SHA calculations and Merkle trees that enable efficient change detection and scalability, handling trillions of records and billions of partitions with ease.
  5. Self-Optimizing Pipelines:
    • Delve into how intelligent persistence and metadata tracking can lead to self-optimizing data pipelines, reducing the need for manual tuning and optimization.
  6. Practical Benefits and Applications:
    • Gain insights into the practical benefits of declarative pipeline orchestration, including zero-operation pipeline copies, data deduplication, and seamless dev-stage-prod pipeline transitions.

Please fill in the form to watch the webinar

Note: By filling and submitting this form you understand and agree that the use of Qubole’s website is subject to the General Website Terms of Use. Additional details regarding Qubole’s collection and use of your personal information, including information about access, retention, rectification, deletion, security, cross-border transfers and other topics, is available in the Privacy Policy. If you have any questions regarding the webform language, please contact [email protected].