Declarative Pipelines & Intelligent Orchestration - Data’s Missing Link - Sean Knapp, Ascend.io

November 25, 2020

The last decade has brought significant advancements and innovations across the data management landscape, spanning large-scale processing engines, cloud-based data warehouses, next-generation data lakes, and unprecedented machine learning tools. But why are these powerful systems being controlled and orchestrated by the most rudimentary technologies? Data movement and transformation are often still dictated by manual, hard-coded triggers and rules, written in slow development cycles and resulting in brittle pipelines. As the number of pipelines and dependencies scale, data engineering teams become bogged down by constant maintenance, combing through code and logs to find hotspots, and constantly tuning infrastructure just to keep things running. This “state of the art” points to a gap in the modern data ecosystem, opening the door to more intelligent forms of data orchestration. As was done with Kubernetes, Terraform, and even React, we can apply a declarative approach to the domain of data pipelines, radically advancing the automation level that can be achieved. In this session, Sean Knapp, CEO and founder of Ascend.io, discusses this journey, including: * Imperative vs Declarative systems * Evolution of orchestration tools * Tradeoffs and design decisions in moving from task-centric to data-centric architectures * Challenges in running a multi-cloud, massive-scale, elastic infrastructure built on Kubernetes and Apache Spark.

Previous Video
Your Data Lake is Moving to the Cloud - What About Your Security Policies? -  Privacera
Your Data Lake is Moving to the Cloud - What About Your Security Policies? - Privacera

Presented by Srikanth Venkat, Privacera. Enterprises migrating to the cloud for increased agility and elas...

Next Video
Apache CarbonData: Data Storage for ACID Ingest, Fast Query, and Machine Learning - Huawei
Apache CarbonData: Data Storage for ACID Ingest, Fast Query, and Machine Learning - Huawei

The growing volume of data requires skills to deal with dozens of new challenges like how to ingest streami...