The last decade has brought significant advancements and innovations across the data management landscape, spanning large-scale processing engines, cloud-based data warehouses, next-generation data lakes, and unprecedented machine learning tools. But why are these powerful systems being controlled and orchestrated by the most rudimentary technologies? Data movement and transformation are often still dictated by manual, hard-coded triggers and rules, written in slow development cycles and resulting in brittle pipelines. As the number of pipelines and dependencies scale, data engineering teams become bogged down by constant maintenance, combing through code and logs to find hotspots, and constantly tuning infrastructure just to keep things running. This “state of the art” points to a gap in the modern data ecosystem, opening the door to more intelligent forms of data orchestration. As was done with Kubernetes, Terraform, and even React, we can apply a declarative approach to the domain of data pipelines, radically advancing the automation level that can be achieved. In this session, Sean Knapp, CEO and founder of Ascend.io, discusses this journey, including:
Free access to Qubole for 30 days to build data pipelines, bring machine learning to production, and analyze any data type from any data source.