Achieving Operational Excellence for data engineering - Shantanu Shirvastarva, Zeotap

November 25, 2020

Zeotap saw a 10x growth with the number of data pipelines and amount of data processed growing at rapid pace in a short span of time. The increasing scale challenged our capability to track production data pipelines across multiple products and infrastructure deployments. To tackle this problem, we came up with Kingpin - a Centralized Job Management Framework. Kingpin addresses some of the common issues faced with data pipeline operations: Workflow Dependency management, Multi-Framework execution support Fault Tolerance, and Alerting mechanisms, Scheduled and On-Demand executions Workflow tracking across multiple deployments. Additionally, we have integrated Kingpin with Data Visualization Tools for a Unified Job Level View with Metric Reporting capabilities to increase system visibility and reduce the operational time, efforts, and costs.

Previous Video
Domain-driven Data Architecture | Qubole
Domain-driven Data Architecture | Qubole

Next Video
Spark optimization with Sparklens - Rohit Karlupia, Qubole
Spark optimization with Sparklens - Rohit Karlupia, Qubole

Debugging slow spark applications when done with trial and error, takes lots of time. Sparklens provides in...