Zeotap saw a 10x growth with the number of data pipelines and amount of data processed growing at rapid pace in a short span of time. The increasing scale challenged our capability to track production data pipelines across multiple products and infrastructure deployments. To tackle this problem, we came up with Kingpin – a Centralized Job Management Framework. Kingpin addresses some of the common issues faced with data pipeline operations: Workflow Dependency management, Multi-Framework execution support Fault Tolerance, and Alerting mechanisms, Scheduled and On-Demand executions Workflow tracking across multiple deployments. Additionally, we have integrated Kingpin with Data Visualization Tools for a Unified Job Level View with Metric Reporting capabilities to increase system visibility and reduce the operational time, efforts, and costs.
Free access to Qubole for 30 days to build data pipelines, bring machine learning to production, and analyze any data type from any data source.