Apache Airflow

Apache Airflow is an open-source tool to programmatically author, schedule, and monitor data workflows. With Airflow, users can author workflows as Directed Acyclic Graphs (DAGs) of tasks. A DAG is the set of tasks needed to complete a pipeline organized to reflect their relationships and inter-dependencies. Qubole provides Apache Airflow as a managed service that is easy to manage and scalable for your data engineering needs.

Apache Air Flow

Airflow on Qubole

Apache Airflow on Qubole improves performance, lowers cost, improves user experience, and enables smooth administration of Airflow clusters. With Apache Airflow on Qubole, data engineers can setup airflow clusters in seconds, use a single panel to track usage, manage clusters and ACLs, easily edit DAGs and plugins and leverage the integrated “Goto QDS” button on Qubole Operator to seamlessly navigate to the corresponding Qubole command page.


Managing Airflow Resources

Reduce the complexity of managing Airflow resources while at the same time allowing you fine-grained controls on cluster type and size, number of executors, airflow version, security, and RBAC policies

Airflow Monitoring and Alerting

Lower downtime and improve business continuity with integrated monitoring service to monitor and automatically start all relevant airflow services. Prometheus with Grafana for better monitoring of resources and alerting to endpoints such as email, slack, pager-duty, etc.

Airflow on Cloud

Future-proof your solution with Airflow on Qubole. It is cloud-agnostic and generally available on AWS, Azure, and Google Cloud Platform.

Airflow vs Apache Airflow 

Airflow Management

Airflow on QuboleApache Airflow
Easy deployment of dags with dag explorer
Auto sync dags from cloud storage
Git integration
Automated log backup and cleanup
Airflow on Anaconda Virtual Environment

Airflow Monitoring

Airflow on QuboleApache Airflow
Integrated Graffana & Prometheus based alerts and monitoring
Integrated monitoring service for improved availability
Monit dashboard for better visibility of service health
Monitoring (Ganglia, DataDog, etc)

Airflow Security

Airflow on QuboleApache Airflow
Integrated RBAC for Airflow clusters
SSO with SAML 2.0 support
Data encryption (at rest and in motion)
Audit end-user activity logs
HIPAA, SOC2 Type2, ISO-27001 compliant environments

Support and Services

Airflow on QuboleApache Airflow
24/7 support from our Airflow experts
Support multiple versions of Airflow