airflow-dq

An Airflow-3-native data-quality & data-contract plugin with an integrated quality dashboard.

You author ODCS data contracts inside the Airflow UI, they compile into real check-tasks in your DAG, and the results land in a built-in dashboard — no external SaaS backend, no separate platform.

pip install airflow-dq

Status: v0.2.1 — feature-complete beta. 327 unit tests, end-to-end integration test in CI, built UI ships in the wheel. Not yet battle-tested in a multi-team production deployment.

Licensed under Apache-2.0. airflow-dq is an independent community project — it is not endorsed by or affiliated with the Apache Software Foundation or the Apache Airflow project (portions of the UI are derived from Apache Airflow, see NOTICE).

Why another DQ tool

Data-quality tooling (Soda, Great Expectations, Elementary, Monte Carlo) is mostly checks-as-code in YAML files next to the pipeline, often backed by external SaaS. None of them give you contract-native DQ that lives entirely inside the orchestrator. The wedge, concretely:

• In-Airflow authoring — a no-code contract editor as an Airflow UI view: introspect live columns, pick rules, test them against the real table before saving, or import your existing dbt schema.yml. Contracts are versioned, immutable ODCS v3.1 YAML.
• Gate / monitor duality — the same contract runs as an inline blocking gate in the producer DAG (on_fail="fail"), or as a record-only, Asset-triggered monitor that audits every table update without touching producer code (on_fail="warn"). Both feed one dashboard.
• Quarantine — the middle ground: on_fail="quarantine" copies failing rows aside for triage (browse, inspect, bulk-resolve in the UI) while the pipeline keeps moving.
• Breaking-change detection — every contract save is diffed against the previous version and classified (column removed / type changed / new required / uniqueness added = breaking); with the git flow enabled, "propose" turns a risky save into a reviewable PR branch.

The integration, whole

from airflow_dq import build_contract_checks

validate_silver = build_contract_checks(
    contract="orders@v3",        # pinned, immutable version
    backend="postgres_dwh",      # Airflow conn_id (env-DSN fallback)
    on_fail="quarantine",        # "fail" | "warn" | "quarantine"
)
bronze >> silver >> validate_silver >> gold

Each rule becomes its own mapped Airflow task (Dynamic Task Mapping), executed as push-down SQL against your warehouse — count, bad-row sample and quarantine all derive from one failing predicate, so they can never disagree.

Feature highlights

• 12 rule types — notNull, unique, valueRange, acceptedValues, pattern (regex), referentialIntegrity, freshness (max-age or row-age, day/hour units), rowCount, schema drift, custom SQL expression, z-score anomaly, cross-table reconciliation. Full semantics in the rules reference.
• ODCS v3.1 SLA checks — slaProperties (latency, frequency, retention) compile into sla.* checks with the same gate/monitor semantics.
• Anomaly detection, two levels — row-level z-score outliers (with baselineWhere to keep the baseline honest) plus server-side metric anomalies flagged on the trend chart.
• 6 warehouse backends — Postgres and DuckDB (stable), Snowflake, BigQuery, Databricks and Trino (experimental). Backends resolve as Airflow Connections with env-DSN fallbacks.
• Alerting with fatigue controls — one grouped alert per (run, contract), retry-safe; channels: Slack-compatible webhook, email, PagerDuty, Opsgenie; per-owner and per-severity routing.
• Auth via the Airflow auth manager — API routes validate Airflow-issued JWTs (header or session cookie); a legacy shared secret remains supported for machine clients.
• Operability — versioned SQL migrations applied automatically, error status for infra failures, idempotent results/quarantine/alert writes, OpenLineage facets per check.

Where to go next

• Getting started — demo stack, first contract, first gate in ~10 minutes.
• Examples — the 15-minute guided tour through five runnable example DAGs.
• Rules reference — every rule type: what it checks, params, generated SQL, gotchas.
• Configuration — every DQ_* knob, connection resolution, auth, alerting.
• Deployment — running the released package in a production-shaped stack.
• REST API — the plugin's JSON API.
• Architecture — how the pieces fit, and the design decisions behind them.