Architecture

airflow-dq is deliberately not a platform: it's a Python package + an Airflow plugin. Contracts are YAML files, checks are ordinary Airflow tasks, results live in one Postgres schema, and the dashboard is a React app served by the plugin. This page is the readable overview; the full decision log (D1–D16, with tradeoffs) lives in Design decisions.

Data flow

flowchart TD
    A[Contract Editor<br/>React view in Airflow UI] -->|409-safe save + diff| B[Contract Store<br/>immutable ODCS-YAML versions + registry DB]
    B -->|version diff| C[Breaking-Change Detector]
    B -->|propose=true| P[Git branch<br/>PR approval flow]
    D[Your DAG] -->|build_contract_checks| E[Contract Compiler<br/>rules + SLA properties]
    B --> E
    E -->|Dynamic Task Mapping| F[One check task per rule]
    F --> G[CheckExecutor<br/>Airflow Connection or DSN]
    G -->|Postgres / DuckDB / Snowflake* / BigQuery* / Databricks* / Trino*| H[(Warehouse)]
    G -->|idempotent write| I[(Results Store<br/>dq schema + migrations)]
    G -->|best-effort| J[OpenLineage facets]
    G -->|grouped, deduped| K[Alerts<br/>webhook / email / PagerDuty / Opsgenie]
    I --> L[FastAPI plugin /dq<br/>auth: Airflow JWT or legacy token]
    L --> M[Overview · Dashboard · Quarantine Triage · Editor<br/>React views in Airflow UI]

* experimental backends

In one sentence: author a contract → it is stored as an immutable version → a DAG calls build_contract_checks() → the compiler turns rules + SLAs into mapped check-tasks → the executor runs push-down SQL against the backend → results (plus samples/quarantine/anomalies) land idempotently in the results store and, on failure, one grouped alert goes to the owner's channel → the embedded dashboard reads the store.

The pieces

Piece	Where	What it does
Contract store	contracts/	One YAML file per immutable version (<id>/<id>.v<N>.yaml), refs like payments@v1; server-side numbering, 409 on conflict, atomic writes; mirrored into a registry DB for the version timeline; optional git "propose" flow.
Compiler	compiler/	Contract → flat list of CheckSpecs: one schema-drift check per dataset, schema-derived notNull/unique, explicit rules, sla.* checks. Pure Python, no Airflow dependency.
Factory & operator	operators/	build_contract_checks() expands one DataContractCheckOperator per spec via Dynamic Task Mapping; owns on_fail policy, where-templating, asOfDate injection, results write, quarantine, alerting, lineage.
Executors	executors/	Pluggable CheckExecutor protocol; shared SQL codegen with a per-engine dialect layer; Airflow-Connection resolution with env-DSN fallbacks; pooled engines + statement timeouts.
Results store	results/	The dq Postgres schema (results, quarantine, metric anomalies, alert groups, contract registry), auto-migrated; idempotent upserts keyed by (dag_id, run_id, task_id, check_name).
Alerting	alerting/	Group-claim dispatch: one alert per (run, contract), severity-aware owner routing, four channel types, all best-effort.
Plugin	plugin/	FastAPI app at /dq (JSON API + served React bundle) registered via the Airflow plugin entry point; auth against the Airflow auth manager.
UI	ui/	React + TypeScript views (Overview, Dashboard, Quarantine, Editor) on the Airflow-native design system, embedded via external_views; ships pre-built in the wheel.

The design decisions, condensed

The full rationale per decision is in Design decisions (D1–D16); the short version:

#	Decision
D1	One mapped task per check (Dynamic Task Mapping) — per-check visibility, logs, retries in the Airflow grid, instead of one opaque "validate" task.
D2	Pluggable CheckExecutor protocol, not a SQL engine — adapters generate SQL per backend; dialect differences are contained in a dialect layer.
D3	ODCS rather than a home-grown contract format.
D4	Own dq schema in its own Postgres — never Airflow's metadata DB.
D5	The wedge vs Soda/GE: visual authoring inside the orchestrator + codegen to real tasks + integrated dashboard, self-hosted OSS.
D6	Failure policies fail / warn / quarantine per deployment of a contract.
D7	Auth is explicit — Airflow 3 doesn't protect plugin routes, so airflow-dq does (auth-manager JWT + legacy token).
D8	Tolerances over binary pass/fail — graded warn/fail thresholds against alert fatigue.
D9	Schema drift as one metadata check with coarse type families (no false positives on decimal vs numeric).
D10	Failures are actionable — every failing check carries a bad-row sample from the same predicate as the count.
D11	Low-code authoring + one expression escape hatch + live preview — the Test button runs the real check before saving.
D12	Gate + monitor duality from one contract — inline blocking gate and Asset-triggered record-only monitor.
D13	Profile-to-draft + breaking-change detection — start from a suggested contract; saves are diffed and classified.
D14	Anomaly detection: statistical, push-down, two levels — row-level z-score and server-side per-run metric anomalies.
D15	Incremental by where scope, backfill-safe by logical date — templated partition scoping; freshness measured as-of the run.
D16	OpenLineage, never in the critical path — DQ assertion facets, best-effort, spec-compliant dataset naming.

Runtime behavior worth knowing

• The results write is the only near-critical-path sink, and even it is best-effort-with-retry (backoff, never raises). Quarantine, alerting, lineage and metric-anomaly detection are strictly best-effort — an outage there never fails your pipeline.
• Infra failures are error, not mystery-red: executor exceptions become status='error' results (recorded + alertable); only on_fail="fail" tasks re-raise them.
• Everything is idempotent per run: results upsert on the natural key, quarantine replaces-on-retry, alerts claim a group key — Airflow retries and clears don't double-count.
• Two connections per check at most: the warehouse (read-only work) and the results store (writes). Quarantine reads bad rows from the former and writes them to the latter, so the warehouse needs no DQ tables.