Skip to content

Getting started

The fastest way to see airflow-dq working is the bundled demo stack: Airflow 3.1 standalone plus a seeded Postgres that doubles as demo warehouse and DQ results store. You need Docker and ~10 minutes. (To run the released PyPI package against a stack you own instead, see Deployment.)

1. Start the demo stack

git clone https://github.com/fbnschneider/airflow-dq.git
cd airflow-dq
cp .env.example .env              # defaults are fine for the demo
docker compose up -d              # Airflow 3.1 standalone + seeded Postgres
make ui-build                     # build the React views (served by the plugin)
docker compose logs airflow | grep "Password for user 'admin'"   # login credentials

Then open:

URL What
http://localhost:8080 Airflow UI (log in as admin with the password from the logs)
http://localhost:8080/dq/app/ The Quality Dashboard (also embedded as Data Quality in the Airflow nav)
http://localhost:8080/dq/api/health Plugin health probe (unauthenticated)

The stack seeds a deliberately dirty orders table and auto-loads five example DAGs (paused, tagged examples). Useful make targets: make test (unit tests), make integration (full compose-up → DAG run → results assertion → teardown), make reseed (refresh the dirty demo data so demo outcomes stop drifting with volume age).

2. Run your first gate

Unpause example_01_quickstart_gate in the Airflow UI and trigger it. The run fails on purpose: the seed is dirty, and one blocking check (orders.amount.valueRange) breaches its fail threshold, so the downstream publish task never runs. That's the gate doing its job.

Expand the dq_check__orders_v3 task in the grid — one mapped task instance per check — then open the dashboard and expand the failing check to see the actual offending rows.

The entire integration behind it is:

from airflow_dq import build_contract_checks

gate = build_contract_checks(
    contract="orders@v3",       # pinned, immutable contract version
    backend="postgres_dwh",     # Airflow conn_id; falls back to DQ_DWH_DSN
    on_fail="fail",             # gate semantics: blocking failures stop the pipeline
)
bronze >> gate >> publish

Three things worth internalizing right away:

  • Only blocking rules can gate. Rules with severity warning reach at most status warn, no matter how red they are. See the severity × on_fail matrix.
  • Tolerances grade the status. A blocking rule with warnIfPercentGreaterThan / failIfPercentGreaterThan thresholds only trips the gate when the failing share crosses the fail threshold.
  • Version pins are immutable. orders@v3 can never silently change under you; a stricter v4 requires you to bump the pin deliberately.

3. Author your first contract

Open Data Quality → Contract Editor:

  1. Type a table name (try payments) and click Load columns — live introspection fills the column dropdowns.
  2. Click ✨ Profile & suggest — the profiler drafts checks from the live data (fully-populated → notNull, distinct==rows → unique, numeric ranges, low-cardinality accepted values).
  3. Adjust the blocks, then hit Test on any block — the check runs against the real table and shows pass/fail plus offending rows before you save anything.
  4. Set an id and version, then Save contract.

Saving diffs against the previous version and classifies the change (red = breaking). Versions are immutable — saving over an existing version returns HTTP 409, and the editor auto-bumps to the next free number.

Prefer starting from what you already have? Paste a dbt schema.yml into the editor's Import from dbt panel (or POST /dq/api/contracts/import/dbt) to get a contract draft from your existing not_null / unique / accepted_values / relationships tests.

4. Wire it to your own table

  1. Create an Airflow Connection for your warehouse (e.g. conn_id postgres_dwh, or any id — the backend argument is the conn_id). Without a matching Connection, postgres* backends fall back to the DQ_DWH_DSN env var. See Configuration.
  2. Author (or hand-write) a contract for the table and note its ref, e.g. my_table@v1.
  3. Drop the factory call between your producer task and its consumers:
validate = build_contract_checks("my_table@v1", backend="postgres_dwh", on_fail="quarantine")
ingest >> validate >> publish

on_fail="quarantine" is the pragmatic default for adoption: bad rows are copied into dq.dq_quarantine for triage in the UI while the pipeline continues.

Next steps

  • The examples tour walks all five example DAGs — quarantine triage, incremental/partition-scoped checks, SLAs + anomaly detection, and asset-triggered monitoring — in about 15 minutes.
  • The rules reference documents every rule's exact semantics and gotchas.
  • Deployment shows a production-shaped stack installing airflow-dq from PyPI.