Deployment

The demo stack in the repo root installs airflow-dq editable from the checkout — great for hacking on it, wrong shape for running it. This page describes the production-shaped setup: the released package baked into your Airflow image, a real warehouse split from the DQ store's credentials, and no repo mount anywhere.

A complete, runnable version of everything below ships in the repo as examples/pypi-deployment/ — a standalone folder (copy it out, docker compose up --build) with a realistic ingest → validate → publish DAG and its contract.

1. Bake the package into the image

Never install at container boot (_PIP_ADDITIONAL_REQUIREMENTS is demo-only). Pin the version:

FROM apache/airflow:3.1.0

# Pin the release. For production installs, prefer the Airflow constraints file
# matching your Airflow/Python version (see below).
RUN pip install --no-cache-dir airflow-dq==0.2.1

For constraint-respecting installs (recommended — this is what the project's CI does):

RUN pip install --no-cache-dir airflow-dq==0.2.1 \
    --constraint https://raw.githubusercontent.com/apache/airflow/constraints-3.1.0/constraints-3.12.txt

The wheel ships the built dashboard UI — no Node, no make ui-build, nothing to serve separately. Installing the package is all it takes for the plugin, the /dq API and the Data Quality nav item to appear.

2. Wire the topology

Three data stores, three roles — keep them distinct:

Store	Purpose	Config
Airflow metadata DB	Airflow's own	AIRFLOW__DATABASE__SQL_ALCHEMY_CONN — airflow-dq never touches it
DQ results store	the dq schema: results, quarantine, anomalies, registry	DQ_RESULTS_DSN — needs a role that can create a schema/tables
Warehouse(s)	the data being checked	Airflow Connection per backend, or DQ_DWH_DSN fallback — read-only role suffices

The results store can be its own Postgres or a database on your warehouse server — it just must not be Airflow's metadata DB. No manual schema bootstrap is needed: the migration runner creates the dq schema and all tables on first use and records state in dq.dq_schema_version; upgrades apply pending migrations automatically (see Configuration → Schema migrations).

Minimal compose sketch (the full example adds healthchecks, an env template and the demo DAG):

services:
  airflow-meta-db:
    image: postgres:16
    environment: {POSTGRES_USER: airflow, POSTGRES_PASSWORD: airflow, POSTGRES_DB: airflow}

  warehouse:
    image: postgres:16
    environment: {POSTGRES_USER: warehouse, POSTGRES_PASSWORD: warehouse, POSTGRES_DB: warehouse}

  airflow:
    build: .                      # the Dockerfile above — package baked in, no repo mount
    ports: ["8085:8080"]
    environment:
      AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@airflow-meta-db/airflow
      DQ_RESULTS_DSN: postgresql+psycopg2://warehouse:warehouse@warehouse:5432/warehouse
      DQ_DWH_DSN:     postgresql+psycopg2://warehouse:warehouse@warehouse:5432/warehouse
      DQ_CONTRACTS_DIR: /opt/airflow/contracts
    volumes:
      - ./dags:/opt/airflow/dags
      - ./contracts:/opt/airflow/contracts
    command: standalone

Two deployment-topology rules that bite when missed:

• DQ_CONTRACTS_DIR must be the same directory on the API server and every worker (shared volume, or bake the contracts into the image alongside your DAGs). The API writes contract versions there; workers read them at run time.
• Every DQ_* variable belongs in the environment of every Airflow component that runs airflow-dq code — API server and workers.

3. Production notes

• Constraints installs. Airflow's constraint files are the supported way to get a dependency-consistent environment; airflow-dq 0.2.1 is installable under Airflow 3.1's constraints (it runs on SQLAlchemy 1.4, which the constraints pin).
• psycopg2. The wheel depends on psycopg2-binary for friction-free installs. For production, follow the psycopg2 project's advice: pip install psycopg2 (source package) so it links your system libpq/libssl, and let it shadow the binary wheel.
• Read-only warehouse role. Checks, editor previews and profiling only ever read the warehouse. Give the warehouse Connection / DQ_DWH_DSN a read-only role — the expression rule's guard is defense-in-depth, not a sandbox. Only DQ_RESULTS_DSN needs write/DDL rights (in its database).
• Auth. Prefer auth-manager JWTs / session-cookie auth (works out of the box). If you set DQ_API_TOKEN for machine clients, treat it as a full-access credential and rotate it like one. See Configuration → Authentication.
• Retention. Results and quarantine grow with every run; set DQ_RESULTS_RETENTION_DAYS once the dashboard's useful history window is clear (0 = keep forever).
• Statement timeouts. DQ_STATEMENT_TIMEOUT_S (default 300s) caps every check/preview/profile query — lower it if your API server shares capacity with heavy interactive use.
• Experimental backends (Snowflake, BigQuery, Databricks, Trino): SQL generation is unit-tested per dialect, live connections are not. Install the matching extra (pip install "airflow-dq[snowflake]") and treat them as preview quality.
• Alert links. Set DQ_BASE_URL to your Airflow's public URL so alert deep-links are clickable.

4. The standalone example, end to end

# copy examples/pypi-deployment/ anywhere, then:
cd pypi-deployment
cp .env.example .env
docker compose up -d --build          # builds the image, installs airflow-dq==0.2.1 from PyPI
docker compose logs airflow | grep "Password for user 'admin'"

# Airflow UI:  http://localhost:8085           (login: admin / password from the logs)
# Health:      http://localhost:8085/dq/api/health
# Dashboard:   http://localhost:8085/dq/app/   (after a payments_pipeline run)

Unpause and trigger payments_pipeline — an ingest → validate → publish DAG that seeds a dirty payments table and validates it against payments@v1 with on_fail="quarantine". The run succeeds end-to-end while the bad rows land in quarantine; the example's README walks through what to inspect (dashboard, quarantine triage, the auto-created dq schema) and how to tear everything down without leftovers (docker compose down -v).