Note

This page includes the repository's DESIGN.md verbatim — the decision log behind the architecture overview.

DESIGN.md — Decisions & Tradeoffs¶

This document captures the why behind the architecture. It is intentionally the senior-signal artifact: "I know the alternatives and I made a deliberate choice."

D1 — Why Dynamic Task Mapping instead of one opaque check task?¶

Each check becomes its own visible task in the DAG graph via .expand(), rather than a black-box "validate" task. Reviewers and on-call engineers see exactly which check failed in the Airflow grid, get per-check retries, and get per-check logs/lineage.

Tradeoff: more tasks in the graph and a small mapping overhead. Acceptable — observability wins.

D2 — Why a pluggable `CheckExecutor` (Protocol) instead of one SQL engine?¶

We do not build a SQL engine. The executor is a Protocol with adapters that generate SQL or delegate to a backend:

DuckDBExecutor / PostgresExecutor — generate SQL, run everywhere (demo + CI).
DatabricksExecutor (stretch) — Databricks SQL + Delta, optionally via DQX (enterprise flex).

This single abstraction is the core design decision: "runs locally on DuckDB for reviewers, scales to Databricks/Delta for the real world."

Tradeoff: dialect differences leak into adapters. Contained by keeping CheckSpec backend-agnostic and pushing dialect into the adapter only.

D3 — Why ODCS instead of a home-grown contract format?¶

ODCS (Open Data Contract Standard) is an existing, governed standard. Inventing a format would be undifferentiated work and a portability dead-end. Using ODCS plays to real-world experience.

D4 — Why a separate metadata DB schema, not Airflow's metadata DB?¶

Contract registry, versions and check-run history live in their own Postgres schema — never pollute Airflow's internal metadata DB. Keeps upgrades safe and the product portable across Airflow deployments.

D5 — Why not just use Soda / Great Expectations?¶

They are checks-as-code in YAML next to the pipeline, frequently backed by external SaaS. The wedge here is visual authoring inside the orchestrator + codegen to real tasks + an integrated dashboard, fully self-hosted OSS. The breaking-change detector ("git for data") is specifically the part those tools don't cover as Airflow-native OSS — the differentiator.

D6 — Failure semantics: `fail` | `warn` | `quarantine`¶

fail — stops the critical pipeline (check task fails, downstream blocked).
warn — pipeline continues, result recorded as warn.
quarantine — copies failing rows (capped) into dq.dq_quarantine for triage; pipeline continues.

D8 — Tolerances over binary pass/fail (anti alert-fatigue)¶

Real data is never 100% clean; a check that goes red on a single null gets disabled. A rule may carry warn/failIfGreaterThan (absolute) or warn/failIfPercentGreaterThan (share) thresholds. With thresholds, status is graded (pass → warn → fail); without them it falls back to binary-by- severity. Example: amount.valueRange is blocking but tolerated, so 2/7 bad (29%) only warns, while freshness (no tolerance) hard-fails. The single biggest lever against the #1 reason DQ tools get switched off.

D9 — Schema-drift is one metadata check, not per-column rules¶

Upstream renaming a column / changing a type is the silent killer. One schema check per dataset compares information_schema to the contract: missing required columns + coarse type-family mismatch (string/number/temporal/boolean), so decimal vs numeric doesn't false-positive while string↔number drift is caught. Also seeds the breaking-change story (stretch).

D10 — Failures must be actionable¶

Every check captures a small sample of the offending rows inline (one query, same predicate as the count, so they can't disagree). The dashboard shows it in an expandable "bad rows" panel — actionable triage, not just a red count. Quarantine persists the rows; it does not rewrite the user's downstream table (owning that write is out of scope).

D11 — Low-code authoring: form library + one expression escape hatch + live preview¶

The editor is no-code for the closed rule library (pick column + rule + params; columns are introspected from the live table) and low-code via a single expression rule — a boolean SQL predicate that flags bad rows. Crucially, that predicate plugs straight into failing_predicate, so count, bad-row sample and quarantine come for free; no parallel code path. We deliberately do not build a drag-drop block engine (the form already is no-code) nor accept arbitrary full SELECTs (breaks portability + safety). The differentiator is live preview: every block has a "Test" button that runs against the real table and shows pass/fail + sample before saving — interactive authoring no other in-Airflow OSS does. Expressions are guarded (identifiers only, read-only predicate, no DDL/DML); trust model = data engineers, same as dbt tests.

D12 — One contract, two deployment patterns (inline gate + asset-triggered monitor)¶

The unit is the contract-per-dataset, not the DAG. The same contract is consumed two ways: (A) inline in the producer DAG as a gate (on_fail="quarantine", blocks/quarantines before data flows on); and (C) an asset-triggered monitor DAG (dq_monitor) that schedule=[asset] and runs whenever the table is reproduced (on_fail="warn", record-only). The producer marks the table updated via outlets=[orders_asset]. This uses Airflow 3 data-aware scheduling so DQ coverage doesn't require wiring checks into every pipeline — and it makes "gate + observability from one definition" literal: both feed the same results store / dashboard. (A central scheduled "sweep" DAG is the third option, better when there's no producer to hang the asset off.)

D13 — Authoring intelligence: profile to draft, diff to warn¶

Two features make the contract editor more than a form. Profiling points at a live table and suggests a starter draft (fully-populated → notNull, distinct==rows → unique, numeric → valueRange from observed min/max, low-cardinality string → acceptedValues) — you refine from a draft, not a blank page. Breaking-change detection diffs the new version against the previous one on save and classifies each change (column removed / type changed / new required / uniqueness added = breaking; optional add / relaxed constraint / rule change = info), surfacing a red warning in the editor. The detector reuses the schema type-family logic (D9); it's the "git for data" wedge that Soda/GE don't ship as Airflow-native OSS.

D14 — Anomaly detection: statistical, push-down, two levels¶

Anomaly detection here is unsupervised and statistical (z-score / baseline), not deep learning — the same family as Databricks Lakehouse Monitoring, and honest about it. Two levels: (1) row-level — the anomaly rule flags rows beyond k·σ of the column mean via pure SQL (ABS(x - (SELECT avg(x))) > k * (SELECT stddev_samp(x))), so it scales (no data pulled into Python) and reuses the sample/quarantine machinery to surface the weird records; (2) metric- level — the dashboard's "quality over time" chart flags a run whose failing-row count exceeds the historical mean + 2σ. A heavier model (e.g. IsolationForest) is a drop-in extension behind the same rule, with the explicit trade-off that it pulls a sample into Python.

D15 — Incremental by scope, backfill-safe by logical date¶

Real tables are too big for full scans on every run, and backfills must not false-alarm. Two mechanisms: (1) every row-level check accepts a guarded where scope (partition filter) that count, sample and quarantine all share — check only the new slice, compare anomaly baselines against full history; (2) freshness evaluates against the run's logical_date (injected by the operator as asOfDate), so a backfill for last month doesn't flag last month's data as stale.

The check operator implements get_openlineage_facets_on_complete and emits each result as DataQualityAssertionsDatasetFacet (+ DataQualityMetricsInputDatasetFacet) on the checked dataset — lineage UIs (Marquez etc.) then show "the pipeline ran AND the dataset met its expectations". The mapping core is pure Python (unit-testable without Airflow); the client conversion uses defensive imports and returns None on any failure — lineage must never break a pipeline. Dialect note: SQL generation now takes a dialect parameter (regex is the only split so far), and all free-form predicates/identifiers/numeric params are guarded at SQL-build time (defense in depth beyond the authoring-side validation).

D7 — Auth is explicit, on purpose¶

Airflow 3 FastAPI plugin endpoints are not auto-protected by Airflow auth. We secure them explicitly (token validated against Airflow's auth manager). A consciously handled auth concern is itself a good signal.

Open questions / parked¶

Exact ODCS v3 rule coverage in the MVP form (~6 most common check types).
OpenLineage facet shape for check results (stretch).
Whether the dashboard also feeds an external BI tool (optional, not the centerpiece).