airflow-dq — Research-Backed Refactor Backlog & Feature Roadmap¶
Date: 2026-07-11 · Baseline: FEATURE-INVENTORY.md (current-state ground truth) · Method: deep-research workflow (5 search angles, 24 sources, 117 extracted claims, 3-vote adversarial verification) + direct primary-source verification of the claims the workflow left unverified.
Evidence labels used throughout: - [V] verified against a primary source (spec/official docs) — highest confidence - [W] workflow-verified 3-0 (or 2-1 where noted) - [B] blog/practitioner-sourced, plausible but unverified — medium confidence - [I] established by the internal feature inventory (code-level fact)
1. Strategic picture (what the research established)¶
- The Airflow-native niche is genuinely open. Elementary is exclusively dbt-native — no orchestrator integration in its positioning at all [W]. Soda and GX are engine-first tools where Airflow is just an operator wrapper; the
common-sqlprovider offers raw check operators with no contracts, UI, or policies. Nobody contests in-orchestrator authoring + gate/monitor duality + breaking-change detection — the wedge is real. - Contract→checks compilation is commodity. datacontract-cli is ODCS-native and runs schema/quality checks against 18–24 data sources already [V]. The compilation step itself differentiates nothing; the differentiated part is how checks live inside Airflow (mapped tasks, gate policies, asset-triggered monitors, quarantine).
- OSS anomaly detection is a paywall gap at the incumbents. Soda gates anomaly detection behind paid plans (RAD: "available soon for Enterprise plan users") [V]; GX OSS has no automatic anomaly detection or native freshness monitoring [B, two independent sources]. Shipping usable, transparent anomaly detection free-and-OSS is a verified differentiation window.
- ODCS v3.1.0 changes the compliance calculus. The spec now rejects fields outside the schema ("Only fields defined in the JSON Schema are allowed"; extensions belong in
customProperties) [V], defines exactly four quality types (text / library / sql / custom, withcustomexplicitly slotting Soda/GX/dbt/dbx/Montecarlo engines) [V], and adds executable SLAs (RFC-0025: latency, retention, availability, frequency, completeness, with cron scheduling semantics) [V]. The current "ODCS-dialect" is therefore a growing liability (fails v3.1 validation, silently strips compliant contracts [I]) — but the SLA addition means an SLA feature would be standards-aligned, not proprietary. - Airflow 3 prescribes the auth path.
fastapi_appsplugin endpoints ship with no auth from Airflow [W] — but Airflow 3 centralizes auth in exactly one pluggable auth manager [W], which issues the JWTs used for the public API [W 2-1], andBaseAuthManagerexposesis_authorized_view/is_authorized_*hooks so plugins can delegate authorization instead of building RBAC [W]. The official Edge3 provider shows the accepted floor (single shared secret, "multi-team isolation is logical only" [W]) and the precedent that a provider may own auto-created/migrated tables in the metadata DB [W 2-1]. - OpenLineage naming is mandatory for stitchable lineage. Dataset namespaces must derive from the datasource, not the orchestrator [W], with exact per-source formats (
postgres://{host}:{port}+{database}.{schema}.{table},snowflake://{org}-{account},bigquery,trino://{host}:{port}…) [V], and the Airflow OL provider explicitly requires OL-compliant dataset names [V]. airflow-dq's bareorders+ custom namespace means its DQ facets can never attach to what other integrations emit [I] — the current lineage output is effectively an island. - Alerting exists but fatigue controls decide adoption. None of GX/dbt tests/Soda Core ship default alerting in OSS [B]; practitioners consistently cite tiered severity, grouping, and dedup as the difference between used and disabled alerting [B]. airflow-dq already has webhook alerting + owner routing — the missing half is exactly the fatigue controls (per-check-run POSTs, gate+monitor double-alerts, retry re-fires [I]).
- One refuted framing: the claim that Elementary paywalls column-level lineage/ML monitoring as the differentiation-bearing features was killed in verification (1-2). Column-level lineage should not anchor the roadmap.
2. Prioritized refactor backlog¶
Priorities weigh production-blocking severity [I] × ecosystem-validated best practice × unblocking value for roadmap features. Effort: S (<1 wk), M (1–3 wk), L (>3 wk) single-engineer classes.
P0 — production-blocking (do before any feature work)¶
| # | Refactor | Why / evidence | Effort |
|---|---|---|---|
| R1 | Auth: Airflow auth-manager JWT validation; kill the token-in-HTML path. Validate auth-manager-issued JWTs on all /dq/api/* routes; delegate viewer-vs-author decisions via is_authorized_view/custom-resource hooks; remove DQ_API_TOKEN injection from serve_ui (the UI, being same-origin, forwards the Airflow session/JWT instead). Shared-secret can remain a documented fallback for machine clients (Edge3 precedent). |
Inventory: any visitor of /dq/app/ gets full author rights [I]. Airflow 3: one pluggable auth manager [W], JWT issuer [W 2-1], is_authorized_view hooks [W], fastapi_apps unprotected by design [W], Edge3 shared-secret floor [W]. |
M |
| R2 | Reliability: best-effort results write + real error status + idempotency keys. Wrap the results write (buffer/retry, never fail the check task on store outage); convert executor exceptions into status='error' CheckResults (the dead schema value [I]); add a unique key (dag_id, run_id, task_id, map_index, check_name) with upsert semantics on results and quarantine; make alert firing idempotent per run. |
Inventory §3.2–3.3: store outage fails all pipelines; retries double-count results/quarantine/alerts [I]. Retry-safety (idempotent tasks) is core Airflow guidance; every sink here violates it [I]. | M |
| R3 | Connections & secrets: backend → real Airflow conn_id. Resolve executors via Airflow Connections/secrets backend (worker-side), keep env-DSN fallback for bare tests; module-level pooled engines with dispose() lifecycle; statement timeouts on every check/preview/profile query. |
Inventory: prefix-string→env-var aliasing = one warehouse per deployment, plaintext creds, engine-per-call connection storms, unbounded queries [I]. Airflow Connections are the ecosystem norm the API already pretends to follow ("connection alias" docstring) [I]. | M |
| R4 | Contract integrity: immutable versions + atomic writes + registry sync. Server-side version numbering (reject overwrite of an existing version), temp-file+rename writes, write the computed diff + checksum into dq_contracts/dq_contract_versions (today dead tables [I]) with Alembic-managed migrations. |
The entire ref-pinning guarantee (orders@v3 in DAG code) is currently unenforced [I]; Edge3 shows provider-owned, auto-migrated tables are accepted practice [W 2-1]. Unblocks F5 (approval flow) and the version-timeline UI. |
M–L |
P1 — correctness & interoperability¶
| # | Refactor | Why / evidence | Effort |
|---|---|---|---|
| R5 | ODCS v3.1 compliance. Move dialect fields (dataset, owner, tolerances-in-params…) into customProperties/sanctioned locations; adopt the four quality types (map rule library → library, expression → sql, keep custom passthrough); semver-string versions; stop silently stripping unknown fields (round-trip preservation). Target: datacontract-cli can lint an airflow-dq contract. |
v3.1 strict validation rejects dialect fields [V]; ODCS is the converging interchange format (datacontract-cli native [V]); silent stripping is data loss [I]. | L |
| R6 | OpenLineage naming compliance. Dataset namespace derived from the executor's connection (postgres://{host}:{port}, later snowflake://{org}-{account} …), name {database}.{schema}.{table}; reconcile the Asset URI scheme with the same identity; log (don't just swallow) conversion failures. |
OL spec naming [V], provider requirement [V], namespace-from-datasource rule [W]; current facets can't stitch [I]. Small change, large interop payoff. | S–M |
| R7 | Quarantine decoupling. Write quarantine rows over the results-store connection (read failing rows from warehouse, insert into metadata DB) or a configurable target; dedup by run key (R2); TTL/retention policy; record result_id linkage. |
Current INSERT..SELECT silently requires warehouse == metadata DB — false in any real deployment [I]. Prereq for F3 (triage UI). | M |
| R8 | Incremental & freshness semantics. Template where scopes with the run's data interval ({{ ds }}-class variables resolved by the operator at runtime); add a max-age freshness mode (table-level max(col) age) alongside the row-level mode; TIMESTAMP column support; honor the parsed-but-ignored unit param. |
Inventory: static scopes make true incremental checking inexpressible; freshness is permanently red on historical tables and DATE-only [I]. Partition-aware checks are a recurring practitioner ask [B]. | M |
| R9 | Alert fatigue controls. Aggregate one alert per (run, contract) with per-check detail; dedup across gate+monitor for the same data update; suppress re-fires on retry (R2 key); severity-tiered routing; absolute alert URL from configurable base. | Double/percheck alert storms [I]; fatigue controls are the adoption-deciding feature per practitioners [B]; incumbent OSS ships no default alerting at all [B] — this converts an existing differentiator into a defensible one. | S–M |
P2 — operability & scale¶
| # | Refactor | Why / evidence | Effort |
|---|---|---|---|
| R10 | Packaging & migrations: ship built UI in the wheel, Alembic migrations (replace initdb-only DDL), constraints-file CI installs, fix the lineage extra to the provider package (legacy openlineage-airflow pinned today [I]), py.typed. |
Wheel currently ships no UI; no migration story; CI installs unpinned Airflow [I]; Edge3 migration precedent [W 2-1]. | M |
| R11 | Per-adapter SQL layer (capability table + adapter-owned codegen) before backend #3 lands — the pattern rule's dialect split is already leaking into shared codegen [I]. | Prereq for F4; D2's own warning [I]. | L |
| R12 | API pagination/filters + retention/rollups on results (time-window params, per-dataset filters, nightly rollup + TTL for raw rows and samples). | Hardcoded 500/60 caps, unbounded growth, trend scans over full history [I]. | M |
| R13 | Docs regeneration (PROJECT_DESCRIPTION/article one release behind: recharts, 29 tests, "OL not built" [I]) + integration test replacing the green-passing echo stub. | Credibility + the entire serving surface has zero automated coverage [I]. | S–M |
3. Prioritized feature roadmap¶
Ranked by (demand evidence × differentiation × wedge fit). Each assumes its prerequisite refactors.
| # | Feature | Demand & differentiation evidence | Wedge fit | Prereqs |
|---|---|---|---|---|
| F1 | Anomaly detection upgrade (OSS, transparent): trailing-window baselines (exclude current run from mean/σ — today's baseline self-masks [I]), per-group baselines, metric-level anomalies computed server-side, persisted and alertable (today client-side, per-page-load [I]); seasonality-naive but honest. | Soda paywalls anomaly detection [V]; GX OSS has none [B]; Elementary has it but only for dbt users [W]. Free, in-Airflow anomaly detection is the clearest verified gap. | High — reuses sample/quarantine machinery (D14) | R2 |
| F2 | ODCS-native SLA checks + scorecards: implement RFC-0025 slaProperties (latency, frequency, retention, availability, completeness) as compiled checks with the same gate/monitor duality; SLA scorecard view per contract/team. |
SLAs are now first-class in the spec with execution semantics [V] — implementing them is standards-aligned, and no Airflow-niche tool does; scorecards are a recurring buyer ask [B]. | High — contracts are already the unit | R5, R8 |
| F3 | Quarantine triage UI + replay: browse dq_quarantine (filter by check/run/dataset), mark resolved, re-validate or export rows; retention controls. |
Quarantine is already a differentiator no OSS incumbent has (Elementary/GX/Soda: none [W/B]) but today it's write-only — invisible value [I]. | High — completes an existing wedge feature | R2, R7 |
| F4 | Warehouse coverage: Snowflake → BigQuery → Databricks → Trino (adapter order by market presence in the sources reviewed), each with OL-compliant naming from day one. | Table stakes for "most complete" — datacontract-cli's 18+ sources show the bar [V]; multi-warehouse is the top capability filter in practitioner comparisons [B]. | Necessary, not differentiating | R3, R6, R11 |
| F5 | Contract approval workflow (PR-based): git-backed contract store option (save = branch + PR via API), param/severity-aware diff (today invisible changes [I]) as the PR check, CODEOWNERS routing; in-UI "propose change" instead of direct save. | Governance/approval flows are the enterprise adoption blocker incumbents solve with paid cloud tiers [B]; extends the verified breaking-change wedge; fixes the mutable-version hole structurally. | Very high — "git for data" made literal | R4 |
| F6 | ODCS custom-engine passthrough / dbt-test import: execute or delegate custom quality blocks (engine: soda/greatExpectations/dbt) and import existing dbt tests into contracts. |
ODCS explicitly slots vendor engines in custom [V] — airflow-dq can be the executor of record for mixed contracts instead of competing head-on; migration path from incumbents lowers switching cost. |
Medium-high | R5 |
| F7 | Multi-contract / team overview dashboard: portfolio view (all contracts, pass-rate, SLA status, anomalies), search/grouping — today one-contract-at-a-time with a flat dropdown [I]. | Needed the day contract count > ~10; Elementary's dashboard is its headline OSS feature [W] — parity here protects the wedge. | Medium | R12 |
| F8 | Alert channels: email + PagerDuty/Opsgenie adapters on top of R9's grouping, with per-severity escalation. | OSS incumbents ship no default alerting [B]; channels beyond webhook are cheap once R9 exists. | Medium | R9 |
Deliberately down-ranked: column-level lineage (the framing that it's differentiation-bearing was refuted in verification [W-refuted]; OL facet quality via R6 delivers most of the value), ML-model-based anomaly (see below).
4. Do-NOT-build list¶
- A generic multi-engine CLI test runner — datacontract-cli owns it, ODCS-native, 18+ sources [V]. Commodity; integrate (F6), don't compete.
- A dbt-native observability clone — Elementary owns that niche outright [W]; airflow-dq's identity is orchestrator-native, not transformation-native.
- Black-box ML anomaly detection as the default — the verified OSS gap is usable, transparent detection (statistical, explainable); heavy models are optional extensions at most (matches the honest-statistical positioning of D14 and avoids competing with Monte Carlo-class platforms on their turf).
- A proprietary RBAC/user system — Airflow 3 mandates exactly one auth manager; plugins delegate (
is_authorized_*) [W]. Building own roles would be architecturally wrong per the spec. - Spec-divergent contract extensions — ODCS v3.1 rejects unknown fields [V]; anything outside
customPropertiesis a dead end. - A data catalog / BI layer — out of wedge; every incumbent's paid tier lives there; the dashboard should stay a DQ surface.
- Airflow 2 compatibility — the plugin surface used (external_views/fastapi_apps) is the endorsed Airflow-3 path [W]; back-porting doubles the maintenance surface for a shrinking base.
5. Competitor matrix (verified state, 2026)¶
| Capability | airflow-dq (today) | Soda Core / Cloud | GX Core / Cloud | dbt tests + Elementary | Databricks DQX | common-sql operators | Monte Carlo class |
|---|---|---|---|---|---|---|---|
| Data contracts (ODCS) | ⚠️ dialect [I] | via custom slot in ODCS [V] |
via custom slot [V] |
✗ (dbt schema tests ≠ contracts) | ✗ | ✗ | ✗ |
| In-orchestrator authoring UI | ✅ unique [I] | ✗ | ✗ (Cloud UI is external) | ✗ (dbt-native [W]) | ✗ (Databricks-native) | ✗ | ✗ external SaaS |
| Gate / monitor duality | ✅ [I] | partial (operator gate) | partial (operator gate) | ✗ | partial | gate only | monitor only |
| Quarantine of bad rows | ✅ write-only [I] | ✗ | ✗ | ✗ | ✅ (its headline) | ✗ | ✗ |
| Anomaly detection in OSS | ✅ z-score [I] | ✗ paid [V] | ✗ [B] | ✅ dbt-only [W] | ✅ Databricks-only | ✗ | paid product |
| Breaking-change detection | ✅ schema-level [I] | ✗ | ✗ | partial (dbt state) | ✗ | ✗ | schema-change monitors (paid) |
| Alerting in OSS | ✅ webhook, no dedup [I] | ✗ [B] | ✗ [B] | ✅ [W] | partial | ✗ | ✅ (paid) |
| Lineage emission (OL) | ⚠️ non-compliant naming [I][V] | partial | partial | test-result lineage [W] | Unity Catalog | via provider | proprietary |
| DQ dashboard in OSS | ✅ [I] | ✗ (Cloud) | ✗ (Cloud) | ✅ [W] | notebooks | ✗ | paid |
| Warehouse breadth | 2 (PG, DuckDB) [I] | broad | broad | = dbt targets | 1 | = Airflow conns | broad |
(DQX cells reflect its Databricks/PySpark scope; not independently re-verified this run.)
6. Suggested sequencing¶
- Hardening wave (P0): R1 auth → R2 reliability/idempotency → R3 connections → R4 contract integrity. Nothing above matters if the demo-grade auth and critical-path store stay as-is.
- Interop wave (P1): R5 ODCS v3.1 + R6 OL naming together (both are "speak the ecosystem's language"), R7–R9 alongside.
- Wedge features: F1 anomaly + F3 quarantine UI (turn existing unique capabilities into visible ones), then F2 SLAs (standards moment: RFC-0025 is new — first-mover window), F5 approval workflow.
- Breadth last: F4 warehouses ride on R11's adapter layer; F6–F8 opportunistically.
7. Evidence register¶
Primary-source verified [V]: OL naming formats (openlineage.io/docs/spec/naming) · OL provider operator requirements + naming mandate (airflow.apache.org providers-openlineage developer guide) · datacontract-cli ODCS-native + 18–24 test sources (repo README) · ODCS v3.1.0 strict validation + RFC-0025 SLAs (bitol.io announcement) · ODCS four quality types + vendor custom slot (spec data-quality page) · Soda anomaly detection Enterprise-gated (docs.soda.io).
Workflow-verified [W] (3-0 unless noted): Elementary dbt-only positioning + OSS feature set (2 claims) · Airflow 3 plugin surfaces supersede FAB · fastapi_apps ship without auth · single pluggable auth manager · auth manager as JWT issuer (2-1) · BaseAuthManager is_authorized_* hooks · Edge3 shared-secret baseline · Edge3 auto-migrated provider tables (2-1) · OL namespaces: jobs-from-scheduler, datasets-from-datasource.
Refuted: Elementary-Cloud-paywall framing of column-level lineage as differentiation-bearing (1-2).
Blog-sourced [B], medium confidence: GX OSS lacks auto-anomaly/freshness · GX/dbt/Soda OSS ship no default alerting · alert-fatigue controls decide adoption · warehouse-priority and practitioner-demand signals (sources: datakitchen 2026 landscape, practitioner comparisons on Medium/pipecode et al. — full URL list in the research run output).
Internal [I]: FEATURE-INVENTORY.md (2026-07-11).