Data Quality Contracts in Databricks: Beyond a Passing Pipeline

Start with the decision the dataset supports

Suppose a daily operations report shows open invoices. An upstream job finishes successfully, but it loads only one of three source regions. The SQL is valid and the dashboard renders. The business still receives a misleading answer. The missing requirement is a contract describing what “ready” means for this particular dataset.

The same problem becomes harder to notice when an AI assistant turns the result into a confident sentence. Databricks’ discussion of data-native agents emphasizes keeping governed data and agent computation together. An additional engineering question remains: even when access is authorized, is the data complete and current enough for this answer? This guide proposes a consumer-facing contract to make that question testable.

Write a small contract before writing more checks

A contract can start as a reviewed document stored beside the transformation code. It should define the dataset at the level a consumer understands, not merely list column types. For an illustrative daily invoice snapshot, the following fields would make the operational promise explicit.

• Grain: one row per invoice and snapshot date; multiple invoice lines must be aggregated or represented by a different key.
• Meaning: amounts use a named currency convention, and status values have agreed definitions.
• Coverage: each expected source region reports a successful batch, including an explicit empty batch when there are no records.
• Freshness: yesterday’s completed business day is ready by an agreed time in a named timezone.
• Ownership: one team owns the producer, a named contact owns the report, and an escalation channel exists.
• Failure response: hold publication, label a previous snapshot as stale, or publish partial data with visible coverage only if consumers have accepted that behavior.

Separate row rules from dataset rules

Databricks pipeline expectations evaluate SQL Boolean constraints on records. Its documented actions can retain invalid rows while recording quality information, drop invalid rows, or fail an update. The constraints do not support subqueries to other tables. Use separate aggregate validation for uniqueness, expected-source coverage, and reconciliation instead of assuming a row expectation verifies the entire contract.

The following illustrative Spark SQL query returns duplicate business keys for one fixed snapshot. Use a run parameter in an actual job. A passing check returns zero rows. This example assumes invoice_id and snapshot_date define the agreed grain; it would be wrong for a table whose grain is invoice line.

SELECT invoice_id, snapshot_date, COUNT(*) AS row_count
FROM curated.invoice_daily
WHERE snapshot_date = DATE '2026-10-13'
GROUP BY invoice_id, snapshot_date
HAVING COUNT(*) > 1;

Check null business keys separately. For freshness, use a batch manifest with a completed business date, source-region identifier, and completion time. The largest event timestamp in the table is insufficient: a single recent row can hide an otherwise incomplete load. Coverage checks should compare the manifest with the expected source list, rather than infer success from whichever sources happened to arrive.

Choose a failure action based on the consumer

Dropping malformed records may be acceptable for a low-stakes exploratory event stream, but it can silently understate a financial total. Failing every minor anomaly can also leave a report permanently stale. Classify each rule by its consequence: warn for a newly observed optional attribute, quarantine an invalid key for repair, and block publication when a required source is missing.

Keep rejected records in a restricted repair path when retention and data handling requirements allow it. Record the rule, batch, and reason, then decide who can correct and replay them. Do not treat quarantine as an unowned second data lake. A useful quality measure includes unresolved age and repair rate, not just how many rows were rejected.

Publish data and readiness together

Make the readiness result part of the consumer interface. A dashboard should show the business date and source coverage. An assistant should receive those fields alongside the query result and decline a current-state answer when the snapshot is not ready. Preserve the last accepted version when appropriate, but label its age explicitly. A stale answer described honestly is easier to reason about than a current-looking answer assembled from partial data.

Test the contract with a deliberately bad batch

• Remove one expected source and confirm publication is held.
• Duplicate one invoice key and confirm the aggregate validation finds it.
• Send an explicit empty batch and confirm it is distinguishable from a missing batch.
• Replay the same batch and check that the output is not duplicated.
• Restore a corrected batch and confirm readiness changes only after all required checks pass.

Version changes to grain, business meaning, and readiness promises with the consumers who depend on them. Adding a column is usually easier than changing what an existing amount represents. The contract earns its place when it changes release and recovery behavior, not when it becomes another document nobody reads.