DQX (Databricks Labs)
DQX is a Databricks Labs framework for data quality checks on PySpark workloads, on streaming and standard DataFrames and tables. The README tagline: “Simplified Data Quality checking at Scale for PySpark Workloads”.
Repository snapshot (GitHub API, 2026-11-18)
| Item | Detail |
|---|---|
| Repo | databrickslabs/dqx, Python |
| Licence | The Databricks License, not an OSI open-source licence. Its text defines the agreement by reference to the Databricks Services terms, so read it before use outside Databricks. |
| Created | 2024-04-23 |
| Activity | last push 2026-11-18; latest release v0.17.2 (2026-11-05) |
| Size | 487 stars, 168 forks |
Capabilities (from the README)
- 80+ built-in checks (null, range, regex, referential, aggregate, geo, PII) at row and dataset level, plus custom check functions.
- Checks as code or as YAML or JSON; failures marked warning or error; invalid rows can be dropped, marked or quarantined.
- Profiling and rule generation from existing data; AI-assisted rule suggestions from business descriptions (DSPy and Databricks Model Serving); Isolation Forest row anomaly detection.
- Generates quality rules from Open Data Contract Standard contracts, including schema validation.
- Summary metrics saved to Delta tables, a Lakeview quality dashboard, and Slack, Teams or webhook alerts.
- Rules stored in files, Unity Catalog tables, Volumes or Lakebase.
- DQX Studio (a no-code UI deployed as a Databricks App) and an MCP server exposing DQX to AI agents, with data access following Unity Catalog permissions.
- Works for batch and Spark Structured Streaming, including Lakeflow Pipelines.
Where it fits against Soda Core
DQX is the Databricks-native choice: everything runs on Spark, Delta and Unity Catalog. It covers quality checks, contracts and dashboards in one framework, but ties you to Databricks. Soda Core is warehouse-neutral.
Sources
- DQX README, its LICENSE file and GitHub API (2026-11-18)