Deequ

Deequ is an AWS Labs library built on Apache Spark for defining “unit tests for data”, which measure data quality in large datasets. The README says it is designed for very large data (“think billions of rows”) held in a distributed filesystem or a warehouse, and works on anything that fits a Spark dataframe: CSV, database tables, logs, flattened JSON.

Repository snapshot (GitHub API, 2026-11-18)

ItemDetail
Repoawslabs/deequ, Scala, Apache-2.0
Created2018-08-07
Activitylast push 2026-10-20; latest release 2.1.0 (2026-09-16)
Size3,660 stars, 584 forks

Requirements

  • Deequ 2.1.0 and later require Java 11; the 2.0.x releases require Java 8.
  • Each release is built for a specific Spark version (artifact suffix such as -spark-3.5); the README table covers Spark 3.1 to 3.5. Spark 3.0 and earlier need a 1.x release.
  • Python users use PyDeequ, a separate repository (awslabs/python-deequ).

Where it fits against Soda Core

Choose Deequ when the data already runs through Spark jobs and you want checks in Scala or, through PyDeequ, Python, computed at Spark scale. It is a library, not a platform: the README read mentions no hosted service, UI or alert routing. Soda Core targets SQL warehouses with YAML checks instead.

Sources