Dataform

by Google Cloud

A managed service for developing, testing, version controlling and scheduling SQL data transformation workflows in BigQuery. Its core language is open source.

Core model

  • SQL with dependency management — extends SQL with dependencies, automated data quality assertions and documentation
  • Compilation — compiles code to standard SQL, runs statements in dependency order, then runs assertion queries
  • Reuse — JavaScript can be used to share code
  • Scheduling — triggered manually or scheduled through Cloud Composer or Workflows
  • Open source — Dataform core is on GitHub and can run locally

Sources