Dataform
by Google Cloud
A managed service for developing, testing, version controlling and scheduling SQL data transformation workflows in BigQuery. Its core language is open source.
Core model
- SQL with dependency management — extends SQL with dependencies, automated data quality assertions and documentation
- Compilation — compiles code to standard SQL, runs statements in dependency order, then runs assertion queries
- Reuse — JavaScript can be used to share code
- Scheduling — triggered manually or scheduled through Cloud Composer or Workflows
- Open source — Dataform core is on GitHub and can run locally