Open lakehouse catalogs
What it is
A lakehouse catalog stores the pointer from a table name to its current metadata and enforces access. With the Iceberg REST catalog protocol, any compliant engine can talk to any compliant catalog, so governance (who may read which table, credential vending to object storage, audit) is enforced once instead of per engine. This is what makes multi-engine setups (Spark + Trino + Snowflake + DuckDB + agents) governable. See Apache Iceberg and the wider comparison in Data catalogs compared.
The catalogs (checked 2026-10-07)
| Catalog | Facts |
|---|---|
| Unity Catalog OSS | GitHub unitycatalog/unitycatalog, Apache 2.0, created 2024-06-13, Java, tagline “Open, Multi-modal Catalog for Data & AI”, last push 2026-10-05; open-sourced June 2024 (Wikipedia). Commercial version: Databricks Unity Catalog. |
| Apache Polaris | Iceberg REST catalog; site describes it as “the open catalog for data and AI assets”; shows as an Apache top-level project (graduation date not verified). Works with Doris, Flink, Spark, Dremio OSS, StarRocks, Trino; Postgres/CockroachDB backends; Keycloak auth. Releases 1.8.0 (2026-09-28), 1.7.0 (2026-08-02). Snowflake’s hosted version is called Snowflake Open Catalog (naming from Snowflake; origin not re-verified). |
| Lakekeeper | Rust Iceberg REST catalog, Apache 2.0, created 2024-04-05; “access control, credential vending and audit for every engine and AI agent”; last push 2026-10-07. |
| Apache Gravitino | ”high-performance, geo-distributed, and federated metadata lake”: unified metadata over Hive, MySQL, PostgreSQL, HDFS, S3; access control, audit, discovery; AI asset management marked work in progress on the site. |
| Cloud-native | AWS Glue Data Catalog with S3 Tables (AWS), BigQuery/BigLake (BigQuery), OneLake (Fabric) |
Role in an enterprise AI rollout
Single place to grant agents and models least-privilege table access, vend short-lived storage credentials, and record audit trails. MCP/agent access to the catalog is vendor-specific; Lakekeeper explicitly targets AI agents in its description.
Strengths and weaknesses (opinion)
- Neutral REST catalogs reduce lock-in; self-hosting adds operations work.
- Fine-grained policy (row/column, ABAC) coverage differs per catalog and per engine; check that engines honour it.
Self-learning
- https://polaris.apache.org/ (free)
- https://github.com/unitycatalog/unitycatalog (free)
- https://github.com/lakekeeper/lakekeeper (free)
- https://gravitino.apache.org/ (free)
Sources (fetched 2026-10-07)
Sites and GitHub APIs above; https://api.github.com/repos/apache/polaris/releases ; https://en.wikipedia.org/wiki/Databricks (secondary)
Open items
- Polaris origin (Snowflake) / incubation and graduation dates, Gravitino graduation date, and Unity Catalog’s foundation (LF AI & Data) not verified. polaris.apache.org/ page fetched via summary only; Wikipedia Apache Polaris page returned 404.