Delta Lake

What it is

An open-source storage layer that adds ACID transactions, time travel, schema evolution and DML (MERGE, UPDATE, DELETE) to Parquet files in object storage, using a transaction log (_delta_log). It is the native table format of Databricks and Microsoft Fabric.

Maker, ownership and history

• Introduced by Databricks in April 2019 (Wikipedia, secondary); delta.io states it joined the Linux Foundation in 2019 and is “not controlled by any single company”. Licence Apache 2.0.
• Major contributors listed by delta.io: Adobe, Amazon, Apple, Databricks, Microsoft and others.
• Releases (GitHub, checked 2026-10-07): 4.4.1 (2026-10-05) safer maintenance for catalog-managed tables, built for Spark 4.2.0/4.1.0/4.0.1; 4.4.0 (2026-08-20) Spark 4.2 support, GENERATED ALWAYS AS IDENTITY columns, generated columns with Unity Catalog, SHOW PARTITIONS; 3.3.3 (2026-08-12) maintenance line.

Editions/deployment

Library plus connectors (Spark, Flink, Trino, Rust/Python delta-rs, Delta Kernel); hosted in Databricks, Fabric, and others. No licence fee for the format; see vendor pricing pages.

Core architecture

Parquet data + JSON/checkpoint transaction log; optimistic concurrency; deletion vectors; column mapping; catalog-managed tables via Unity Catalog and the Delta REST catalog (Delta Connect is part of the UC Delta APIs per delta.io).

UniForm and Iceberg interop

UniForm generates Iceberg metadata asynchronously next to Delta metadata, without rewriting Parquet files, so Iceberg clients read the same data (Databricks docs): table must be in Unity Catalog, column mapping on, IcebergCompatV2 (older V1), read-only for Iceberg clients, deletion vectors cannot be enabled together (per the page checked; IcebergCompatV3 exists per Microsoft docs). delta.io also lists Hudi reading via UniForm. Fabric OneLake does comparable metadata virtualization (see Apache Iceberg).

Role in an enterprise AI rollout

Feature tables, training sets, RAG source tables and Delta Sharing for governed distribution; versioned time travel gives reproducible training data and lineage anchors.

AI features

Format itself has none; AI features live in Databricks and Fabric.

Integrations

Spark, Flink, Trino, Snowflake and BigQuery readers (per delta.io), Unity Catalog (Unity Catalog), Delta Kernel in ClickHouse (Rust).

Strengths and weaknesses (opinion)

• Mature, deep Spark/Databricks integration, simple log design.
• Iceberg has broader neutral vendor momentum; UniForm is read-only for Iceberg readers and carries feature restrictions.

Self-learning

• https://delta.io/ (project site, free)
• https://docs.databricks.com/aws/en/delta/uniform (UniForm, free)
• https://github.com/delta-io/delta (source and releases, free)
• Compare: Apache Iceberg, Hudi and Paimon

Sources (fetched 2026-10-07)

https://delta.io/ ; https://api.github.com/repos/delta-io/delta/releases ; https://docs.databricks.com/aws/en/delta/uniform ; https://learn.microsoft.com/en-us/fabric/onelake/onelake-iceberg-tables ; https://en.wikipedia.org/wiki/Databricks (secondary)

Open items

• “Delta 4.x” feature list beyond the release notes seen (e.g. 4.0 headline features) not verified.
• UniForm behaviour on newest Databricks runtimes may have moved on from the docs page fetched.