If you are building a modern data platform, you are almost certainly building a Data Lakehouse. And if you are building a Data Lakehouse, you have to make a choice between the big three open table formats: Apache Iceberg, Delta Lake, and (less commonly today) Apache Hudi.
Both Iceberg and Delta Lake solve the same fundamental problem: they bring ACID transactions, time travel, and schema evolution to data lakes (which are otherwise just folders full of Parquet files).
So, which one should you choose? Let’s break down the technical differences and the ecosystem lock-in.
1. Delta Lake (The Databricks Standard)
Delta Lake was originally developed by Databricks and later open-sourced. It is deeply integrated into the Databricks ecosystem and is the default storage format for Databricks Lakehouse, Microsoft Fabric, and Azure Synapse.
Technical Approach
Delta Lake relies on a _delta_log directory containing JSON files (and occasionally Parquet checkpoints) that record every transaction (insert, update, delete). When a compute engine reads a Delta table, it first reads this log to figure out exactly which Parquet files represent the current state of the table.
Pros of Delta Lake
- Out-of-the-box Optimization: If you use Databricks, Delta Lake works flawlessly. Features like
OPTIMIZE, Z-Ordering, and Liquid Clustering are heavily optimized. - Ecosystem Momentum: Microsoft Fabric uses Delta as its native format. If you are in the Azure ecosystem, Delta is the path of least resistance.
- Streaming: Delta has incredibly strong integration with Structured Streaming in Spark.
2. Apache Iceberg (The Vendor-Agnostic Choice)
Apache Iceberg was created at Netflix to solve problems they had with Hive scale. Unlike Delta, which started at a vendor and was open-sourced, Iceberg started as a pure open-source project and has gained massive support from vendors like Snowflake, AWS, and Tabular (now acquired by Databricks).
Technical Approach
Iceberg uses a hierarchical metadata structure (Metadata file → Manifest List → Manifest File → Data Files). This means compute engines do not have to list directories in object storage (which is slow on AWS S3). Iceberg scales to massive tables (petabytes) incredibly well because of this design.
Pros of Apache Iceberg
- Vendor Neutrality: Snowflake, AWS Athena, Starburst, and Google BigQuery all have native, first-class support for Iceberg.
- Hidden Partitioning: Iceberg handles partitioning under the hood. You don’t have to rewrite queries if you change your partitioning strategy (e.g., from daily to hourly).
- Massive Scale: The metadata tree structure is arguably better designed for tables with millions of files compared to Delta’s transaction log.
The Verdict: Which Should You Choose?
The choice rarely comes down to technical features—both formats are excellent and constantly copying features from each other. The choice comes down to your compute engine.
Note: Databricks recently acquired Tabular (the company behind Iceberg) and announced “UniForm”, which allows Delta tables to be read as Iceberg tables automatically. The format wars are slowly converging into interoperability.