A driver’s location from 5 minutes ago is useless to dispatch current rides. A finished trip barely get touched by the product again, once payment is processed and the customer journey closed. Yet the data still matter. Analytics, audits, and compliance want them. The challenge is that operational databases are not the most economical choice to store large amounts of historical data.
The big data storage problem
Data lakes emerged as a practical answer. Early versions often ran on HDFS. Today, it is mostly object storages like S3, Google Cloud Storage, and Azure Blob Storage. Dump the data there at lower cost, and query engines like Spark could read them directly when needed.
This reduced the burden on operational systems, and kept the history queryable. But moving data into a lake uncovered a new problem.
A table is more than a folder of files
One of Hive’s most important contributions was that a collection of files could behave like a table. Its metastore kept track of files paths, their partitions, and which schema readers should expect. Instead of dealing with raw files directly, query engines could interact with something like a normal database table.
That worked fine until tables started changing. Imagine a process adding new files while another updates or removes old ones. How can a reader know which version of the table is complete and valid? More importantly how do you prevent it reading a mixture of old and new files halfway through an update?
The problem was no longer storing files, but coordinating their changes.
Hive later added ACID transactions to tackle that problem but they were tied to Hive and its Hadoop-era machinery. Open table formats address the same problem differently: the table’s state and its transactions live inside the format itself, so no particular engine or metastore own them. If you ever spent time wrestling with Ambari clusters at 2 a.m., you probably know why those formats became popular. 😄
What an open table format adds
An open table format is a set of rules that describe what a table is, how it changes, and how different engines agree on its state.
The files hold the data. The metadata on top turns the files into a table.
Compared to a file format like Parquet, an open table format usually adds:
Table metadata. Information about the schema, partitions, file statistics, table history, active files, and sometimes delete markers.
Atomic commit rules. They allow the table to move from one valid state to another without exposing half-finished updates. Readers either see the old state or the new one, never something in between.
Interoperability. A shared definition of the table state that different compatible engines can understand without inventing their own interpretation.
The format itself is neither a query engine nor a storage system. Think of it as a protocol.
That may sound like a small distinction, but it becomes important once the same table is accessed by multiple platforms. The table no longer belongs to Spark, Trino, Snowflake, or any single engine. Everyone agrees to follow the same set of rules.
Before going any further, let’s take a look at the JSON below. It is the last version of the metadata file from one of the table formats explored in the lab accompanying this article. Some field names are intuitive enough to guess their purpose. Others are less obvious. That’s okay. For now, just observe what’s there. By the end of the article, the structure and its role in managing table state should feel much clearer.
{
"format-version":2,
"table-uuid":"8afb6d78-e20c-400f-9c99-7d45fe239a72",
"location":"s3a://lakehouse-lab/olist/runs/olist_article_3_2_1_20261006T092918Z_a1e10448/iceberg/lab/orders",
"last-sequence-number":3,
"last-updated-ms":1791279029768,
"last-column-id":8,
"current-schema-id":0,
"schemas":[
{
"type":"struct",
"schema-id":0,
"fields":[
{"id":1,"name":"order_id","required":false,"type":"string"},
{"id":2,"name":"customer_id","required":false,"type":"string"},
{"id":3,"name":"order_status","required":false,"type":"string"},
{"id":4,"name":"order_purchase_timestamp","required":false,"type":"timestamptz"},
{"id":5,"name":"order_approved_at","required":false,"type":"timestamptz"},
{"id":6,"name":"order_delivered_carrier_date","required":false,"type":"timestamptz"},
{"id":7,"name":"order_delivered_customer_date","required":false,"type":"timestamptz"},
{"id":8,"name":"order_estimated_delivery_date","required":false,"type":"timestamptz"}
]
}
],
"default-spec-id":0,
"partition-specs":[{"spec-id":0,"fields":[]}],
"last-partition-id":999,
"default-sort-order-id":0,
"sort-orders":[{"order-id":0,"fields":[]}],
"properties":{
"owner":"spark",
"write.update.mode":"copy-on-write",
"write.delete.mode":"copy-on-write",
"write.parquet.compression-codec":"zstd"
},
"current-snapshot-id":5484902428416146878,
"refs":{"main":{"snapshot-id":5484902428416146878,"type":"branch"}},
"snapshots":[
{
"sequence-number":1,
"snapshot-id":8045760629618699002,
"timestamp-ms":1791278977055,
"summary":{
"operation":"append",
"spark.app.id":"local-1791278966905",
"manifests-created":"1",
"manifests-kept":"0",
"manifests-replaced":"0",
"added-data-files":"2",
"added-records":"3",
"added-files-size":"6124",
"changed-partition-count":"1",
"total-records":"3",
"total-files-size":"6124",
"total-data-files":"2",
"total-delete-files":"0",
"total-position-deletes":"0",
"total-equality-deletes":"0",
"engine-version":"3.5.7",
"app-id":"local-1791278966905",
"engine-name":"spark",
"iceberg-version":"Apache Iceberg 1.11.0 (commit 6976e020b894f6a6777704df2b8c4458cb291ae9)",
"app-name":"open-table-formats-lab-1"
},
"manifest-list":"s3a://lakehouse-lab/olist/runs/olist_article_3_2_1_20261006T092918Z_a1e10448/iceberg/lab/orders/metadata/snap-8045760629618699002-1-98e0f66a-ad0a-4233-a7b1-cd7bfea963a7.avro",
"schema-id":0
},
{
"sequence-number":2,
"snapshot-id":7626417279546034493,
"parent-snapshot-id":8045760629618699002,
"timestamp-ms":1791279016591,
"summary":{
"operation":"append",
"spark.app.id":"local-1791278966905",
"manifests-created":"1",
"manifests-kept":"1",
"manifests-replaced":"0",
"added-data-files":"2",
"added-records":"2",
"added-files-size":"6123",
"changed-partition-count":"1",
"total-records":"5",
"total-files-size":"12247",
"total-data-files":"4",
"total-delete-files":"0",
"total-position-deletes":"0",
"total-equality-deletes":"0",
"engine-version":"3.5.7",
"app-id":"local-1791278966905",
"engine-name":"spark",
"iceberg-version":"Apache Iceberg 1.11.0 (commit 6976e020b894f6a6777704df2b8c4458cb291ae9)",
"app-name":"open-table-formats-lab-1"},
"manifest-list":"s3a://lakehouse-lab/olist/runs/olist_article_3_2_1_20261006T092918Z_a1e10448/iceberg/lab/orders/metadata/snap-7626417279546034493-1-e5994a38-6f82-4e0e-948c-e11bba1c7f27.avro",
"schema-id":0
},
{
"sequence-number":3,
"snapshot-id":5484902428416146878,
"parent-snapshot-id":7626417279546034493,
"timestamp-ms":1791279029768,
"summary":{
"operation":"overwrite",
"spark.app.id":"local-1791278966905",
"manifests-created":"2",
"manifests-kept":"1",
"manifests-replaced":"1",
"added-data-files":"1",
"deleted-data-files":"1",
"added-records":"1",
"deleted-records":"2",
"added-files-size":"3104",
"removed-files-size":"3063",
"changed-partition-count":"1",
"total-records":"4",
"total-files-size":"12288",
"total-data-files":"4",
"total-delete-files":"0",
"total-position-deletes":"0",
"total-equality-deletes":"0",
"engine-version":"3.5.7",
"app-id":"local-1791278966905",
"engine-name":"spark",
"iceberg-version":"Apache Iceberg 1.11.0 (commit 6976e020b894f6a6777704df2b8c4458cb291ae9)",
"app-name":"open-table-formats-lab-1"
},
"manifest-list":"s3a://lakehouse-lab/olist/runs/olist_article_3_2_1_20261006T092918Z_a1e10448/iceberg/lab/orders/metadata/snap-5484902428416146878-1-aed271a5-0ade-4677-b553-70d274103627.avro",
"schema-id":0
}
],
"statistics":[],
"partition-statistics":[],
"snapshot-log":[
{"timestamp-ms":1791278977055,"snapshot-id":8045760629618699002},
{"timestamp-ms":1791279016591,"snapshot-id":7626417279546034493},
{"timestamp-ms":1791279029768,"snapshot-id":5484902428416146878}
],
"metadata-log":[
{
"timestamp-ms":1791278977059,
"metadata-file":"s3a://lakehouse-lab/olist/runs/olist_article_3_2_1_20261006T092918Z_a1e10448/iceberg/lab/orders/metadata/v1.metadata.json"
},
{
"timestamp-ms":1791279016591,
"metadata-file":"s3a://lakehouse-lab/olist/runs/olist_article_3_2_1_20261006T092918Z_a1e10448/iceberg/lab/orders/metadata/v2.metadata.json"
}
]
}Do you even need one ? Maybe not
Sometimes a data warehouse is the right choice. If it provides the performance, governance, reliability, and operational simplicity needed, introducing a lakehouse layer may add more complexity than value. Open table formats starts to pay off when multiple platforms need to work on the same data. Instead of maintaining separate ingestion copies for different engines, you expose same tables through one protocol.
They do not replace data warehouses. They provide a transactional layer that warehouses, processing frameworks, and analytics engines can all operate on.
That design sits at the core of the lakehouse architecture: warehouse-style table management on top of open storage.
Benefits and practical limits
Open table formats adoption should be rooted in practical motivations:
Revisit an earlier version of a table to reproduce previous analyses (e.g. for audits).
Update, delete, and change schema or partition without rebuilding entire datasets.
Improve query performance by using metadata and file statistics to avoid unnecessary scans.
Reduce data duplication since several platforms can work on the same tables.
The benefits are real, but so are the trade-offs 😅
Support varies across platforms. Reading a table is one thing. Writing to it, handling deletes, supporting newer specification versions, managing branches, or dealing with complex data types can be another story entirely. Always test the operations your workloads depend on, not just
SELECT *.Maintenance does not disappear. Their small files need compaction. Old snapshots need retention policies. Orphaned files need cleanup. Permissions, governance, monitoring, catalogs, and operational processes are still your responsibility.
The format manages the table. It does not manage your analytics platform.
It is not an operational database. Open table formats provide transactional guarantees for analytical workloads, but they are not designed for high-concurrency OLTP traffic, millisecond lookups, or database constraints such as referential integrity and primary-key enforcement.
Respect the protocol: don’t touch the files.
One of the easiest ways to break an open table is to forget that it is a table. Yes, its data live as files in object storage, but they should never be directly manipulated.
Directly deleting, replacing, or editing objects in storage can leave metadata pointing to a state that no longer exists.
In best case, the readers miss data. In worst case, the table becomes unreadable. Every changes should go through the format’s own write and commit process, because that is the only thing keeping metadata and files in sync.
“Open” means many systems can implement and understand the protocol. It does not mean any tool can safely modify the storage directory without understanding the rules that hold the table together.
Open table formats did not make data engineering simpler. They moved complexity to a place where multiple systems can agree on it.
Four open table formats you’ll hear about
Apache Iceberg
Iceberg is usually the first format evaluated when interoperability is the primary goal. It began at Netflix and entered Apache incubation in 2018. It uses snapshots, manifest metadata, stable field IDs, hidden partitioning, and partition evolution. If your requirements include broad multi-engine analytics, changing partition strategies, and vendor-neutral table metadata, Iceberg is often the safest place to start.
Delta Lake
Created at Databricks and released as open source in 2019, Delta’s ordered transaction log provides reliable table versioning and supports operations such as updates, deletes, and MERGE, which are part of many modern data pipelines. Its surrounding ecosystem also makes it attractive for platforms that support unified batch and streaming workloads. Many teams don’t really choose Delta because they want Delta. They choose it because they are already heavily invested in the Spark and Databricks ecosystem.
Apache Hudi
Hudi puts a lot of attention on datasets that change frequently. It includes capabilities for incremental processing, upserts, and change capture. Those features are valuable to reduce downstream processing costs when records are constantly being updated rather than just appended. Originally developed at Uber and open-sourced in 2017, Hudi deserves serious consideration for workloads that resemble a stream of ongoing corrections and updates.
Apache Paimon
Paimon is the newest entrant in the group and has gained attention in Flink-heavy environments. It began as Flink Table Store in 2022 before becoming an Apache top-level project in 2024. Its design is closely aligned with streaming workloads, where updates arrive continuously and table state needs to evolve quickly. Features such as primary-key tables and changelog generation make Paimon worth having on the radar for organizations building around Flink.
None of these formats is universally better than the others. The best is the one that aligns with the team’s workloads and operating model.
Inspect the metadata
In the accompanying lab1, Spark reads a sample of olist orders dataset and creates equivalent Iceberg, Delta Lake, and Hudi Copy-on-Write tables in local S3-compatible object storage.
The source data remains unchanged. To observe how table state evolves, the lab initialised each table’s three orders, inserted two more, and deleted one: 3 → 5 → 4. At each stage, the resulting rows are validated across all three formats.
lab is the Iceberg namespace, adding a level before orders. Delta and Hudi use explicit table paths.
What identifies the current table state?
Begin with the three-row table. Each format maintains metadata that tells readers which files belong to the current table version:
Iceberg uses a
current-snapshot-idthat points to a snapshot, which references manifests containing data-file metadata.Delta Lake reconstructs table state from the sequence of actions stored in
_delta_log.Hudi uses completed timeline instants and file-group versions to define the current Copy-on-Write snapshot.
Although implemented differently, all three formats answer the same question: Which files represent the table right now?
What commits the two inserted orders?
Next, two additional orders are inserted.
Iceberg records a new append snapshot containing references to the newly added files.
Delta Lake creates a new log version containing
addactions.Hudi completes an
INSERTcommit withnumInserts: 2,and records the resulting file-group state.
After the commit, all three tables return five orders. The important distinction is that inserted rows and added files are different concepts. Table formats track file-level changes, while query results reflect row-level outcomes.
How does deletion change the referenced files?
Finally, one order is deleted. This run uses Iceberg copy-on-write, Delta without deletion vectors, and Hudi Copy-on-Write.
Iceberg creates a new snapshot that replaces the affected file reference.
Delta Lake records both a
removeaction and anaddaction in a new log version.Hudi writes a new base-file version with
numDeletes: 1,and records the delete in its timeline metadata.
Again, all three tables return the same result: four orders. However, older physical files still exist in storage. An object listing alone cannot determine the current contents of the table. The answer lives in the metadata layer.
The JSON shared earlier is the lab’s iceberg table final state metadata. It describes the table’s structure, settings, and snapshot history, not the order data itself. Explore the lab repository to better understand its properties interpretation.
In the lab, older files remained in storage, yet each table still returned the current four orders. The state of the table was defined by its metadata, not by a directory listing.
Open table formats make that state explicit and define how it evolves over time. When choosing a format, the most important considerations are the workloads, the operations supported by the engines that access it, and the level of maintenance the team can realistically sustain.
NB: Research included documentation, articles, and AI-assisted learning summaries.