Metadata Over Compute: Visualizing Table History with dlt and Iceberg

If you’ve been following the Tabular – Databricks news, you’ve probably heard the phrase “metadata over compute.” It sounds like marketing, but it’s a real architectural idea. We use dlt (data load tool) to ingest into Apache Iceberg and watch how table metadata evolves and why it stays independent of the catalog.

The Thesis

Most data platforms tie table metadata to the compute engine. Your schema, partition info, and history live inside Snowflake, BigQuery, or Databricks. Move tables, and you leave that metadata behind.

Apache Iceberg inverts that. The source of truth is the metadata itself: JSON and Avro files that sit next to your data. The catalog (Hive, Glue, Snowflake Polaris, Tabular) is just a pointer: “this table’s metadata lives here.”

The metadata is portable. You can swap catalogs without losing history or lineage.

That’s “metadata over compute” in practice.

Operation → Metadata, in Succession

We use dlt’s filesystem destination with table_format="iceberg". Same pipeline, three runs.

1. Append five rows

@dlt.resource(table_format="iceberg", table_name="events")
def events_append():
    yield [{"id": 1, "event": "deploy", "ts": "..."}, ...]

pipeline.run(events_append())

Metadata responds:

  metadata/ has 2 file(s):
    00000-....metadata.json
    00001-....metadata.json
  current-snapshot-id: 8570791162370934054
  snapshots (history): 1
    [1] append: parent=None  added=5 deleted=0 total_records=5

Each commit creates a new metadata file. Iceberg keeps the old ones.

2. Merge updates for rows 1 and 2

@dlt.resource(table_format="iceberg", table_name="events",
              write_disposition="merge", primary_key="id")
def events_overwrite():
    yield [{"id": 1, "event": "deploy-v2", ...}, {"id": 2, "event": "build-v2", ...}, ...]

pipeline.run(events_overwrite())

Metadata responds:

  snapshots (history): 3
    [1] append: added=5 deleted=0 total_records=5
    [2] delete: added=0 deleted=5 total_records=0
    [3] append: added=5 deleted=0 total_records=5

Merge is modeled as delete + append. The chain is explicit.

3. Replace the table without row 3

@dlt.resource(table_format="iceberg", table_name="events", write_disposition="replace")
def events_after_delete():
    yield [{"id": 1, ...}, {"id": 2, ...}, {"id": 4, ...}, {"id": 5, ...}]

pipeline.run(events_after_delete())

Metadata responds:

  snapshots (history): 4
    ...
    [4] overwrite: added=4 deleted=5 total_records=4

Replace drops the old data and writes only what remains. Final state: 4 rows. The full chain stays in the metadata.

Where Does the Data Live?

The metadata points to the Parquet files. It doesn’t store the column values themselves.

Why This Matters

1. Portability — Metadata is just files. You can move them to another catalog (Polaris or Tabular) and keep the same history and lineage.
2. Auditability — snapshot-log and metadata-log give a clear record of what changed and when.
3. Time travel — Snapshots form a chain through parent. You can query the table as it looked at any past snapshot.
4. Catalog independence — The catalog is only a mapping from a table name to its metadata location. The same metadata files work with Hive, Glue, Polaris, or Tabular.