What Iceberg Replaced

Iceberg is an open table format that sits between compute engines and underlying storage. It does not store data directl

Iceberg is an open table format that sits between compute engines and underlying storage. It does not store data directly nor execute SQL. Instead, it manages table metadata, snapshots, partitioning, and file layout. When people ask “Iceberg replaced what,” the answer isn’t S3, HDFS, Parquet, or Spark. It replaces the traditional metadata and partitioning paradigm that has long been centered on Hive table format in data lakes.

In classic Hive setups, data is often organized by the partition column in the directory structure. An orders table partitioned by dt might look like this on disk:

/warehouse/orders/dt=2024-05-01/part-0001.parquet
/warehouse/orders/dt=2024-05-02/part-0002.parquet

The Hive Metastore tracks schema, partition lists, and storage locations. At query time, the engine reads the metastore, then lists the directory contents. While simple, this approach has clear limitations: partition columns must appear explicitly in the table schema, queries require explicit partition filtering, changing partitioning usually requires rewriting data, large numbers of partitions become a metastore bottleneck, and concurrent writing, atomic commits, and snapshot isolation are weak. Moreover, different compute engines offer inconsistent support for Hive table semantics.

Iceberg shifts the management of table state from directory structure to metadata files, manifests, and snapshots. Partitions can be hidden from the user; queries can filter on a business-time column, and the engine automatically prunes partitions. Partition evolution is also supported: old data continues to be read under the old specification, while new data follows the new rule, without a full table rewrite.
In traditional data lakes, query planning often relies on listing files. When data resides in object storage, listing directories at scale is both slow and prone to inconsistency. Iceberg records each data file’s path, partitioning, and column statistics in manifest files. The query engine reads metadata and manifests first, then determines exactly which files to scan.

This yields several immediate benefits:

  • Faster query planning, avoiding full directory listings
  • Snapshot isolation, allowing reads and writes to proceed without blocking
  • Time travel and rollback capabilities
  • Support for incremental reading and change data capture use cases
  • Easier small-file compaction and data rewriting

Thus, Iceberg replaces the old “rely on directory listings and partition-path inference for table state” model.

Replacing Part of Data Warehouse Table Capabilities

Iceberg provides ACID transactions, MERGE INTO, DELETE, UPDATE, schema evolution, partition evolution, and time travel. These capabilities were previously largely confined to data warehouses or proprietary table formats. With Iceberg, data lakes can take on many tasks that once required a dedicated warehouse, such as near-real-time upsert writes, historical querying via snapshots, safe column type modifications, and multi-engine sharing of a single table.

However, Iceberg does not replace a full data warehouse. It lacks a SQL execution engine, resource scheduler, permission system, BI semantic layer, materialized view management, and a cost-based optimizer. It works in concert with Spark, Flink, Trino, Presto, StarRocks, and similar engines.

Relationship with Delta Lake and Hudi

Iceberg, Delta Lake, and Hudi all fall under the data lake table format umbrella. They address highly overlapping problems: ACID guarantees, schema evolution, time travel, and incremental processing. The relationship is more one of competing solutions or scenario complementarity than Iceberg necessarily replacing one over the other. If a system already uses Delta Lake or Hudi, migration to Iceberg depends on multi-engine support, ecosystem tooling, operational cost, and team familiarity.

What Iceberg Does Not Replace

  • It does not replace object storage or HDFS; data still lives in S3, OSS, HDFS, etc.
  • It does not replace file formats like Parquet, ORC, or Avro; Iceberg typically organizes data on top of these formats.
  • It does not replace compute engines such as Spark, Flink, or Trino.
  • It does not necessarily replace the Hive Metastore; Hive Metastore can serve as an Iceberg catalog.
  • It does not replace Kafka or other messaging systems, nor does it replace scheduling, monitoring, or permission platforms.

Example: From a Hive Partition Table to an Iceberg Table

Using Spark SQL as an illustration, creating a daily-partitioned Iceberg table looks like this:

CREATE TABLE prod.db.orders (
  order_id BIGINT,
  user_id BIGINT,
  amount DECIMAL(10,2),
  ts TIMESTAMP
) USING iceberg
PARTITIONED BY (days(ts));

INSERT INTO prod.db.orders
VALUES (1, 1001, 99.50, TIMESTAMP '2024-05-01 10:00:00');

-- Query without explicitly writing the partition column
SELECT *
FROM prod.db.orders
WHERE ts >= TIMESTAMP '2024-05-01 00:00:0