Tag: iceberg

The pitch is intoxicating and mostly true: keep your data in open Apache Iceberg format on your own object storage, let external engines read it, and let Snowflake’s Dynamic Tables handle the low-latency transformations on top — one declarative pipeline, no lock-in, a real lakehouse. In 2026, with Iceberg v3 generally available on Snowflake since May 7, teams are migrating native tables to dynamic Iceberg tables expecting exactly that. Most of them hit the same wall in the same order.

The wall is this: “open and interoperable” describes the storage format, not the write path, and “low latency” describes Dynamic Tables under conditions that partitioned Iceberg writes and cross-engine change tracking quietly violate. The migration doesn’t fail loudly. It succeeds, ships, and then your incremental pipeline starts doing full refreshes you didn’t ask for, your partitioned writes fan out into a metadata problem, and the external engine you promised could write to these tables turns out to be read-only. This is the guide to the traps — the ones that don’t show up in the quickstart — and how to design around them before they cost you a quarter.

TL;DR

→ Iceberg v3 (GA on Snowflake May 2026) brings deletion vectors, row lineage (native CDC), VARIANT, and multi-argument partition transforms. You cannot upgrade v2 to v3 in place — you recreate the table. Plan the migration, don’t expect an ALTER.

→ Dynamic Iceberg tables support PARTITION_BY, TARGET_FILE_SIZE, and PATH_LAYOUT. PATH_LAYOUT = HIERARCHICAL only produces Hive-style partitioned paths when paired with PARTITION_BY — and over-partitioning (more than a few thousand partitions) turns your metadata layer into the bottleneck.

→ The cross-engine reality: Snowflake-managed Iceberg tables are read-write for Snowflake, read-only for external engines. Writes from external engines to Snowflake-managed v3 tables via Horizon Catalog aren’t supported yet. External engines can only write to externally-managed tables.

→ Dynamic Tables track changes at the row level for native tables but the file level for externally-managed Iceberg base tables. Frequent copy-on-write on the external table degrades incremental refresh — a file changes, the whole file is “changed.”

→ INSERT OVERWRITE on a base table resets change-tracking metadata and forces a full refresh. Row lineage (v3) and primary keys with RELY are how you keep incrementality alive across rewrites.

→ Deletion vectors (v3 merge-on-read) are governed by heuristics: Snowflake only writes a deletion vector if fewer than ~5% of a file’s rows are deleted and the file is larger than ~1.6 MB. External engines that don’t understand v3 deletion vectors force you to set ICEBERG_MERGE_ON_READ_BEHAVIOR = 'DISABLED' (copy-on-write) for compatibility.

What v3 actually changed, and why in-place upgrade isn’t a thing

The target architecture: a Bronze/Silver/Gold lakehouse where Dynamic Iceberg tables handle incremental transforms and write open Iceberg that external engines can read. The traps live in the arrows, not the boxes.

Iceberg v3 is a genuine step change, not a point release. It adds deletion vectors (up to ~10x faster DML by avoiding positional-delete merges at read time), row lineage for native change data capture, a VARIANT type for semi-structured data with structured-query performance, default column values, geometry/geography types, nanosecond timestamps, and multi-argument partition transforms. On Snowflake it went to preview in March 2026 and GA on May 7, 2026.

Here’s the first thing that trips migrations: you can’t upgrade an Iceberg table from v2 to v3. There is no ALTER TABLE ... SET ICEBERG_VERSION = 3 that rewrites your existing table in place. You configure the default Iceberg version and create new v3 tables, migrating data into them. This matters because teams plan the migration as a flag flip and discover it’s a recreate-and-backfill — which, for a large partitioned table, is a real project with a real compute bill, not a maintenance-window toggle. The related gotcha: v2 tables using copy-on-write represent an updated or relocated row in a standard stream as a DELETE followed by an INSERT for the same row, so any CDC logic you built on v2 stream semantics needs re-validation against v3’s row lineage before you cut over.

The partitioned-write trap: HIERARCHICAL paths and the metadata ceiling

Dynamic Iceberg tables expose three storage-shaping properties: PARTITION_BY, TARGET_FILE_SIZE, and PATH_LAYOUT. The one that surprises people is PATH_LAYOUT. It defaults to FLAT, meaning all Parquet data files land directly under the data/ directory. Set it to HIERARCHICAL and Snowflake writes Hive-style partitioned paths — but only in combination with PARTITION_BY. Setting HIERARCHICAL without a partition spec does nothing useful; the two are a pair.

A minimal partitioned dynamic Iceberg table looks like this:

CREATE DYNAMIC ICEBERG TABLE my_dt (
  product_id NUMBER, product_name STRING, order_time TIMESTAMP_NTZ
)
  TARGET_LAG = '20 minutes'
  WAREHOUSE = my_wh
  EXTERNAL_VOLUME = 'my_vol'
  CATALOG = 'SNOWFLAKE'
  BASE_LOCATION = 'my_dt'
  PARTITION BY (YEAR(order_time))
  PATH_LAYOUT = HIERARCHICAL
  AS SELECT product_id, product_name, order_time FROM staging;

The trap isn’t the syntax; it’s the partition cardinality. Iceberg’s metadata tracks files per partition, and every partition you create adds manifest overhead. Snowflake’s own guidance is blunt: avoid creating more than a few thousand partitions, and test query performance against your actual workload before finalizing a partitioning strategy. The failure mode when you ignore this is quietly brutal — partition by DAY(event_time) on a table with a few years of history and a high-cardinality secondary key, and you can generate tens of thousands of tiny partitions, each with its own small files. Now your Dynamic Table refresh spends its time in metadata planning rather than moving data, and your “low-latency” pipeline has a latency floor set by manifest bookkeeping.

The design rule that keeps you out of trouble: partition on the coarsest grain that still prunes your dominant query pattern (usually a month or a broad category), let TARGET_FILE_SIZE and Snowflake’s file management handle within-partition layout, and reach for clustering rather than finer partitions when you need more selective pruning. Hierarchical paths are for interoperability and human-navigable storage, not a license to over-partition.

The cross-engine write trap: “interoperable” is asymmetric

This is the one that derails architecture diagrams. The interoperability story — external engines like Spark and Trino reading your Iceberg data — is real, but it runs in one direction for Snowflake-managed tables. Snowflake-managed Iceberg tables are read-write for Snowflake and read-only for external engines. As of the v3 GA, reading Snowflake-managed v3 tables from an external engine via the Horizon Iceberg REST Catalog API is generally available; writing from external engines to Snowflake-managed v3 tables through Horizon is explicitly not supported yet.

If your architecture needs an external engine to write Iceberg that Snowflake then transforms, you must use externally-managed tables — data written by Spark into a catalog like AWS Glue, which Snowflake reads via a catalog integration and a linked database. That’s a supported and powerful pattern (it’s the canonical Bronze layer of an open lakehouse), but it’s a different architecture with different semantics than “Snowflake-managed tables that everyone can write to,” which does not exist today. Decide early which engine owns writes for each table, because that choice dictates managed-vs-external, and switching later means a migration. A further sharp edge: you can’t write with vended credentials to cloned or converted tables, and you can’t write at all to a table that was converted from externally-managed to Snowflake-managed — conversions are one-way for write access.

The change-tracking trap: file-level vs row-level

The granularity of change tracking decides how much work an incremental refresh does. Row-level (native) processes a tight delta; file-level (external Iceberg) can reprocess an entire file because one row moved.

Dynamic Tables get their speed from incremental refresh — processing only what changed since the last refresh. The catch that native-table migrators don’t see coming: Dynamic Tables track changes at the file level for externally-managed Iceberg base tables, whereas they track at the row level for native Snowflake tables. That single difference reshapes your performance profile.

With a native base table, if one row in a micro-partition changes, Snowflake knows it was that row, and the incremental refresh processes a tight delta. With an externally-managed Iceberg base table, change tracking is file-granular: a copy-on-write update that rewrites a data file marks the entire file as changed, so the refresh reprocesses every row in it, even if one row moved. On a table with frequent small updates and copy-on-write behavior, this inflates the change set dramatically and can make an “incremental” refresh behave like it’s doing far more work than the actual data change justifies. Snowflake’s documentation states it plainly: frequent copy-on-write operations on externally-managed Iceberg tables may impact incremental-refresh performance.

Then there’s the metadata reset. INSERT OVERWRITE on a base table — a common pattern for batch reloads — resets change-tracking metadata, and the next Dynamic Table refresh falls back to a full recomputation. If your ingestion rewrites tables wholesale, your downstream “incremental” pipeline isn’t incremental at all.

How v3 features rescue the change-tracking story

The good news is that v3 exists partly to solve this, and using its features deliberately is the difference between a fast lakehouse and a slow one.

Row lineage is the headline. In v3, tables track _row_id (a stable unique identifier assigned to each row) and _last_updated_sequence_number (the commit that last touched the row). This lets any compliant engine reliably match the same row across snapshots and detect row-level changes — native CDC in the format itself, not bolted on. Row lineage is supported for both Snowflake-managed and externally-managed v3 tables and underpins append-only and standard streams on Snowflake-managed v3 tables.

Primary keys with RELY are the pragmatic rescue for the INSERT OVERWRITE problem. If you declare a reliable primary key on the base table, Snowflake compares rows by key value instead of leaning on change-tracking columns — so even when a table is fully rewritten, it computes the minimal set of actual changes rather than reprocessing everything. This is also how you enable incremental refresh downstream of a full-refresh dynamic table, by giving Snowflake a stable identity to diff against. For append-only CDC, the QUALIFY ROW_NUMBER() OVER (PARTITION BY id ORDER BY updated_at DESC) = 1 pattern gives you latest-row-per-key with a derived unique key, handling out-of-order arrival without extra logic.

Deletion vectors replace v2’s positional deletes for merge-on-read, and they’re governed by heuristics worth knowing: Snowflake writes a deletion vector only if fewer than ~5% of a data file’s rows are deleted and the file exceeds ~1.6 MB; otherwise it rewrites the file (copy-on-write). You control the behavior with ICEBERG_MERGE_ON_READ_BEHAVIOR. The compatibility trap: if an external engine in your stack doesn’t yet understand v3 deletion vectors, you must set that parameter to 'DISABLED' to force copy-on-write, or the external engine will misread the table. Interoperability constrains you to the capabilities of the least capable engine that touches the table.

The gotchas nobody warns you about

Change tracking must be on, with non-zero Time Travel, on every underlying object. Incremental refresh silently depends on it. Snowflake will try to enable it automatically for incremental dynamic tables, but if you recreate a base object you must re-enable it — and a base object with Time Travel set to zero quietly breaks incrementality.

The GRANT syntax has a trap for dynamic Iceberg tables. To grant access to future dynamic Iceberg tables in a schema, you use GRANT … ON FUTURE ICEBERG TABLES without the DYNAMIC keyword. The intuitive ON FUTURE DYNAMIC ICEBERG TABLES does not cover them, so a reasonable-looking grant leaves new tables inaccessible.

Gen2 warehouses matter more than you’d expect. Snowflake’s Dynamic Table performance work — measured up to ~2.8x faster refresh over the past year — is specifically tied to Gen2 warehouses for patterns like top-level aggregates, QUALIFY row/rank = 1, cluster-by, and joins. If your incremental pipeline is on Gen1, you’re leaving a large multiple of refresh speed on the table before any Iceberg tuning.

Cross-region and cross-cloud tables bill for transfer. A Snowflake-managed Iceberg table whose external volume sits in a different region or cloud than your account incurs cross-region data-transfer charges under the DATA_LAKE transfer type. Keep external volumes in the same region as your account unless you have a deliberate DR reason not to.

A migration order that avoids the traps

Sequence matters. First, decide per table who owns writes — if an external engine writes, it’s externally-managed; if only Snowflake writes, Snowflake-managed — because that’s the irreversible-ish decision. Second, set your default Iceberg version to v3 and plan recreate-and-backfill for existing v2 tables rather than expecting an upgrade. Third, choose a coarse partition grain (validated against real query patterns, staying well under a few thousand partitions) and use clustering for finer pruning. Fourth, make change tracking deliberate: declare reliable primary keys where base tables get rewritten, lean on row lineage for CDC, and confirm change tracking plus non-zero Time Travel on every base object. Fifth, pin ICEBERG_MERGE_ON_READ_BEHAVIOR to match the least-capable engine that reads the table. Then move workloads to Gen2 warehouses and measure incremental-refresh times against your latency target before you call it done.

The one principle

“Open Iceberg lakehouse with low-latency Dynamic Tables” is true only when the write path, the partition cardinality, and the change-tracking granularity all line up — and by default they don’t. Migrating native tables to dynamic Iceberg v3 is a design exercise, not a format swap: decide who writes, partition coarsely, give Snowflake a stable row identity to diff against, and constrain merge-on-read to your least-capable engine. Get those four right and the lakehouse is genuinely fast and open. Get them wrong and you’ve built a slow data lake with extra steps, one full refresh at a time.

Related reading: Create dynamic Apache Iceberg tables (official docs) · Manage Iceberg tables: row lineage & deletion vectors · Snowflake Iceberg v3: When to Migrate · dbt State on Snowflake: Skip Unchanged Models · Dynamic Airflow DAGs via Snowflake Metadata

Most data engineers I talk to still store everything in Snowflake native format. It’s simple: load data, query data, done. But here’s what nobody’s talking about: if you’re querying that data from anywhere else — Spark, Databricks, even just a local Python script — you’re paying a hidden “data tax.” Redundant storage, egress fees, ETL pipeline complexity. For Fortune 500 companies, that tax runs $2 million to $7 million a year. And Snowflake’s new Apache Iceberg v3 support (GA May 2026) actually changes the math. But migrating is a choice, not a reflex — and there are specific gotchas that’ll bite you if you don’t plan right.

The honest decision tree: migrate if you’re paying egress fees or running multi-engine queries.

TL;DR

→ Apache Iceberg v3 is GA on Snowflake (May 2026). New features: deletion vectors (10x faster DML), row lineage for CDC, VARIANT type for semi-structured data, nanosecond timestamps, default column values.

→ Month-to-month costs are roughly equal to native Snowflake tables (compute identical, storage ~$23/TB native vs ~$0.023/GB S3, negligible difference).

→ Migrate if: (a) you query from Spark/Databricks (egress fees kill you), (b) you’re paying >$500/month for Snowflake storage, (c) you want single copy of truth across multiple engines.

→ Don’t migrate if: you only query from Snowflake, storage bill is small, and you’re not building a multi-engine architecture.

→ New gotcha: You can’t upgrade v2 tables in-place to v3. No writing to v3 tables via external engines (Spark) yet. External engine compaction gets billed starting May 21, 2026.

→ Real win: Snowflake Storage for Iceberg (GA April 2026) means you don’t manage S3 buckets. Snowflake handles it, with Fail-safe recovery built in.

→ The “data tax” of $2M–$7M annually on Fortune 500 costs more than Iceberg migration ever will.

The mental model that’s keeping you locked in

Here’s the picture most teams hold: Snowflake stores data. We query it in Snowflake. Done. Native tables, simple syntax, life is easy. And if you only query in Snowflake, that model works fine. You get the speed, the simplicity, the integration with dbt, the Time Travel.

But the moment you have data living in two systems — Snowflake for reporting, Spark for ML training, Databricks for a BI tool, even just a DuckDB instance on your laptop — you’ve broken the simple model. Now you have two copies of the data, or worse, a pipeline that’s constantly syncing between them. You’re paying Snowflake egress fees to get data out ($0.02 per GB across regions, $0.08 between clouds). You’re rebuilding the same transformation logic in both systems. You’re managing schema evolution in two places. The complexity compounds.

Iceberg was built to solve exactly this. One copy of the data, on open cloud storage (S3, Azure Blob, GCS), readable by any engine that supports the Iceberg format. Snowflake, Spark, Databricks, Trino, DuckDB. All of them see the same table, the same schema, the same snapshot. No replication, no egress fees, no syncing.

But Iceberg isn’t free. It trades simplicity for flexibility. And for teams that genuinely don’t need that flexibility, native tables are still the right call.

The hidden cost of locking data into proprietary formats. For large teams, it’s massive.

What changed in Iceberg v3, and why it matters

Iceberg v2 shipped in 2023 and covered the basics: open format, ACID transactions, schema evolution, snapshots. v3 (released June 2025, GA on Snowflake May 7, 2026) added seven new capabilities. Only three actually change how you’d use it.

Deletion vectors. In v2, if you deleted or updated a row, Iceberg had to rewrite the entire data file (copy-on-write). Slow and expensive. v3 adds deletion vectors — a separate, small metadata file that marks rows as deleted without touching the original data. Result: 10x faster DML operations on large tables. If you’re doing frequent small updates (common in streaming ingestion), v3 matters.

Row lineage. v3 tracks which rows were inserted, updated, or deleted with metadata fields (_row_id, _last_updated_sequence_number). This is how Snowflake implements change data capture (CDC) without external tooling. A Dynamic Iceberg Table can now refresh incrementally on only the rows that changed, not the whole partition. Critical for SCD2 and CDC pipelines.

VARIANT type. v2 forced you to choose: store JSON as a string (slow parsing at query time) or explode it into a wide schema (thousands of nullable columns, query disasters). v3 adds native VARIANT support, and Snowflake automatically shreds it (extracts nested fields and indexes them) at write time. Query performance on semi-structured data jumps dramatically. This alone is why observability platforms are betting on Iceberg.

The other four (default column values, geometry/geography types, nanosecond timestamps, partition transform improvements) are niche. Don’t worry about them unless you hit them.

The cost math: Native vs Iceberg in real dollars

Let’s be honest: most articles skip the cost comparison and jump to “Iceberg is cheaper!” It usually isn’t, month-to-month. Here’s why.

The real numbers. On a month-to-month basis, they’re nearly identical. The wins come from elsewhere.

For a typical 10 TB table with 1,000 queries per month (small-to-medium workload):

Native Snowflake: Compute 2,000 credits ($6,000) + Snowflake storage 10TB at $23/TB ($230) = $6,230/month.

Iceberg (Snowflake-managed storage, GA April 2026): Compute 2,000 credits ($6,000) + S3 storage 10TB (10,240 GB × $0.023/GB = $235) + compaction bundled ($0) = $6,235/month.

Basically the same. Where Iceberg wins is not in monthly costs. It wins in:

Egress fees. If you query that 10 TB table from a Databricks cluster once a month, native Snowflake costs 10,000 GB × $0.08/GB (cross-cloud egress) = $800. Iceberg: $0. Over a year, that’s $9,600. At any real-world scale (multi-engine queries), egress dominates.

No data duplication. If you’re currently syncing data between Snowflake and Databricks (ETL pipeline, manual export, Fivetran), that pipeline costs money too. Shared Iceberg table means you stop paying to move the data. One table, multiple readers.

Storage simplicity. With Snowflake Storage for Iceberg (new, April 2026), you don’t manage S3 buckets yourself. Snowflake handles encryption, replication, Fail-safe recovery. You save the operational tax of bucket management, lifecycle policies, and debugging storage issues.

So here’s the honest scorecard:

For Snowflake-only users: Native tables win. Simpler, no migration pain, costs are identical.

For multi-engine shops (Snowflake + Spark + Databricks): Iceberg wins. Egress fees alone justify the migration, and you get single source of truth as a bonus.

The gotchas that will hurt your migration

You can’t upgrade v2 tables to v3 in-place. There’s no ALTER TABLE ... SET ICEBERG_VERSION = 3. To get v3, you have to CREATE a new table. That means copying data (compute cost, time), repointing your queries, and hoping nothing breaks downstream. On large tables, this is a multi-day operation.

External engines can’t write v3 tables yet. You can read v3 tables from Spark, Trino, DuckDB, all day. But writing is blocked. Snowflake says it’s “planned,” but if you’re building a shared Iceberg table that Spark needs to update, you’re stuck on v2. This is a major limitation if you’re counting on true multi-engine write access.

Compaction gets billed starting May 21, 2026. When an external engine writes to an Iceberg table (via Spark, Trino, etc.), it creates small data files. Snowflake’s compaction automatically consolidates them into bigger files for query performance. Until May 21, that was free. Now it costs credits. Budget for ongoing compaction maintenance if you have heavy external write workloads.

⚠️ Don’t convert cloned tables with vended credentials. If you clone a native Snowflake table and then convert it to Iceberg, you can’t write to it with vended credentials (external query engine creds). You’d have to connect the external engine directly to your S3 bucket, defeating the whole point. Create the Iceberg table fresh if you’re using vended creds.

Schema changes are cheap but metadata bloat is real. Iceberg tracks every schema change as a separate metadata version. On tables with thousands of ALTER COLUMN operations, metadata can get unwieldy. Compact your metadata regularly with CALL SYSTEM$OPTIMIZE(...).

The mistakes teams make when migrating

1. Migrating for the wrong reason. “Everyone’s talking about Iceberg, so we should move.” Wrong. Migrate only if you have a concrete use case: egress fees, multi-engine queries, or storage cost >$500/month. Otherwise you’re trading simplicity for nothing.

2. Not testing external engine read performance first. Iceberg’s query performance depends heavily on your cloud setup, partitioning strategy, and how many small files are sitting around. Test Spark/Databricks queries on a small Iceberg table before migrating your 100 TB production table. You might find that your workload is slower on Iceberg, not faster.

3. Assuming v3 is backward-compatible with v2. It’s not. Engines that only understand v2 (like older Spark runtimes, Trino versions) will fail on v3 tables. Check that every tool in your stack supports v3 *before* upgrading. v2 → v3 is one-way; there’s no downgrade.

4. Ignoring the partition evolution story. Iceberg lets you change your partitioning scheme without rewriting the whole table. It’s a huge feature, but it’s also easy to mess up. Bad partitioning (e.g., partitioning by a column with 10 million distinct values) creates a partition explosion. Get your partitioning right before you migrate, not after.

5. Migrating everything at once. Pick one critical table, migrate it, test multi-engine queries for a month, then move the rest. Iceberg is mature enough for production, but it’s not old enough that every edge case is documented. Be intentional.

When to actually migrate: The real decision

Stop and ask yourself: Do you actually need Iceberg?

Yes, if: You query the same data from Snowflake and Spark/Databricks. You’re paying egress fees. You have data warehouses in multiple clouds and want to query across them. You’re building a data lakehouse and want to ditch proprietary formats.

No, if: You only query from Snowflake. Your storage bill is <$500/month. You’re using Snowflake’s Time Travel, zero-copy clones, and other native features heavily. You don’t need to share data with other engines.

For most teams, the answer is no. And that’s okay. Native Snowflake tables are extremely good. Simple, fast, well-integrated with dbt. There’s no shame in staying native.

But for teams hitting the “data tax” — redundant copies, egress fees, multi-engine complexity — Iceberg v3 actually delivers. The gotchas are real, but they’re manageable. The cost savings are modest month-to-month, but the flexibility is transformative.

The one principle that matters

Interoperability beats simplicity when you’re already paying for fragmentation. If your current architecture already costs you $800/month in egress, $300/month in ETL pipelines, and engineering time chasing sync issues, Iceberg’s “complexity” is actually a simplification. You’re not adding complexity; you’re replacing it with a standard.

If you’re simple and integrated today, stay there. Don’t pay the cost of flexibility you don’t need. But if you’re paying the data tax, Iceberg’s math changes fast.

Related reading: Snowflake Apache Iceberg tables (official docs) · Snowflake Time Travel: The Real Architecture · Snowflake Optima: 15x Faster Queries at Zero Cost · Query Snowflake in DuckDB and Cut Costs