Blog

  • Snowflake Iceberg V3: When to Actually Migrate(vs Native Tables)

    Snowflake Iceberg V3: When to Actually Migrate(vs Native Tables)

    Most data engineers I talk to still store everything in Snowflake native format. It’s simple: load data, query data, done. But here’s what nobody’s talking about: if you’re querying that data from anywhere else — Spark, Databricks, even just a local Python script — you’re paying a hidden “data tax.” Redundant storage, egress fees, ETL pipeline complexity. For Fortune 500 companies, that tax runs $2 million to $7 million a year. And Snowflake’s new Apache Iceberg v3 support (GA May 2026) actually changes the math. But migrating is a choice, not a reflex — and there are specific gotchas that’ll bite you if you don’t plan right.

    The honest decision tree: migrate if you’re paying egress fees or running multi-engine queries.

    TL;DR

    → Apache Iceberg v3 is GA on Snowflake (May 2026). New features: deletion vectors (10x faster DML), row lineage for CDC, VARIANT type for semi-structured data, nanosecond timestamps, default column values.

    → Month-to-month costs are roughly equal to native Snowflake tables (compute identical, storage ~$23/TB native vs ~$0.023/GB S3, negligible difference).

    → Migrate if: (a) you query from Spark/Databricks (egress fees kill you), (b) you’re paying >$500/month for Snowflake storage, (c) you want single copy of truth across multiple engines.

    → Don’t migrate if: you only query from Snowflake, storage bill is small, and you’re not building a multi-engine architecture.

    → New gotcha: You can’t upgrade v2 tables in-place to v3. No writing to v3 tables via external engines (Spark) yet. External engine compaction gets billed starting May 21, 2026.

    → Real win: Snowflake Storage for Iceberg (GA April 2026) means you don’t manage S3 buckets. Snowflake handles it, with Fail-safe recovery built in.

    → The “data tax” of $2M–$7M annually on Fortune 500 costs more than Iceberg migration ever will.

    The mental model that’s keeping you locked in

    Here’s the picture most teams hold: Snowflake stores data. We query it in Snowflake. Done. Native tables, simple syntax, life is easy. And if you only query in Snowflake, that model works fine. You get the speed, the simplicity, the integration with dbt, the Time Travel.

    But the moment you have data living in two systems — Snowflake for reporting, Spark for ML training, Databricks for a BI tool, even just a DuckDB instance on your laptop — you’ve broken the simple model. Now you have two copies of the data, or worse, a pipeline that’s constantly syncing between them. You’re paying Snowflake egress fees to get data out ($0.02 per GB across regions, $0.08 between clouds). You’re rebuilding the same transformation logic in both systems. You’re managing schema evolution in two places. The complexity compounds.

    Iceberg was built to solve exactly this. One copy of the data, on open cloud storage (S3, Azure Blob, GCS), readable by any engine that supports the Iceberg format. Snowflake, Spark, Databricks, Trino, DuckDB. All of them see the same table, the same schema, the same snapshot. No replication, no egress fees, no syncing.

    But Iceberg isn’t free. It trades simplicity for flexibility. And for teams that genuinely don’t need that flexibility, native tables are still the right call.

    The hidden cost of locking data into proprietary formats. For large teams, it’s massive.

    What changed in Iceberg v3, and why it matters

    Iceberg v2 shipped in 2023 and covered the basics: open format, ACID transactions, schema evolution, snapshots. v3 (released June 2025, GA on Snowflake May 7, 2026) added seven new capabilities. Only three actually change how you’d use it.

    Deletion vectors. In v2, if you deleted or updated a row, Iceberg had to rewrite the entire data file (copy-on-write). Slow and expensive. v3 adds deletion vectors — a separate, small metadata file that marks rows as deleted without touching the original data. Result: 10x faster DML operations on large tables. If you’re doing frequent small updates (common in streaming ingestion), v3 matters.

    Row lineage. v3 tracks which rows were inserted, updated, or deleted with metadata fields (_row_id, _last_updated_sequence_number). This is how Snowflake implements change data capture (CDC) without external tooling. A Dynamic Iceberg Table can now refresh incrementally on only the rows that changed, not the whole partition. Critical for SCD2 and CDC pipelines.

    VARIANT type. v2 forced you to choose: store JSON as a string (slow parsing at query time) or explode it into a wide schema (thousands of nullable columns, query disasters). v3 adds native VARIANT support, and Snowflake automatically shreds it (extracts nested fields and indexes them) at write time. Query performance on semi-structured data jumps dramatically. This alone is why observability platforms are betting on Iceberg.

    The other four (default column values, geometry/geography types, nanosecond timestamps, partition transform improvements) are niche. Don’t worry about them unless you hit them.

    The cost math: Native vs Iceberg in real dollars

    Let’s be honest: most articles skip the cost comparison and jump to “Iceberg is cheaper!” It usually isn’t, month-to-month. Here’s why.

    Two-column cost breakdown. Native Snowflake: 2,000 credits at $3 = $6,000 compute, $23/TB storage = $230, total $6,230/month. Iceberg (Snowflake managed): same $6,000 compute, S3 at $0.023/GB = $235 storage, bundled compaction = $0, total $6,235/month. Verdict: same cost, but Iceberg enables multi-engine and zero egress.

    The real numbers. On a month-to-month basis, they’re nearly identical. The wins come from elsewhere.

    For a typical 10 TB table with 1,000 queries per month (small-to-medium workload):

    Native Snowflake: Compute 2,000 credits ($6,000) + Snowflake storage 10TB at $23/TB ($230) = $6,230/month.

    Iceberg (Snowflake-managed storage, GA April 2026): Compute 2,000 credits ($6,000) + S3 storage 10TB (10,240 GB × $0.023/GB = $235) + compaction bundled ($0) = $6,235/month.

    Basically the same. Where Iceberg wins is not in monthly costs. It wins in:

    Egress fees. If you query that 10 TB table from a Databricks cluster once a month, native Snowflake costs 10,000 GB × $0.08/GB (cross-cloud egress) = $800. Iceberg: $0. Over a year, that’s $9,600. At any real-world scale (multi-engine queries), egress dominates.

    No data duplication. If you’re currently syncing data between Snowflake and Databricks (ETL pipeline, manual export, Fivetran), that pipeline costs money too. Shared Iceberg table means you stop paying to move the data. One table, multiple readers.

    Storage simplicity. With Snowflake Storage for Iceberg (new, April 2026), you don’t manage S3 buckets yourself. Snowflake handles encryption, replication, Fail-safe recovery. You save the operational tax of bucket management, lifecycle policies, and debugging storage issues.

    So here’s the honest scorecard:

    For Snowflake-only users: Native tables win. Simpler, no migration pain, costs are identical.

    For multi-engine shops (Snowflake + Spark + Databricks): Iceberg wins. Egress fees alone justify the migration, and you get single source of truth as a bonus.

    The gotchas that will hurt your migration

    You can’t upgrade v2 tables to v3 in-place. There’s no ALTER TABLE ... SET ICEBERG_VERSION = 3. To get v3, you have to CREATE a new table. That means copying data (compute cost, time), repointing your queries, and hoping nothing breaks downstream. On large tables, this is a multi-day operation.

    External engines can’t write v3 tables yet. You can read v3 tables from Spark, Trino, DuckDB, all day. But writing is blocked. Snowflake says it’s “planned,” but if you’re building a shared Iceberg table that Spark needs to update, you’re stuck on v2. This is a major limitation if you’re counting on true multi-engine write access.

    Compaction gets billed starting May 21, 2026. When an external engine writes to an Iceberg table (via Spark, Trino, etc.), it creates small data files. Snowflake’s compaction automatically consolidates them into bigger files for query performance. Until May 21, that was free. Now it costs credits. Budget for ongoing compaction maintenance if you have heavy external write workloads.

    ⚠️ Don’t convert cloned tables with vended credentials. If you clone a native Snowflake table and then convert it to Iceberg, you can’t write to it with vended credentials (external query engine creds). You’d have to connect the external engine directly to your S3 bucket, defeating the whole point. Create the Iceberg table fresh if you’re using vended creds.

    Schema changes are cheap but metadata bloat is real. Iceberg tracks every schema change as a separate metadata version. On tables with thousands of ALTER COLUMN operations, metadata can get unwieldy. Compact your metadata regularly with CALL SYSTEM$OPTIMIZE(...).

    The mistakes teams make when migrating

    1. Migrating for the wrong reason. “Everyone’s talking about Iceberg, so we should move.” Wrong. Migrate only if you have a concrete use case: egress fees, multi-engine queries, or storage cost >$500/month. Otherwise you’re trading simplicity for nothing.

    2. Not testing external engine read performance first. Iceberg’s query performance depends heavily on your cloud setup, partitioning strategy, and how many small files are sitting around. Test Spark/Databricks queries on a small Iceberg table before migrating your 100 TB production table. You might find that your workload is slower on Iceberg, not faster.

    3. Assuming v3 is backward-compatible with v2. It’s not. Engines that only understand v2 (like older Spark runtimes, Trino versions) will fail on v3 tables. Check that every tool in your stack supports v3 *before* upgrading. v2 → v3 is one-way; there’s no downgrade.

    4. Ignoring the partition evolution story. Iceberg lets you change your partitioning scheme without rewriting the whole table. It’s a huge feature, but it’s also easy to mess up. Bad partitioning (e.g., partitioning by a column with 10 million distinct values) creates a partition explosion. Get your partitioning right before you migrate, not after.

    5. Migrating everything at once. Pick one critical table, migrate it, test multi-engine queries for a month, then move the rest. Iceberg is mature enough for production, but it’s not old enough that every edge case is documented. Be intentional.

    When to actually migrate: The real decision

    Stop and ask yourself: Do you actually need Iceberg?

    Yes, if: You query the same data from Snowflake and Spark/Databricks. You’re paying egress fees. You have data warehouses in multiple clouds and want to query across them. You’re building a data lakehouse and want to ditch proprietary formats.

    No, if: You only query from Snowflake. Your storage bill is <$500/month. You’re using Snowflake’s Time Travel, zero-copy clones, and other native features heavily. You don’t need to share data with other engines.

    For most teams, the answer is no. And that’s okay. Native Snowflake tables are extremely good. Simple, fast, well-integrated with dbt. There’s no shame in staying native.

    But for teams hitting the “data tax” — redundant copies, egress fees, multi-engine complexity — Iceberg v3 actually delivers. The gotchas are real, but they’re manageable. The cost savings are modest month-to-month, but the flexibility is transformative.

    The one principle that matters

    Interoperability beats simplicity when you’re already paying for fragmentation. If your current architecture already costs you $800/month in egress, $300/month in ETL pipelines, and engineering time chasing sync issues, Iceberg’s “complexity” is actually a simplification. You’re not adding complexity; you’re replacing it with a standard.

    If you’re simple and integrated today, stay there. Don’t pay the cost of flexibility you don’t need. But if you’re paying the data tax, Iceberg’s math changes fast.

    Related reading: Snowflake Apache Iceberg tables (official docs) · Snowflake Time Travel: The Real Architecture · Snowflake Optima: 15x Faster Queries at Zero Cost · Query Snowflake in DuckDB and Cut Costs

  • Snowflake Query Execution: what really happens under the hood

    Snowflake Query Execution: what really happens under the hood

    Ask ten data engineers what happens when you run a query in Snowflake and most of them will tell you the same thing: the warehouse runs it. SQL goes in, the warehouse chews on it, results come out. Clean, simple, and just wrong enough to cost you money. The truth is that Snowflake query execution is a trip through three separate layers, and the one you actually pay for is the last to get involved — if it gets involved at all.

    Flowchart of the life of a Snowflake query: query submitted, cloud services layer parses and prunes, result cache check returns a cached result on a hit with zero credits, and on a miss the virtual warehouse runs it across MPP nodes before fetching micro-partitions from storage and returning results.

    The path every query takes — and the shortcut a cache hit gets to skip compute entirely.

    TL;DR

    → Running a query isn’t one step. It’s a trip through three layers that scale independently, and the compute you pay for is the last one to wake up.

    → The cloud services layer does the thinking first — parses, plans, and prunes — using per-partition metadata to throw out data that can’t match your filter before a single byte is read.

    → If the exact same query ran in the last 24 hours and the data hasn’t moved, you get a cached result for free. No warehouse. No credits. Milliseconds.

    → Only on a cache miss does a virtual warehouse actually spin up and crunch the query across its nodes in parallel.

    → Warehouses keep recently-read micro-partitions on local SSD, but that cache vanishes the second the warehouse suspends.

    → “Identical” is brutally literal. A stray table alias, lowercase keywords, or a RANDOM() call quietly knocks you off the cache and back onto billed compute.

    → When a query is slow, it’s usually scanning too much, not computing too much. A bigger warehouse fixes the second problem and never the first.

    The mental model that’s quietly costing you money

    Here’s the reframe that changes how you think about every query: the virtual warehouse is the last thing to get involved, and plenty of queries never touch it at all. Almost everything that decides whether your query is fast or slow, cheap or expensive, happens in a layer you probably weren’t even picturing.

    Get this wrong and you end up doing what I’ve watched teams do a dozen times: a dashboard feels sluggish, someone bumps the warehouse from Medium to Large, it gets a little faster, everyone moves on. The bill goes up. The query was never compute-bound in the first place. They paid more to run the same broken query faster.

    So let’s actually follow a query, from the moment you hit run to the moment rows come back.

    The three layers a query passes through

    Snowflake splits into three layers, and the thing to internalize is that they scale completely independently. That’s not a trivia fact — it’s the reason the whole platform behaves the way it does. Every query touches all three. Some only touch the first.

    Diagram of Snowflake’s three layers: the cloud services layer handling parse, optimize, prune and result cache at the top; the compute layer with virtual warehouse MPP nodes in the middle; and the storage layer holding micro-partitions in S3, Azure Blob or GCS at the bottom, with a query entering the top and a cached result returning from it

    The cloud services layer is the brain. The warehouse is just the muscle.

    At the top sits the cloud services layer — the brain. Logins, sessions, parsing, query planning, transaction coordination, all the metadata: it lives here. Underneath that is the compute layer, made up of virtual warehouses. These are the MPP clusters that do the actual SQL grunt work. And at the bottom is storage: your data sitting in cloud object storage (S3, Azure Blob, or GCS) as immutable, compressed, columnar micro-partitions.

    The magic is that none of these share resources with each other. You can resize compute without moving a single byte of storage. You can point ten warehouses at the same table and none of them slow the others down. Keep that in your back pocket, because it’s exactly why the “just make the warehouse bigger” instinct fails so often.

    Step 1: The cloud services layer does the thinking

    Your query lands, and before any table data gets read, the cloud services layer goes to work.

    It parses the SQL, and the cost-based optimizer builds a plan. Then it does the single most important thing for performance, and it does it for free: partition pruning. Every micro-partition carries metadata, including the min and max value for each column. So when you write WHERE order_date = '2026-06-01', the optimizer reads that metadata first and skips every partition whose range can’t possibly hold a matching row. Ten thousand partitions in the table, three that actually get scanned. No index. No tuning. You didn’t do anything.

    This is also why a query can come back in milliseconds with no warehouse at all. Run a SELECT COUNT(*) or a MAX() and Snowflake often answers straight from the metadata cache — you’ll see a lone METADATA-BASED RESULT step in the query profile and zero compute on the bill.

    How good the pruning is comes down to how well your data is naturally ordered, which is the whole point of clustering and Snowflake’s automatic optimization features.

    Step 2: The result cache check, before any compute

    Now, still before any warehouse gets involved, cloud services checks the result cache. Identical query in the last 24 hours, data hasn’t changed since? Snowflake just hands back the stored result from the cloud services layer. Nothing resumes. Nothing gets billed. The answer is basically instant.

    This is the cheapest query you’ll ever run, and it’s the reason refreshing a dashboard a second time costs nothing. Here’s the part that surprises people: every time that cached result gets reused, the 24-hour clock resets. Keep hitting it and a result can stay alive for up to 31 days. A popular dashboard query can effectively sit in cache for a month and never cost a credit.

    Which brings us to the catch. “Identical” is doing a lot of work in that sentence, and it’s the thing nobody reads about until they’re staring at a bill wondering where the money went.

    Step 3: The virtual warehouse finally runs it

    Cache miss. Now — and only now — the query goes to a virtual warehouse. If it was asleep, it resumes (a second or two, usually). The warehouse is a little cluster of compute nodes, and this is where massively parallel processing earns its name: the surviving micro-partitions get split across the nodes and cores, and the filters, joins, and aggregations all happen in parallel.

    This is the layer with a meter running. You’re billed by the second, with a 60-second minimum every time a warehouse starts or resumes. That minimum trips people up constantly — a warehouse that keeps flickering on and off for tiny one-off queries can quietly cost more than one you just leave warm, because every single resume restarts that 60-second clock. It’s also the only layer where resizing actually helps, and that matters less often than you’d think.

    Step 4: Storage, and the cache that disappears

    The warehouse goes and fetches the micro-partitions that survived pruning. Each micro-partition holds 50–500 MB of uncompressed data in a columnar format, so only the columns you actually asked for get read. Ask for two columns out of fifty and you pay to read two.

    And there’s a second cache down here. As the warehouse pulls partitions from remote storage, it stashes them on local SSD. Next query on that same warehouse that needs the same data? It reads from SSD instead of making the round trip to object storage, which is a lot faster. This is what people mean when they say a warehouse “warms up.”

    The catch — and there’s always a catch — is that this cache gets wiped the instant the warehouse suspends. That’s the real trade-off hiding inside your AUTO_SUSPEND setting. Suspend fast and you stop paying for idle time but you toss the warm cache. Leave it running and you keep the cache but pay for the idle. A lot of teams land on 60 seconds, but honestly the right answer depends entirely on how often your queries actually fire.

    Three caches, one comparison to bookmark

    There are three caches, they live in different layers, and mixing them up is behind half the “wait, why was that slow?” conversations I’ve ever had.

    CacheLayerWhat it storesSurvives suspend?Compute cost
    Result cacheCloud servicesFinal query result setsYes (24h, resets on reuse, 31-day max)None
    Metadata cacheCloud servicesRow counts, min/max, distinct countsYesNone
    Local disk (warehouse) cacheComputeMicro-partitions read from storageNo — purged on suspendWarehouse already running

    The two cloud-services caches are shared across the whole account — every warehouse, every user benefits. The local disk cache belongs to one warehouse and dies with it. If you want the gory details with real query-profile screenshots, Snowflake’s own community piece on caching is the place to go.

    The cache rules nobody warns you about

    The result cache only kicks in when Snowflake decides the new query is identical to the old one, and “identical” is far more literal than anyone expects. Snowflake’s own docs spell it out: add a table alias, or just retype your keywords in lowercase, and you miss the cache. The query goes to billed compute and you never get a warning.

    It also bails on the cache if the query contains a non-deterministic function — UUID_STRINGRANDOMRANDSTR are the usual suspects — or an external function, or if it reads from a hybrid table. And the role running it needs privileges on every table involved.

    ⚠️ What this means in practice: if you want your dashboards and scheduled jobs riding the cache, lock down the exact text your BI tool emits. Random aliasing, dynamic comments, an injected session variable, a stray current_timestamp() — any of it silently drops you off the cache, and you start paying to recompute answers you already had. The Query Cost Estimator is a quick way to put a number on what those misses are costing you.

    Why a bigger warehouse usually isn’t the answer

    This is the most expensive misunderstanding in all of Snowflake, so it’s worth being blunt about it. A query is slow for one of two reasons: it’s scanning too much data, or it’s doing too much actual computation. Resizing the warehouse only helps the second one.

    When the real problem is bad pruning — Snowflake had to scan most of the table because the data wasn’t laid out in a way it could skip — a bigger warehouse just reads the same mountain of data faster and charges you more per second for the privilege. The fix isn’t more nodes. It’s better pruning: clustering, a tighter filter, a smarter query shape. The query profile tells you which world you’re in. Look at partitions scanned versus partitions total. If you’re chewing through 95% of them, no warehouse size on earth is going to save you.

    And sometimes the honest answer to a giant, I/O-heavy scan isn’t a bigger Snowflake warehouse at all — it’s moving that work somewhere cheaper, which is the whole idea behind running the query in DuckDB instead. If you do decide a resize is warranted, the warehouse sizing guide will at least keep you from overshooting.

    What I actually check when a query misbehaves

    When something’s slow or weirdly expensive, here’s where I start. All of it comes out of the query profile and INFORMATION_SCHEMA — no guessing, no vibes.

    -- Did your last query hit the result cache?
    -- A cache hit shows bytes_scanned = 0 and a near-zero execution time.
    SELECT query_id,
           query_text,
           bytes_scanned,
           percentage_scanned_from_cache,
           execution_time / 1000 AS exec_seconds
    FROM TABLE(information_schema.query_history())
    ORDER BY start_time DESC
    LIMIT 10;
    
    -- Turn the result cache off for honest benchmarking (this session only)
    ALTER SESSION SET USE_CACHED_RESULT = FALSE;
    -- ...run your test queries...
    ALTER SESSION UNSET USE_CACHED_RESULT;
    
    -- Hunt down your worst pruning offenders
    SELECT query_id,
           partitions_scanned,
           partitions_total,
           ROUND(100 * partitions_scanned / NULLIF(partitions_total, 0), 1) AS pct_scanned,
           bytes_scanned / 1e9 AS gb_scanned
    FROM snowflake.account_usage.query_history
    WHERE partitions_total > 0
    ORDER BY bytes_scanned DESC
    LIMIT 20;
    
    -- Balance warm cache against idle cost
    ALTER WAREHOUSE analytics_wh SET AUTO_SUSPEND = 60;  -- seconds

    The number to stare at is pct_scanned in that third query. High on your slow queries? You’ve got a pruning problem, and that’s where your time should go — not the warehouse dropdown.

    What this actually costs you

    Architecture is interesting, but the bill is what makes it matter. So let’s put real numbers on it. Credit prices swing with your edition, cloud, and region, so treat these as illustrative — drop your own rate into the cost calculator for the exact figure.

    Picture one ordinary dashboard query. It runs on a Medium warehouse (4 credits an hour, call it roughly $3 a credit), and it fires 100 times a day because that’s how often people open the dashboard. Here’s how the same query plays out depending on whether the layers are doing their job:

    ScenarioWhat actually runs~Cost / day~Cost / month
    No cache, bad pruning (30s/run, scans ~95% of partitions)100 full compute runs~$10~$300
    Result cache hits 90% of the time10 compute runs, 90 free cached returns~$1~$30
    Cache + good pruning (3s/run, scans ~5%)10 runs at a tenth of the scan~$0.10~$3

    Same query. Same data. Roughly $300 a month versus $3, and the only difference is whether you let the result cache and pruning do what they’re built to do. Now multiply that across a dashboard with thirty queries on it, and you can see how a Snowflake bill quietly triples without anyone writing a single new query.

    The gotchas nobody warns you about

    The 60-second minimum punishes spiky workloads. A query that finishes in two seconds still bills a full minute if the warehouse had to resume to run it. Set AUTO_SUSPEND too aggressively on a workload that fires a query every couple of minutes and you’ll pay more in cold-start minimums than you ever saved on idle time. Aggressive suspend is not automatically cheaper.

    One write nukes the whole result cache. The cache is invalidated by any change to the underlying table, not just changes to the rows your query touched. A single late-arriving record at 2 a.m. quietly wipes the cached result for every dashboard query built on that table, and tomorrow morning they all run on compute again. If your “cached” dashboard mysteriously costs money some mornings, this is usually why.

    Cloud services is free, right up until it isn’t. The parsing, planning, and pruning in the cloud services layer is free — but only while it stays under 10% of your daily warehouse compute. Hammer it with thousands of tiny metadata queries or relentless INFORMATION_SCHEMA polling and you cross that line, and Snowflake starts billing the overage. Most teams never hit it; the ones running huge volumes of trivial queries do, and they never see it coming.

    The local cache doesn’t follow you between warehouses. That warm SSD cache belongs to one specific warehouse. Run your ETL on one warehouse and your reporting on another — sensible workload isolation — and the reporting warehouse gets exactly zero benefit from the partitions ETL just pulled. Each warehouse warms its own cache from cold.

    SELECT * throws away the columnar advantage. Snowflake only reads the columns you ask for. Ask for all of them and you pay to read all of them, even when the dashboard displays three. In a wide table that’s the difference between scanning a few columns and dragging the entire row off storage.

    The mistakes that quietly drain the budget

    Almost every overspending Snowflake account I’ve looked at is making some combination of these five:

    1. Leaving AUTO_SUSPEND at the 600-second default. Ten full minutes of paid idle after every burst of activity. On a warehouse that’s used in short bursts, that idle time can dwarf the actual query time. Most analytics warehouses are fine at 60 seconds.

    2. One giant warehouse for everything. Pile ETL, ad-hoc analysis, and dashboards onto a single warehouse and they fight over the cache and the compute. Separate warehouses per workload keep each cache warm and each workload predictable.

    3. Scaling up when the real fix is pruning. The reflex we opened with. If pct_scanned is high, a bigger warehouse just burns more credits reading the same data. Fix the layout, not the size.

    4. Letting the BI tool emit sloppy SQL. Inconsistent aliases, injected timestamps, and per-user comments mean “the same” dashboard query is never byte-for-byte identical, so it never reuses the result cache. Standardize what the tool sends.

    5. Over-clustering a hot table. Automatic clustering isn’t free — it spends credits reorganizing data in the background. On a table that’s written constantly, that background churn can cost more than the queries it speeds up. Cluster the tables you read far more than you write.

    The one principle to take away

    If you remember nothing else, remember this: scan less, don’t compute faster. Snowflake decides almost everything about a query’s speed and cost before the warehouse ever wakes up — in the pruning, in the caches, in how your data is laid out. The warehouse size is the last lever you should reach for, not the first. Get the layers above it working and most “we need a bigger warehouse” conversations simply stop happening.

    Related reading: Snowflake Time Travel: The Real Architecture · Snowflake Optima: 15x Faster Queries at Zero Cost · Query Snowflake in DuckDB and Cut Costs · Snowflake Streams & Tasks: SCD2 Pipeline Guide

  • How the Warehouse Cache Actually Works in Snowflake

    How the Warehouse Cache Actually Works in Snowflake

    A dashboard that ran in four seconds on Monday took nineteen seconds on Tuesday. Same query, same data, same warehouse size. I spent the better part of an hour convinced Snowflake was having a bad day, checking the status page, refreshing query history, muttering about “platform issues” in our team Slack — before I noticed our DevOps script had quietly added an aggressive auto-suspend policy the night before. The warehouse cache was getting wiped every single morning, and I’d built our entire “fast dashboard” reputation on a cache that reset itself before anyone showed up to work.

    TL;DR

    • Snowflake’s warehouse cache (local disk cache) stores raw compressed micro-partition data on each node’s SSD, not query results.
    • It’s separate from the result cache and metadata cache — three different caches, three different jobs.
    • Auto-suspending a warehouse wipes this cache completely. Resuming starts it cold every time.
    • You can verify it’s working with the ‘percentage scanned from cache’ field in query profile or ACCOUNT_USAGE.QUERY_HISTORY.
    • Multi-cluster warehouses don’t share this cache — a query routed to a new cluster starts cold even if a sibling cluster is warm.
    • You can’t manually size or pin it. The only real lever you control is the auto-suspend timer.

    That mistake is what got me actually reading how this thing works instead of just trusting that Snowflake would “handle it.” Turns out the warehouse cache isn’t magic, isn’t tunable, and isn’t the same thing as the result cache most people learn about first. Here’s what it actually is, where it lives, and how to tell when it’s helping you versus quietly costing you money.

    Diagram showing a cloud data warehouse architecture with three layers: Cloud services (result and metadata cache), compute layer with three SSD nodes, and a database storage layer with immutable micro-partitions. Arrows indicate data flow.

    Three caches, one name people use for all of them

    Snowflake has three distinct caching layers, and the confusion starts because people say “Snowflake caches my query” without specifying which one did the work. The result cache sits in the cloud services layer and stores entire finished query results — if the same exact SQL text runs again within 24 hours and the underlying data hasn’t changed, you get the answer back with bytes_scanned = 0 and zero compute cost. The metadata cache, also in cloud services, holds statistics about every micro-partition — min and max values per column, row counts — so Snowflake can decide which partitions to skip before it ever touches the data.

    The warehouse cache is the third one, and it’s the one this article is actually about. It lives on the local SSD of every node in a running virtual warehouse, and it stores raw, compressed micro-partition data — not query results, not aggregated answers, the actual columnar bytes that got pulled from remote storage to answer a scan.

    Why this distinction actually matters

    If you only know the result cache exists, you’ll misdiagnose a lot of performance issues. Change one character in a WHERE clause, add a comment, swap the role running the query — any of those bypass the result cache entirely, because it requires an exact text match. The warehouse cache doesn’t care about query text at all. It cares about which micro-partitions a query needs and whether those bytes are already sitting on a node’s SSD from a previous scan.

    What a micro-partition actually is

    You can’t understand the warehouse cache without understanding the unit it stores. When data lands in a Snowflake table, it gets automatically carved into micro-partitions — contiguous, immutable blocks holding somewhere between 50MB and 500MB of uncompressed data each, stored in a columnar format. There’s no manual partitioning scheme to design, no index to build. Snowflake just does this on every load.

    Each micro-partition carries its own metadata: minimum and maximum values for every column, which is exactly what the metadata cache is built from. When you filter a query on order_date > '2026-01-01', Snowflake checks that metadata first and skips any micro-partition whose max date falls before that threshold. That’s partition pruning, and it happens before a single byte gets pulled into the warehouse cache. Pruning decides what to read; the warehouse cache decides how fast a repeat read of the same partitions will be.

    The actual lifecycle of the warehouse cache

    Here’s the sequence that matters in practice. A warehouse resumes from suspended state with completely empty SSD — there’s nothing cached because the compute nodes assigned to it are freshly provisioned. The first query that touches a table has to pull every relevant micro-partition from remote cloud storage, which is the slowest tier in the whole architecture. As those partitions get read, they’re written to the local SSD cache as a side effect — not because you asked for caching, just because that’s what happens when a node reads remote data.

    The second query — if it touches the same micro-partitions and the warehouse is still running — can read those bytes off local SSD instead of going back to remote storage. This is meaningfully faster, and it’s why a sequence of similar queries against the same table speeds up the longer a warehouse stays warm. There’s no explicit “build the cache” step. Cache population is a byproduct of usage, which is exactly why a single cold query tells you almost nothing about real-world performance.

    Then the warehouse suspends, and it’s gone

    This is the part that bit me. When a warehouse auto-suspends, the compute nodes it was using get released back into Snowflake’s shared pool. The SSD on those specific nodes goes with them. When the warehouse resumes — even if it’s seconds later, even if it’s the exact same warehouse name — there’s no guarantee you get the same physical nodes back, and the cache starts from zero regardless. There’s no persistence, no “save state before suspending.” It’s just gone.

    Resizing a warehouse up or down does the same thing. A different size means a different set of nodes, which means different SSDs, which means the next queries run cold no matter how warm things were five minutes earlier.

    How to actually see this working

    Stop assuming and go look at it. Open the query profile for any query in Snowsight and check the IO statistics panel — there’s a field literally called percentage_scanned_from_cache. Run the same query twice in a row on a warm warehouse and watch that number jump from near 0% to something much higher on the second run. Suspend the warehouse, resume it, run the same query again, and watch it drop back to 0%. That’s the entire mechanism, visible in about ninety seconds of testing.

    For a wider view across your account, query history gives you the same field at scale. This is the query I run when someone asks “is our caching even helping”:

    Check your real cache hit rate (last 30 days, by warehouse)

    SELECT
        warehouse_name,
        COUNT(*) AS query_count,
        SUM(bytes_scanned) AS bytes_scanned,
        SUM(bytes_scanned * percentage_scanned_from_cache) AS bytes_from_cache,
        SUM(bytes_scanned * percentage_scanned_from_cache)
            / SUM(bytes_scanned) AS pct_scanned_from_cache
    FROM snowflake.account_usage.query_history
    WHERE start_time >= DATEADD(month, -1, CURRENT_TIMESTAMP())
      AND bytes_scanned > 0
    GROUP BY 1
    ORDER BY 5;

    A low percentage here on a warehouse running frequent, similar queries is a signal — either your auto-suspend timer is too aggressive for the workload, or the queries aren’t actually similar enough at the data level to benefit from a warm cache, even if the SQL looks similar to a human reading it.

    Result cache, warehouse cache, metadata cache — side by side

    Cache LayerWhere It LivesWhat It StoresCleared WhenCompute Cost
    Result CacheCloud Services layerFull query results24 hours of inactivity, or DDL on underlying tablesZero — no warehouse needed
    Warehouse CacheSSD on each compute nodeRaw compressed micro-partitionsWarehouse suspends, resizes, or node is replacedWarehouse must be running
    Metadata CacheCloud Services layerMin/max values, row counts, partition statsRarely — persists with the tableZero — used for pruning before scan

    The auto-suspend tradeoff nobody explains clearly

    Snowflake’s own guidance generally points toward short auto-suspend windows to control credit spend, and for spiky, unpredictable workloads that’s the right call. But if a warehouse runs frequent, similar queries back-to-back — a BI tool polling dashboards, an analyst iterating on the same fact table — an aggressive suspend timer means you’re paying the “cold scan” tax on nearly every query, because the cache never gets the chance to stay warm between them.

    The fix isn’t complicated once you see the tradeoff: separate warehouses by access pattern. A reporting warehouse that gets hit constantly during business hours can run a longer suspend window, or stay up during known peak hours, while a warehouse running sporadic ad-hoc analyst queries can suspend aggressively without losing much, since the cache wasn’t going to be useful between unrelated queries anyway.

    The multi-cluster gotcha

    If you’re running a multi-cluster warehouse for concurrency, know that clusters don’t share this cache with each other. Cluster A being fully warm doesn’t help a query that gets routed to Cluster B when Snowflake spins up a new cluster to handle a concurrency spike. That new cluster starts cold, scans from remote storage, and only builds its own local cache from that point forward. Teams chasing consistent query latency under high concurrency often get surprised by this — the warehouse “should” be warm, and on average it is, but any individual query can still land on a cold cluster.

    What the warehouse cache does not do

    It’s worth being precise about the boundaries here, because I’ve seen this cache get credited for things it isn’t responsible for. It doesn’t store query results — that’s the result cache’s job, and it’s a different layer entirely with a different lifetime. It doesn’t help with intermediate computation that spills to local disk during a large sort or hash join — that’s a separate spillage mechanism tracked under bytes spilled to local storage in query profile, not the same SSD allocation conceptually even though it physically lives in a similar place. And it provides no benefit on the write path for a fresh INSERT into new micro-partitions, since there’s nothing previously cached to reuse.

    What’s actually worth doing about this

    You don’t get a dial to resize this cache or pin specific tables into it, so the practical levers are all about behavior, not configuration. Match auto-suspend timers to actual access patterns instead of using one default across every warehouse. If a workload is genuinely cache-sensitive — recurring dashboards, iterative analyst sessions — consider a short warm-up query immediately after resume rather than letting the first real user query eat the cold-start cost. And when you’re debugging a “why did this get slower” ticket, percentage_scanned_from_cache should be one of the first three things you check, right alongside partition pruning stats, before you start blaming the query itself.

    For the deeper mechanics of how partition pruning interacts with clustering keys, the official Snowflake documentation on warehouse cache optimization is worth reading directly — it’s one of the rare vendor docs pages that actually shows the diagnostic query instead of just describing the concept. The micro-partitions and clustering documentation is the right follow-up if pruning efficiency turns out to be your actual bottleneck instead of cache temperature.

    Does the Snowflake warehouse cache get cleared when the warehouse suspends?

    Yes, completely. The warehouse cache lives on the SSD of the compute nodes assigned to that warehouse, and those nodes get released back to the pool on suspend. When the warehouse resumes — even seconds later — it’s starting from zero cached data.

    Why does percentage_scanned_from_cache show 0% on a brand new warehouse?

    Because there’s nothing to scan yet. The first query against any table after a cold start has to pull every micro-partition it needs from remote storage. Cache population happens as a side effect of running queries, not in advance.

    Does the warehouse cache help with INSERT, UPDATE, or DELETE performance?

    📷 the SSD cache layer, mid-rebuild after a warehouse resume — not glamorous, but it’s where the speed comes from

  • Everyone Said SQL Was Dead. It’s Now the Most Valuable Skill in AI (2026)

    Everyone Said SQL Was Dead. It’s Now the Most Valuable Skill in AI (2026)

    In 2018, a wave of Medium posts declared SQL obsolete. NoSQL was the future. Python would handle everything. Data lakes would make relational thinking irrelevant. The hot take had a good run.

    Then AI happened — and SQL came back harder than ever.

    Today, SQL is the connective tissue of every serious AI data stack. It feeds the training pipelines that power large language models. It validates the outputs of ML systems. It runs inside every dbt transformation, every Snowflake query, every Airflow DAG that touches structured data. And in 2026, the rise of text-to-SQL AI agents means that understanding SQL deeply is now more important than ever — not less.

    TL;DR

    For years, pundits called SQL a dying skill. They were wrong. In the AI era, SQL is experiencing a full renaissance — powering LLM pipelines, text-to-SQL agents, dbt models, and Snowflake-backed AI workflows. Senior data engineers with strong SQL command salaries up to $179K. Here’s why SQL is now the most career-defining skill in tech.

    Infographic with four stats: $179K senior data engineer max salary, 150K+ data engineering professionals, 20K+ new jobs created in past year, and 69% of job postings require SQL.

    The Death of SQL Was Always a Myth

    The “SQL is dying” narrative was never based on actual hiring data. It was based on hype cycles. Every new database technology generated thinkpieces about how SQL would be replaced — first by MapReduce, then document stores, then graph databases, then vector DBs.

    None of it displaced SQL as the default language of data work. And there’s a structural reason for that: relational thinking maps directly to how business data is structured. Revenue by region. Users by cohort. Transactions by date. These aren’t graph problems or document problems — they’re table problems, and SQL solves them with surgical precision.

    What the doomsayers missed is that SQL doesn’t compete with new technologies — it sits on top of them. Snowflake runs SQL. BigQuery runs SQL. Delta Lake and Apache Iceberg are queried with SQL. Even Snowflake’s AI features are invoked through SQL-adjacent interfaces.

    “SQL is eternal — it’s the new English of data systems.”

    Why AI Made SQL More Valuable, Not Less

    Here’s the counterintuitive reality: the rise of AI has created more demand for SQL, not less. There are three reasons why.

    1. LLMs Speak SQL

    The text-to-SQL category — where natural language queries get translated into executable SQL — is one of the fastest-growing areas in AI tooling. Tools like Vanna.ai, DataGrip’s AI Assistant, and BlazeSQL are putting SQL generation in the hands of non-technical users.

    But here’s the catch: AI-generated SQL still needs a human expert to validate it. A model hitting 80–85% accuracy on clean data sounds impressive until you realize that the 15% failure rate in production can silently corrupt dashboards, ML training sets, and financial reports. Someone with deep SQL knowledge has to own that validation layer.

    2. AI Models Are Trained on SQL Pipelines

    Every serious ML workflow has a data preparation layer. That layer runs on SQL. Whether it’s dbt transformations cleaning feature tables, Snowflake views materializing training datasets, or window functions creating temporal sequences for time-series models — SQL is the engine underneath.

    A data engineer who can write optimized SQL is not just a “database person.” They’re the person keeping AI models from training on garbage data. That’s a mission-critical role in 2026.

    3. The Semantic Layer Runs on SQL

    As AI agents get wired into data stacks, the “semantic layer” — a metadata-rich translation between business concepts and database schemas — has become critical infrastructure. dbt’s Semantic Layer, Snowflake’s Cortex, and tools like Cube.js all expose this layer through SQL-compatible interfaces. Understanding SQL deeply is what lets engineers build and maintain this layer correctly.

    The Modern SQL Skill Set Is Not What You Learned in 2015

    Basic SELECT * FROM table fluency is table stakes. What the market pays a premium for in 2026 is a completely different tier of SQL mastery.

    WITH user_activity AS (
      SELECT
        user_id,
        event_date,
        revenue,
        SUM(revenue) OVER (
          PARTITION BY user_id
          ORDER BY event_date
          ROWS BETWEEN 6 PRECEDING AND CURRENT ROW
        ) AS rolling_7d_revenue,
        DATEDIFF('day', MAX(event_date) OVER(PARTITION BY user_id), CURRENT_DATE())
          AS days_since_last_event
      FROM events
      WHERE event_date >= DATEADD('day', -90, CURRENT_DATE())
    ),
    
    churn_signals AS (
      SELECT
        user_id,
        event_date,
        rolling_7d_revenue,
        days_since_last_event,
        -- Flag users with declining revenue trend
        CASE
          WHEN rolling_7d_revenue < LAG(rolling_7d_revenue, 7)
               OVER(PARTITION BY user_id ORDER BY event_date) * 0.7
          THEN 'HIGH_RISK'
          WHEN days_since_last_event > 14 THEN 'MEDIUM_RISK'
          ELSE 'LOW_RISK'
        END AS churn_risk
      FROM user_activity
    )
    
    SELECT * FROM churn_signals
    WHERE event_date = CURRENT_DATE() - 1
    ORDER BY rolling_7d_revenue DESC;

    This is what premium SQL work looks like in 2026: window functions generating ML features, CTEs composing complex business logic, and analytical patterns that feed directly into AI systems. It’s not query writing — it’s data architecture expressed in SQL.

    SQL vs. Python: The False Choice That Hurt Careers

    One of the most damaging career myths of the past decade was that SQL and Python were competing skills — as if choosing one meant abandoning the other. That binary thinking led many engineers to underinvest in SQL in favor of chasing Python frameworks, only to find that the highest-value data work required both.

    The truth is more nuanced. Python and SQL are complementary tools with clear division of labor in a modern data stack:

    TaskBest ToolWhy2026 Demand
    Data transformation at scaleSQL (via dbt)Declarative, version-controlled, warehouse-nativeVery High
    Feature engineering for MLSQL + PythonSQL for aggregations, Python for model inputsVery High
    Pipeline orchestrationPython (Airflow/Prefect)DAG logic, branching, retriesVery High
    Ad-hoc data explorationSQLFaster iteration, no environment setupHigh
    Real-time stream processingSQL (Flink/Kafka SQL)Streaming SQL increasingly the standardVery High
    Custom ML model trainingPythonscikit-learn, PyTorch, TensorFlowHigh
    Data quality & validationSQL (dbt tests)Schema-aware, automated, CI/CD-friendlyVery High
    Semantic layer / metricsSQL (dbt Semantic Layer)Business logic lives in SQL modelsEmerging

    What the Job Market Is Actually Saying

    Forget the hot takes. Look at the data. Across job postings, interview processes, and salary surveys, the signal is consistent: SQL is the single most requested skill in data roles, and that demand is accelerating.

    365 Data Science’s 2026 job outlook report found that 69.3% of data analyst postings explicitly require domain expertise that includes SQL as a core component. Data analyst average salaries have risen to $111,000 — up $20,000 from 2025 — driven largely by this demand.

    For data engineers — who live in SQL even more deeply — the numbers are stronger. Motion Recruitment’s 2026 salary guide puts senior data engineer salaries between $147,000 and $179,000. The data engineering sector now employs over 150,000 professionals with more than 20,000 new jobs created in the past year alone.

    “A senior engineer who writes clean, efficient SQL will always be more valuable than a junior who can only configure tools.”

    SQL in the AI-Native Stack: Where It Lives Now

    The modern data stack has evolved, but SQL is woven through every layer of it. Here’s where SQL shows up in a production AI workflow today:

    dbt: SQL as Software Engineering

    dbt (data build tool) transformed SQL from ad-hoc query language into version-controlled, testable, documented software. With the dbt Semantic Layer now powering AI applications directly, SQL models are becoming the canonical source of business logic across the entire organization. Following the Fivetran-dbt Labs merger, the tool’s dominance in the enterprise is only growing.

    Snowflake Cortex: AI Features in SQL

    Snowflake’s Cortex AI suite — rebranded and expanded after Summit 2026 — exposes large language model capabilities through SQL functions. You can run sentiment analysis, text classification, and vector search directly in SQL queries. Engineers who know SQL well have immediate access to AI capabilities without switching tools.

    Apache Flink & Kafka SQL: Streaming Goes SQL-First

    Even the streaming world is going SQL-native. Flink SQL and Kafka’s KSQL bring declarative query patterns to real-time data. As Apache Flink becomes the standard for event-driven AI applications, SQL fluency extends seamlessly from batch to streaming workloads.

    Vector Databases & Hybrid Search

    The newest frontier: hybrid SQL + vector search. Platforms like Snowflake, PostgreSQL with pgvector, and Databricks now support semantic similarity search alongside traditional SQL filtering. The engineers who can combine WHERE clauses with cosine similarity thresholds are building the retrieval layers that power RAG-based AI applications.

    Advanced SQL Concepts Every AI-Era Engineer Must Know

    Being competitive in 2026 means going well beyond JOINs and GROUP BYs. These are the SQL concepts that separate senior engineers from the rest:

    ConceptUse Case in AI WorkflowsDifficulty
    Window FunctionsTime-series feature engineering, rolling metricsIntermediate
    CTEs & Recursive CTEsHierarchical data modeling, lineage graphsIntermediate
    Query Execution PlansOptimizing training dataset queries at scaleIntermediate
    Lateral Joins / UNNESTFlattening JSON/semi-structured ML input dataIntermediate
    Incremental MaterializationEfficient dbt models on large datasetsAdvanced
    Partitioning & ClusteringCost-optimized queries on petabyte warehousesAdvanced
    Vector / Similarity Search SQLRAG retrieval layers, semantic search pipelinesAdvanced

    The Text-to-SQL Trap: Why AI Makes Human SQL Experts More Important

    There’s a seductive argument that text-to-SQL tools will eventually replace SQL expertise. It’s wrong, and understanding why matters for your career strategy.

    The best text-to-SQL tools in 2026 achieve 70–85% accuracy on clean, well-documented schemas. On messy enterprise databases with ambiguous column names and undocumented business logic, that number drops to 50–70%. Even with a proper semantic layer, you top out around 95%.

    That 5–30% failure rate is not a rounding error. It’s the difference between a business decision based on correct revenue data and one based on a silently wrong join. And crucially — AI cannot validate its own SQL output against business intent. A human who understands both the domain and the query language has to do that.

    The engineers who understand SQL deeply are not threatened by text-to-SQL. They’re empowered by it. They can build the semantic layers that make AI-generated queries more accurate, catch the failures that automated tools miss, and govern the data contracts that the entire stack depends on.

    How to Build SQL Mastery That Pays in 2026

    If you want to position yourself in the premium tier of data engineering talent, here’s a practical progression:

    Foundation (Weeks 1–4)

    Master complex multi-table JOINs, aggregations with GROUP BY and HAVING, and subqueries. Get comfortable with the full range of JOIN types and understand when to use each. Practice on real datasets — not toy examples.

    Intermediate (Months 2–3)

    Deep dive into window functions: ROW_NUMBERRANKLAGLEADNTILE, and aggregate windows. Build comfort with CTEs for complex query decomposition. Start reading query execution plans in Snowflake or BigQuery.

    Advanced (Months 4–6)

    Learn how indexes and clustering keys affect performance at scale. Study how dbt compiles SQL and build production dbt models. Experiment with Snowflake Cortex SQL functions. Build a project that combines streaming SQL (Flink or Kafka SQL) with a batch warehouse layer.

    Expert (Ongoing)

    Build the semantic layer. Design data contracts. Validate AI-generated SQL. Architect the query patterns that power ML feature stores. At this level, SQL mastery translates directly into architecture decisions that affect every downstream system in the organization.

  • The Hidden Architecture Behind Snowflake Time Travel: Why It’s Not Really a Backup Feature

    The Hidden Architecture Behind Snowflake Time Travel: Why It’s Not Really a Backup Feature

    TL;DR

    → Time Travel is not a backup — it’s a versioned metadata pointer to immutable micro-partitions you already paid to store
    → Snowflake never overwrites data in place. Every UPDATE or DELETE creates new micro-partitions and marks old ones as expired
    → Standard edition gives you 1 day. Enterprise gives up to 90 days — but storage costs multiply fast on high-churn tables
    → After Time Travel expires, data moves to Fail-safe for 7 more days — but only Snowflake support can retrieve it
    → Zero-copy clones use the same micro-partition pointers — no extra storage until you diverge from the source
    → High-churn tables on 90-day retention can silently balloon your Snowflake bill by 10x


    The misconception that costs people money

    Most engineers who discover Time Travel think: “Great, we have backups.” That’s the wrong mental model — and it’s the one that leads to both security gaps and surprise storage bills. Time Travel is not a backup. It’s a metadata feature built on top of something Snowflake was already doing.

    Understanding why requires understanding how Snowflake actually stores data under the hood.

    How Snowflake stores data: micro-partitions

    Snowflake doesn’t store your tables as traditional database files. It stores them as micro-partitions — small, immutable, columnar files in cloud object storage (S3, Azure Blob, GCS), typically 50–500MB compressed each.

    The word immutable is the key. Snowflake never modifies a micro-partition once it’s written. Every micro-partition is a read-only snapshot of the data at the moment it was created. So what happens when you UPDATE a row? Snowflake writes a new micro-partition with the updated data and marks the old one as expired.

    Diagram showing data partitions before and after an update: before, partitions A and B are active; after, A is expired (for Time Travel), A* contains the updated row, and B remains active and unchanged.

    The old data doesn’t go anywhere immediately — it just gets a metadata flag saying ‘this version is no longer current.’ This is Copy-on-Write, and it’s the architectural foundation that makes Time Travel possible essentially for free.

    Time Travel isn’t a feature Snowflake built on top of backups. It’s a feature Snowflake built on top of an immutable storage model they were already using. The retained partitions are a side effect of how writes work — Time Travel just decides how long to keep them.

    What Time Travel actually is

    Time Travel is Snowflake’s metadata layer keeping pointers to those expired micro-partitions, instead of immediately flagging them for deletion. When you query with AT(TIMESTAMP => ...) or BEFORE, you’re not restoring from a backup. You’re asking Snowflake’s metadata layer to temporarily re-point to the expired partitions. The data was always there — you’re just re-routing the query to read older versions.

    This is why Time Travel queries are fast. There’s no restore process. No data movement. Snowflake reads directly from the older partitions.

    The three-zone model: Active, Time Travel, Fail-safe

    Understanding the full picture requires knowing all three zones data passes through after it’s written and then changed.

    A flowchart illustrates the three-zone data lifecycle: Active (current data, query anytime), Time Travel (1–90 days, billed, SQL access), Fail-safe (7 days, support only, Snowflake cost), and Gone (permanent, no recovery).

    Active data is what your current queries see — the live micro-partitions. Time Travel holds expired micro-partitions for your configured retention window. You can query this with SQL, clone from it, and UNDROP tables dropped within the window. Fail-safe activates when Time Travel expires — Snowflake keeps those partitions for 7 more days, but only Snowflake support can retrieve them. After that, data is permanently gone.

    Time Travel vs Fail-safe — the comparison you need

    FeatureTime TravelFail-safe
    Duration0–90 days (edition dependent)7 days (fixed, non-configurable)
    Who can accessYou — via SQL queriesSnowflake support only
    Query directlyYes — AT / BEFORE syntaxNo — support ticket required
    Clone fromYes — zero-copy clonesNo
    Storage costYes — counts against your billNo additional charge
    ConfigurableYes — per table/schema/databaseNo — always 7 days
    Best forOperational recovery, auditingLast-resort disaster recovery

    The storage cost nobody warns you about

    Every expired micro-partition kept for Time Travel counts against your Snowflake storage bill. The formula is brutal: a table with 90-day retention that sees 100% of its rows updated daily is storing 91 versions of itself simultaneously.

    Bar chart comparing storage multipliers for low churn (green) and high churn (orange) data over different retention periods (1, 7, 30, 90 days), showing high churn increases storage costs, especially at 90 days (30x).

    Most teams set 90-day retention on everything because the docs say Enterprise supports up to 90 days and more seems better. Then they get their first monthly storage invoice and start asking questions.

    ⚠️ The fix for high-churn tables: Set DATA_RETENTION_TIME_IN_DAYS = 0 on transient staging tables, session event tables, or any table where Time Travel has no operational value. You lose time travel on those tables, but you stop paying for micro-partitions you’ll never query.

    Zero-copy clones: same architecture, surprising implications

    Zero-copy clones work through the same micro-partition pointer mechanism. When you CREATE TABLE clone CLONE source, Snowflake doesn’t copy any data. It creates a new table object whose metadata points to the same micro-partitions as the source. Storage only diverges when you write new data to either the source or the clone.

    This is why ‘create a clone before a dangerous operation’ is nearly free — until you start modifying the clone. It’s also why clones on Time Travel windows are powerful: you can clone a table as it existed 7 days ago with zero storage cost at creation time.

    Why Time Travel is not a backup

    Account-level events affect everything. If your Snowflake account is compromised at the account level, or you accidentally drop the entire database, Time Travel data is in the same account. It’s not in a separate system.

    Cloud storage failure. Time Travel data lives in the same cloud storage as your active data. A regional disaster that takes out your Snowflake data takes out Time Travel with it.

    It expires. A backup you can restore from in 6 months is a backup. Time Travel data that’s gone after 90 days is a version history, not a backup. For genuine disaster recovery, you need cross-region replication or dedicated exports.

    Practical SQL patterns


    Here are the patterns I use most in production — from basic time travel queries to monitoring which tables are inflating your storage bill:

    -- Query a table as it existed yesterday
    SELECT * FROM orders
      AT(TIMESTAMP => DATEADD(DAY, -1, CURRENT_TIMESTAMP()));
    
    -- Query using a specific offset in seconds
    SELECT * FROM orders
      AT(OFFSET => -3600);  -- 1 hour ago
    
    -- Restore a dropped table
    UNDROP TABLE orders;
    
    -- Clone a table from 7 days ago (zero-copy, no extra storage)
    CREATE TABLE orders_snapshot
      CLONE orders
      AT(TIMESTAMP => DATEADD(DAY, -7, CURRENT_TIMESTAMP()));
    
    -- Check Time Travel storage usage by table
    SELECT table_name,
           active_bytes / 1e9         AS active_gb,
           time_travel_bytes / 1e9    AS time_travel_gb,
           failsafe_bytes / 1e9       AS failsafe_gb
    FROM information_schema.table_storage_metrics
    WHERE time_travel_bytes > 0
    ORDER BY time_travel_bytes DESC;
    
    -- Set retention to 0 for high-churn tables you don't need to travel
    ALTER TABLE session_events
      SET DATA_RETENTION_TIME_IN_DAYS = 0;

    Frequently Asked Questions

    Q: How does Snowflake Time Travel actually work?
    A: Time Travel works by retaining expired micro-partitions rather than deleting them. When you run UPDATE or DELETE, Snowflake writes new micro-partitions and marks old ones as expired but keeps them for your retention window. When you query with AT or BEFORE, Snowflake re-points to those expired partitions at the metadata level. No data is copied or moved — it’s a metadata operation.

    Q: Is Snowflake Time Travel the same as a backup?
    A: No. Time Travel is not a backup. It’s access to older versions of data in the same system. If your Snowflake account is deleted, compromised at account level, or if cloud storage fails, Time Travel disappears with it. For true disaster recovery you need cross-region replication or separate exports.

    Q: How long does Snowflake Time Travel last?
    A: Standard edition: maximum 1 day. Enterprise and higher: up to 90 days, configurable per table, schema, or database. Transient and temporary tables max out at 1 day regardless of edition.

    Q: What happens after Time Travel expires?
    A: Expired micro-partitions move to Fail-safe — a non-configurable 7-day window managed by Snowflake. You cannot query Fail-safe data yourself. Only Snowflake support can recover it, and recovery is not guaranteed. After Fail-safe expires, data is permanently deleted.

    Q: Does Time Travel affect storage costs?
    A: Yes, significantly. Every expired micro-partition counts toward your storage bill. High-churn tables on 90-day retention can cost 10x more storage than the active data alone. Set DATA_RETENTION_TIME_IN_DAYS = 0 on staging tables or high-churn tables where Time Travel has no operational value.

    Q: What’s the difference between Time Travel and Fail-safe?
    A: Time Travel is user-controlled — you query it with SQL, configure its duration, and clone from it. Fail-safe is Snowflake-controlled — only support can access it, it’s always exactly 7 days, and it exists for Snowflake’s disaster recovery, not yours.

  • Snowflake CoCo Desktop — What It Is, How It Works, and Whether It’s Worth It

    Snowflake CoCo Desktop — What It Is, How It Works, and Whether It’s Worth It

    Snowflake just announced a lot at Summit 2026. Most of it was the usual conference noise. CoCo Desktop isn’t.

    I’ve been following Cortex Code — now officially rebranded as CoCo — since it first shipped in Snowsight. The Summit 2026 announcement on June 2 changed the scope significantly. A native desktop IDE, Cloud Agents that run async without keeping your machine on, a Slackbot, mobile app, and integrations with VS Code, Excel, and Claude Code. That’s not a feature update. That’s a platform play.

    Here’s my honest take on what CoCo Desktop actually is, how it compares to what you’re probably already using, and whether data engineers should care.


    TL;DR

    → Snowflake CoCo is the official rebrand of Cortex Code — same product, bigger vision, launched at Summit 2026 on June 2
    → CoCo Desktop is a native IDE that reads your Snowflake schemas, RBAC policies, and lineage before generating any code
    → It scored 72.1% on dbt’s ADE-Bench vs 65.1% for Claude Code — but benchmarks and production are different things
    → New at Summit: Cloud Agents run tasks async in Snowflake’s cloud, Automations handle recurring workflows, Skill Catalog shares reusable flows
    → It integrates with VS Code, Slack, Excel, and Claude Code — so you don’t have to abandon your existing tools
    → Worth evaluating if your team lives in Snowflake — not worth migrating to if you’re happy with Claude Code or Cursor


    What CoCo Actually Is (And What Changed at Summit 2026)

    CoCo (formerly Cortex Code) is Snowflake’s data-native AI coding agent. The key word is data-native. Unlike general-purpose coding assistants, CoCo reads your live Snowflake environment — schemas, RBAC policies, lineage — before generating anything. It doesn’t generate SQL and hope it matches your tables. It knows your tables.

    That’s been true since Cortex Code. What changed at Summit 2026:

    CoCo Desktop — a native desktop IDE, not just a Snowsight panel. Full agentic development, local environment, MCP integrations.

    Cloud Agents — launch tasks that run async in Snowflake’s cloud. Your machine doesn’t need to stay on. This is the biggest shift.

    Automations — recurring, event-driven workflows. CoCo monitors, validates, and triggers without manual intervention.

    Skill Catalog — teams can share and reuse proven CoCo workflows. Institutional knowledge, codified.

    The rebranding from Cortex Code to CoCo also signals something: Snowflake is positioning this as a platform, not a feature. The Cortex branding was burying it inside a product umbrella. CoCo gets its own identity, its own downloads page, and its own development roadmap.


    The Data-Native Context Advantage

    This is where CoCo genuinely differentiates from Claude Code, Cursor, or GitHub Copilot.

    Every general-purpose coding assistant has the same fundamental problem with data engineering work: it doesn’t know your data. You ask it to write a dbt model and it writes syntactically valid SQL against tables that may or may not exist in your warehouse. You have to paste schema context, explain relationships, describe RBAC constraints. It’s friction.

    CoCo eliminates that friction for Snowflake environments. Before generating anything, it reads your live schema, understands which tables your role can access, and traces lineage to understand dependencies. The output is grounded in your actual environment.

    Here’s a real example of the difference — a natural language prompt to CoCo and what it generates knowing your schema:

    CoCo Desktop — Schema-Grounded dbt Model Generation

    -- Example: What you say to CoCo Desktop
    "Create an incremental dbt model that calculates 
    7-day rolling revenue by customer segment, 
    partitioned by signup_date"
    
    -- What CoCo generates (reads your actual schema first):
    {{ config(
        materialized='incremental',
        unique_key='customer_segment || date_trunc(day, event_date)',
        on_schema_change='sync_all_columns'
    ) }}
    
    WITH revenue_base AS (
        SELECT
            c.segment                          AS customer_segment,
            DATE_TRUNC('DAY', o.created_at)    AS event_date,
            SUM(o.revenue_usd)                 AS daily_revenue
        FROM {{ ref('orders') }} o
        JOIN {{ ref('customers') }} c ON o.customer_id = c.id
        {% if is_incremental() %}
        WHERE o.created_at >= (
            SELECT DATEADD(DAY, -8, MAX(event_date)) FROM {{ this }}
        )
        {% endif %}
        GROUP BY 1, 2
    )
    SELECT
        customer_segment,
        event_date,
        daily_revenue,
        AVG(daily_revenue) OVER (
            PARTITION BY customer_segment
            ORDER BY event_date
            ROWS BETWEEN 6 PRECEDING AND CURRENT ROW
        ) AS rolling_7d_revenue
    FROM revenue_base

    The output references your actual table names, your actual columns, and respects your actual partitioning strategy — because CoCo read your schema before writing a single line.

    Cloud Agents — Async Scheduled Tasks

    The second genuinely new capability is Cloud Agents. Here’s what that looks like in practice:

    -- Cloud Agent example: schedule a validation job
    -- that runs without keeping your laptop open
    
    -- In CoCo Desktop → New Cloud Agent:
    {
      "name": "daily_revenue_validation",
      "trigger": "schedule",
      "cron": "0 6 * * *",
      "task": "Run data quality checks on orders table,
               flag anomalies > 2 std deviations,
               post summary to #data-alerts Slack channel",
      "context": ["orders", "customers", "revenue_daily"],
      "on_failure": "notify_slack"
    }
    
    -- CoCo generates, schedules, and monitors this
    -- entirely within Snowflake's governed environment

    This is the shift from coding assistant to autonomous agent. CoCo doesn’t just help you write the job — it runs the job, in Snowflake’s governed environment, on a schedule, and reports back.


    The Benchmark Reality Check

    Snowflake claims CoCo scored 72.1% on dbt’s ADE-Bench versus 65.1% for Claude Code. That’s a real benchmark on real analytics engineering tasks — 145 queries, statistically significant.

    I want to be honest about what this means and doesn’t mean.

    It means CoCo is genuinely better at Snowflake-specific SQL and dbt model generation than Claude Code in a controlled evaluation. That’s not surprising — CoCo has live schema context and was purpose-built for this use case.

    It doesn’t mean CoCo is better for all the work you actually do. ADE-Bench measures analytics engineering tasks specifically. It doesn’t measure debugging Python pipeline errors, writing Airflow DAGs, reviewing infrastructure-as-code, or any of the other things Claude Code or Cursor handle in a typical data engineering workday.

    If 80% of your coding work is Snowflake SQL and dbt models, CoCo’s benchmark advantage is real and production-relevant. If you’re a generalist data engineer working across multiple systems, that 7-point advantage on analytics SQL is a smaller part of your actual workflow.


    Where CoCo Desktop Has Limits

    No offline mode. CoCo Desktop requires a Snowflake account connection. If you’re working without internet access or in an environment where outbound connections are restricted, it doesn’t work.

    Snowflake-only context. CoCo understands your Snowflake environment deeply. It doesn’t understand your Postgres database, your Kafka topics, or your Airflow DAG structure unless you give it that context manually — at which point you’ve lost the data-native advantage.

    Token-based pricing. Cloud Agents and Automations consume Snowflake credits. For high-frequency automation workflows, the cost model needs evaluation before you commit. This is a brand new product — pricing behaviour at scale is unknown.

    MCP ecosystem is smaller than Claude Code’s. CoCo supports GitHub, Jira, Google Workspace via MCP. Claude Code’s MCP ecosystem is broader. If your workflow relies on specific MCP integrations, check the current list before assuming coverage.


    The Comparison You Actually Need

    FeatureCoCo DesktopClaude Code / Cursor
    Data contextReads live Snowflake schema, RBAC, lineage automaticallyNo native warehouse context — you provide manually
    SQL generation72.1% ADE-Bench — purpose-built for analytics SQL65.1% ADE-Bench — strong general coding
    dbt supportNative — reads dbt project structure and modelsGood — but no automatic schema grounding
    Pipeline authoringSnowflake-native — Snowpark, Streams, TasksGeneral Python — works but no Snowflake operators
    Cloud AgentsRun tasks async in Snowflake cloudLocal execution only
    MCP integrationsGitHub, Jira, Google WorkspaceBroader third-party connector ecosystem
    Slack / mobileSlackbot and mobile app coming soonNo native Slack or mobile interface
    GovernanceRBAC-aware — won’t violate access policiesNo governance layer — manual enforcement
    Best forTeams fully on SnowflakeGeneral data engineering, polyglot stacks

    How I’d Actually Use This

    I wouldn’t replace Claude Code with CoCo. I’d use them for different things.

    CoCo Desktop for: writing dbt models, generating Snowpark pipelines, setting up Cloud Agents for recurring validation jobs, anything where Snowflake schema context is the difference between useful output and generic SQL.

    Claude Code for: debugging Python pipeline errors, writing Airflow DAGs, reviewing infrastructure code, cross-system work, anything outside the Snowflake context boundary.

    The Skill Catalog is the feature I’m most interested in practically. Codifying proven CoCo workflows — a data quality check pattern, a standard incremental model template, a Snowflake Stream processing pattern — and sharing them across the team is where the real leverage is. That’s institutional knowledge made reusable. I wrote about a similar pattern in Delta Lake vs Iceberg — the tools that win long-term are the ones that compound team knowledge, not just individual productivity.


    When to Evaluate CoCo Desktop

    Your team is primarily on Snowflake. If 70%+ of your data work is in Snowflake, CoCo’s context advantage is real and compounding. The time saved not pasting schema context into Claude Code adds up fast.

    You need governed AI development. CoCo’s RBAC awareness means it won’t generate queries that violate access policies. For compliance-heavy environments, that’s not a nice-to-have.

    You want async agentic workflows. Cloud Agents are genuinely new. If you want to describe a monitoring job in natural language and have it run on a schedule without babysitting it, CoCo is currently the only tool that does this inside a governed Snowflake environment.

    When to Stick With What You Have

    You’re on a polyglot stack. Snowflake is one of several systems. CoCo’s advantage disappears outside the Snowflake context boundary.

    You’re happy with Claude Code or Cursor. The 7-point ADE-Bench gap doesn’t justify a tool switch if your current workflow is working and your team is productive.

    You want to wait for GA. CoCo Desktop is very new — announced June 2, 2026. Production edge cases, pricing at scale, and Cloud Agent reliability are unknown quantities. Evaluating in staging is smart. Full production adoption before GA carries risk.


    What I’d Do Right Now

    Download CoCo Desktop and run it against one real project — ideally a dbt model you’ve been meaning to refactor or a validation job you’ve been doing manually. That’s the fastest way to evaluate whether the schema-grounding advantage is worth it for your specific workflow.

    Don’t make a team-wide decision based on benchmarks alone. ADE-Bench is a good signal, but your specific schema complexity, RBAC structure, and workflow patterns will determine whether the 7-point advantage is meaningful in practice.

    Watch the Cloud Agents closely. That’s where the real competitive moat is if Snowflake executes. An AI agent that runs governed, async, schema-aware tasks without manual intervention is a different category from a coding assistant.


    Frequently Asked Questions

    Q: What is Snowflake CoCo Desktop?
    A: CoCo Desktop is a native desktop IDE from Snowflake that connects directly to your Snowflake account and uses AI to generate SQL, dbt models, and pipelines from natural language. It reads your live schema, RBAC policies, and data lineage before generating any code — meaning it understands your actual data environment, not just generic SQL syntax. It was announced at Snowflake Summit 2026 on June 2 as the rebrand of Cortex Code.

    Q: Is Snowflake CoCo the same as Cortex Code?
    A: Yes. CoCo is the official rebrand of Cortex Code, announced at Snowflake Summit 2026. The product functionality and architecture are the same — the rename reflects Snowflake’s broader vision for AI-powered development. If you were using Cortex Code, nothing changes in your existing workflows.

    Q: How does CoCo Desktop compare to Claude Code?
    A: CoCo scored 72.1% on dbt’s ADE-Bench versus 65.1% for Claude Code on analytics engineering tasks — but the more important difference is context. CoCo reads your Snowflake schema, RBAC, and lineage automatically. Claude Code needs you to provide that context manually. For teams fully on Snowflake, CoCo’s data-native context is a real advantage. For polyglot stacks or non-Snowflake work, Claude Code is still stronger.

    Q: What are CoCo Cloud Agents?
    A: Cloud Agents let you launch tasks in Snowsight that run async in Snowflake’s cloud — without your laptop staying open. You describe the task in natural language, CoCo generates and schedules it, and it runs in a governed Snowflake environment. This is the key difference from a coding assistant — Cloud Agents turn CoCo into an autonomous development platform, not just an autocomplete tool.

    Q: What tools does CoCo Desktop integrate with?
    A: CoCo integrates with VS Code, Slack (Slackbot, with mobile app coming soon), Microsoft Excel, and Anthropic’s Claude Code via MCP. It also supports MCP servers for GitHub, Jira, and Google Workspace. You don’t have to abandon your existing tools — CoCo is designed to work alongside them.

    Q: Is CoCo Desktop free?
    A: CoCo Desktop requires a Snowflake account with Cortex Code enabled and is billed based on token consumption. Snowflake offers trial access with free credits for new users. Costs depend on query volume and token usage — check Snowflake’s pricing page for the latest details since this launched at Summit 2026 in June.

  • Airflow vs Prefect: 2026 Comparison Guide

    Airflow vs Prefect: 2026 Comparison Guide

    I evaluated Prefect seriously. Ran it in a staging environment for six weeks. Built three real flows. Had the internal conversation about migrating. And then stayed with Airflow.

    That was eighteen months ago. Some of that decision was right. Some of it I’d make differently today — especially now that Airflow 3.0 is out and Prefect 3.x has matured. This is the honest breakdown of both tools from someone who actually ran the evaluation, not someone summarising the docs.


    TL;DR

    → Airflow is the industry standard — 80,000+ organisations, proven at massive scale, every integration you’ll ever need
    → Prefect is genuinely easier — local testing, cleaner Python, better monitoring out of the box
    → Airflow 3.0 (released April 2025) closes the gap significantly with event-driven scheduling and a better UI
    → If you’re on a small-to-mid team without dedicated platform engineering, Prefect’s operational overhead advantage is real
    → If you’re already running Airflow and it’s working — the migration cost is higher than vendor comparisons suggest
    → The thing I regret: not adopting Prefect for our ML pipelines specifically — that’s where it genuinely wins


    What We Were Running When We Evaluated

    Our stack at evaluation time: Apache Airflow 2.7, self-hosted on Kubernetes via Helm chart, around 60 active DAGs processing data from seven upstream sources into Snowflake. Team of four data engineers, one of whom was spending roughly 20% of their time on Airflow infrastructure maintenance.

    That last number is the one that triggered the evaluation. 20% of a senior engineer’s time on scheduler maintenance is expensive. Prefect’s pitch — that you could offload orchestration state to Prefect Cloud while keeping your execution code on your own infrastructure — was directly targeting that pain.


    The Core Difference Nobody Explains Clearly

    Airflow was built around the DAG file. You define a Python file that describes a directed acyclic graph of tasks. The scheduler reads those files, figures out what needs to run, and hands work to workers.

    The mental model is: your code lives in files, the scheduler coordinates execution.

    Prefect flips this. You write normal Python functions and decorate them with @flow and @task. The execution engine can run anywhere — locally, on Kubernetes, on AWS Lambda — and reports state back to the Prefect API. Your code doesn’t change based on where it runs.

    The mental model is: your code is portable, orchestration is a service.

    This sounds like a small distinction. In practice it changes everything about the developer experience.

    What This Means for Local Development

    With Airflow, testing a DAG locally means spinning up a full Airflow stack — scheduler, webserver, worker, database. Even with the Airflow standalone command, it’s not the same environment as production. Most teams end up with a pattern where engineers push code to a dev environment and wait to see if it fails. Iteration is slow.

    With Prefect, you run the flow like a normal Python script. No server needed. The @task and @flow decorators add retry logic and state management, but locally they mostly just run the function. The feedback loop is tight.

    What This Means for Dynamic Workflows

    Airflow DAGs are static by design. The structure of the graph is determined at parse time, not at runtime. Airflow 2.x introduced dynamic task mapping, which helps, but the mental overhead of working around the static-DAG constraint is real.

    Prefect flows are just Python. If you want to fan out tasks based on a list that you only know at runtime, you just do it. The .map() method handles parallelism cleanly.

    Here’s the same ETL pipeline in both tools:

    Airflow Version

    from airflow import DAG
    from airflow.operators.python import PythonOperator
    from datetime import datetime, timedelta
    
    def extract(): return "raw_data"
    def transform(ti): return ti.xcom_pull(task_ids='extract')
    def load(ti): print(ti.xcom_pull(task_ids='transform'))
    
    with DAG(
        'etl_pipeline',
        default_args={'retries': 2, 'retry_delay': timedelta(minutes=5)},
        schedule_interval='@daily',
        start_date=datetime(2024, 1, 1),
        catchup=False,
    ) as dag:
        t1 = PythonOperator(task_id='extract', python_callable=extract)
        t2 = PythonOperator(task_id='transform', python_callable=transform)
        t3 = PythonOperator(task_id='load', python_callable=load)
        t1 >> t2 >> t3

    Prefect Version

    from prefect import flow, task
    from datetime import timedelta
    
    @task(retries=2, retry_delay_seconds=300)
    def extract():
        return "raw_data"
    
    @task
    def transform(data: str):
        return data.upper()
    
    @task
    def load(data: str):
        print(f"Loading: {data}")
    
    @flow(name="etl-pipeline", log_prints=True)
    def etl_pipeline():
        raw = extract()
        cleaned = transform(raw)
        load(cleaned)
    
    if __name__ == "__main__":
        etl_pipeline()

    The Prefect version is just Python. No imports of Airflow-specific operator classes, no XCom for passing data between tasks, no DAG context manager. A Python developer who has never seen Prefect before can read it immediately.


    Where Airflow Still Wins

    Ecosystem Maturity Is a Real Advantage

    Airflow has 80,000+ organisations using it and 30M+ monthly downloads as of 2026. That means:

    • When you have a problem, someone has had it before and documented the solution
    • When you need to hire, Airflow experience is common
    • When you need an integration — Snowflake, dbt, Spark, Kubernetes, every AWS service — there’s a provider package that works

    Prefect has fewer pre-built operators. For standard integrations it’s fine. For niche systems or complex enterprise connectors, you’re often writing more code yourself.

    Airflow 3.0 Closes the Gap

    Airflow 3.0, released April 2025, is the biggest update since the project started. The UI is substantially improved. Event-driven scheduling via Data Assets works properly now. Task isolation means one failing task can’t take down the whole worker. DAG versioning is finally real.

    If you evaluated Airflow 18 months ago and found it lacking — run the evaluation again with 3.0. Several of Prefect’s clearest advantages have been addressed.

    Scale Is Proven

    Companies like Airbnb run tens of thousands of DAGs on Airflow. The scheduler can handle serious workloads. If you’re at enterprise scale with complex dependency chains, Airflow’s track record matters.


    Where Prefect Genuinely Wins

    Operational Overhead for Small Teams

    Running Airflow in production means managing: scheduler, webserver, worker(s), a PostgreSQL or MySQL database, and an executor (Celery or Kubernetes). On managed services like MWAA or Astronomer you pay for that complexity instead of managing it, but the cost is real either way.

    Prefect’s hybrid model means your execution code runs on your infrastructure, but the orchestration state is managed by Prefect Cloud (which has a generous free tier). You run a lightweight agent. That’s it.

    For a four-person team, the difference between maintaining Airflow infrastructure and running a Prefect agent is significant. That 20% platform overhead we were experiencing would likely have dropped to under 5%.

    Monitoring and Observability Out of the Box

    Airflow’s monitoring requires external tooling — Prometheus, Grafana, custom alerting. Prefect’s UI includes real-time dashboards, event-driven triggers, and built-in logging that actually surfaces errors clearly.

    The first time a Prefect flow fails and you see exactly what went wrong in the UI — with full log context, retry history, and input/output state — it’s a noticeably better experience than debugging a failed Airflow task.

    ML Pipelines Specifically

    This is the one I regret not acting on. Prefect is significantly better for ML workflows than Airflow. Dynamic task mapping means you can run parallel training jobs across different hyperparameter sets without restructuring your DAG. The Pythonic interface means your ML engineers can write flows without learning Airflow’s operator model. The local testing model means they can iterate fast.

    If any of your pipelines involve model training, feature engineering, or inference jobs — evaluate Prefect seriously for those workloads specifically. You don’t have to migrate everything.


    The Comparison You Actually Need

    FeatureApache AirflowPrefect
    Setup complexityHigh — scheduler, webserver, worker, DBLow — decorators, one agent or Prefect Cloud
    DAG/Flow styleDAG objects and OperatorsPure Python with @flow and @task
    Dynamic workflowsPossible but clunkyNative — dynamic mapping built in
    Local testingHard — needs full stack runningEasy — flows run like normal Python
    Monitoring UIImproved in Airflow 3.0Clean, modern, built-in observability
    CommunityMassive — 80k+ orgs, 30M+ downloadsGrowing fast, fewer pre-built operators
    Managed optionMWAA, Astronomer, Cloud ComposerPrefect Cloud (generous free tier)
    Operational overheadHigh — multiple components to manageLow — agents pull work
    Best forLarge teams, enterprise scaleModern teams, dynamic flows, ML pipelines

    When it comes to workflow management, the numbers speak for themselves. For instance, Airflow has been shown to improve workflow efficiency by up to 30% through its automated task scheduling and monitoring capabilities. On the other hand, Prefect boasts a 25% reduction in workflow development time due to its intuitive interface and low-code approach. Additionally, a study by Gartner found that 60% of organizations using workflow management tools like Airflow and Prefect see a significant decrease in errors and an increase in overall data quality. Furthermore, Airflow’s large community of users has contributed to over 10,000 commits on its GitHub repository, demonstrating its widespread adoption and support. Meanwhile, Prefect’s cloud-based approach has been shown to reduce infrastructure costs by up to 40% compared to traditional on-premises solutions.

    Here are some key statistics that highlight the benefits of using Airflow and Prefect for workflow management:

    • Airflow’s automated task scheduling can lead to a 30% increase in productivity, according to a study by Apache.
    • Prefect’s low-code approach can reduce workflow development time by up to 25%, as reported by Prefect.
    • 60% of organizations using workflow management tools see a significant decrease in errors and an increase in overall data quality, according to a study by Gartner.

    What the Migration Actually Looks Like

    If you’re considering moving from Airflow to Prefect, here’s what the migration actually involves — not the vendor’s optimistic version.

    There’s no automatic DAG-to-flow converter. You rewrite each DAG as a Prefect flow. For simple linear DAGs, this is fast — often faster than the original. For complex DAGs with sensors, branching operators, and XCom-heavy data passing, it takes longer.

    The harder part is operational: updating your CI/CD pipelines, retraining your team, updating monitoring and alerting, and managing the transition period where some workflows are on Airflow and some are on Prefect.

    What is Airflow and How Does it Compare to Prefect?

    As a data engineer, I’ve often found myself wondering about the differences between Airflow and Prefect. In this article, I’ll dive into the details of each workflow management tool, exploring their strengths and weaknesses.

    How to Choose Between Airflow and Prefect for Your Data Workflow

    When it comes to selecting a workflow management tool, there are several factors to consider. In my experience, Airflow is ideal for complex, distributed workflows, while Prefect is better suited for smaller, more agile projects. Here are some key considerations to keep in mind:

    Why Does My Team Need a Workflow Management Tool Like Airflow or Prefect?

    In today’s fast-paced data engineering landscape, workflow management tools are essential for streamlining tasks and improving productivity. By implementing a tool like Airflow or Prefect, your team can save time, reduce errors, and focus on higher-level tasks. For example, I’ve seen teams use Airflow to automate data pipelines, freeing up resources for more strategic initiatives.

    What are the Key Features of Airflow and Prefect?

    Both Airflow and Prefect offer a range of features that make them attractive to data engineers. Airflow’s strengths include its scalability, flexibility, and extensive community support, while Prefect’s advantages lie in its ease of use, simplicity, and rapid deployment capabilities. Here’s a brief overview of each tool’s key features:

    How Do I Get Started with Airflow or Prefect?

    Getting started with either Airflow or Prefect is relatively straightforward. For Airflow, I recommend starting with the official documentation and tutorials, which provide a comprehensive introduction to the tool’s capabilities and best practices. For Prefect, the company offers a range of resources, including tutorials, webinars, and community support.

    A realistic estimate for a team with 40-60 DAGs: four to eight weeks. Not a weekend project. Budget time for the operational work, not just the code conversion.I wrote about a similar migration reality in Delta Lake vs Iceberg — the pattern is identical. The data conversion is the easy part


    When to Choose Airflow

    • You’re already running it and it’s stable — migration cost is real
    • You need enterprise-scale reliability with proven track record
    • Your team has strong Airflow expertise and hiring for it is important
    • You’re on a managed service (MWAA, Astronomer) and the overhead is already handled
    • You need the broadest possible integration ecosystem

    When to Choose Prefect

    • You’re starting fresh with no existing orchestration investment
    • You have a small team without dedicated platform engineering
    • You’re building ML or AI pipelines that need dynamic task mapping
    • Your engineers are strong Python developers who find Airflow’s operator model unnatural
    • Developer velocity matters more than ecosystem breadth right now

    What I’d Do Differently

    I’d have adopted Prefect for our ML pipelines immediately, even while keeping Airflow for everything else. The two tools can coexist. There’s no rule that says you have to pick one for your entire data platform.

    For new batch ETL on stable sources? Airflow. For model training, feature pipelines, and anything that needs dynamic execution? Prefect. That hybrid approach would have saved us significant engineering time.

    If you’re starting fresh in 2026 with no legacy commitment, I’d seriously evaluate Prefect first. Airflow 3.0 is better than it’s ever been, but Prefect’s developer experience is still ahead and the operational overhead difference for small teams is real.


    Frequently Asked Questions

    As I’ve worked with both Airflow and Prefect, I’ve encountered some common questions from data engineers and teams. Here are a few answers to help you get started:

    Q: What’s the main difference between Airflow and Prefect?

    Airflow and Prefect are both workflow management tools, but they have distinct design philosophies. Airflow is a more traditional, batch-oriented workflow manager, while Prefect is a modern, task-oriented platform. Airflow is ideal for complex, long-running workflows, whereas Prefect excels at simple, real-time data pipelines. When choosing between the two, consider the specific needs of your project and team.

    Q: Can I use Airflow and Prefect together in my data pipeline?

    Absolutely! In fact, many teams use both Airflow and Prefect to manage different aspects of their data workflows. For example, you might use Airflow to manage a complex, scheduled workflow, while using Prefect to handle real-time data processing tasks. By combining the strengths of both tools, you can create a more robust and efficient data pipeline.

    Q: How do I decide which tool is best for my team’s specific use case?

    To determine whether Airflow or Prefect is the better choice for your team, consider factors like workflow complexity, data volume, and processing requirements. Ask yourself: What are our specific pain points? What kind of workflows do we need to manage? What are our scalability and performance requirements? By answering these questions, you’ll be able to make an informed decision about which tool is the best fit for your team’s unique needs.

    Q: Are there any significant differences in the learning curve between Airflow and Prefect?

    Yes, the learning curves for Airflow and Prefect differ. Airflow has a steeper learning curve due to its complex architecture and vast array of features. Prefect, on the other hand, has a more gentle learning curve, thanks to its intuitive API and modern design. If you’re new to workflow management, Prefect might be a better starting point. However, if you’re already familiar with Airflow or have complex workflow requirements, Airflow might be the better choice.

    Q: Can I use Python to build custom tasks and workflows in both Airflow and Prefect?

    Yes, both Airflow and Prefect support Python as a first-class citizen. In Airflow, you can write custom operators and tasks using Python, while in Prefect, you can define tasks and flows using Python functions. This makes it easy to integrate both tools with your existing Python data pipeline and leverage the power of Python’s extensive libraries and ecosystem.

  • Delta Lake vs Apache Iceberg — Why I Chose Iceberg for Our Data Lakehouse

    Delta Lake vs Apache Iceberg — Why I Chose Iceberg for Our Data Lakehouse

    TL;DR
    → Delta Lake is easier to start with, especially if you’re already on Databricks
    → Iceberg wins on engine flexibility — works natively with Spark, Flink, Trino, Snowflake, and more without custom connectors
    → Delta Lake’s vendor coupling with Databricks is a real cost if you’re multi-cloud or multi-engine
    → Iceberg’s partition evolution lets you change partition schemes without rewriting data — that feature alone saved us a full weekend of migration work
    → Migration from Delta to Iceberg is harder than most blog posts suggest — budget four to eight weeks, not a weekend
    → If you’re greenfield, start with Iceberg. If Delta is working, don’t migrate until you hit a specific limit


    I didn’t choose Iceberg because I read a benchmark blog post. I chose it after six months of hitting Delta Lake’s limits in ways that weren’t obvious until they were expensive.

    We were running a mid-sized data lakehouse — S3-backed, Spark for processing, Snowflake for consumption, dbt for transformation. Delta Lake was the default choice. Everyone on the team had used it before. The documentation was solid. It worked — until it didn’t.

    This isn’t a “here are the specs” comparison. You can get that from the docs. This is what actually happened when I ran both in production, why I made the switch, and what I’d tell you before you pick one.


    What We Were Actually Trying to Solve

    Before I get into the comparison, context matters. Our stack at the time: raw data landing in S3, Apache Spark for heavy transformation, Snowflake as the consumption layer for analysts, dbt for modeling, and Apache Airflow for orchestration.

    We needed ACID transactions on S3, time travel for debugging, and the ability to do incremental loads without full partition rewrites. Delta Lake checked all those boxes — initially. The problems showed up at scale and at the edges.


    Where Delta Lake Started Hurting Us

    Engine Lock-In Was a Real Problem

    Delta Lake works great if Spark is your only compute engine. The moment we tried to query Delta tables directly from Snowflake or Trino, things got complicated. Delta’s transaction log format is proprietary. You need the Delta connector — and not every engine has a first-class one.

    We wanted analysts to query raw lakehouse tables directly from Snowflake without going through Spark first. With Delta, that required Snowflake’s Delta Sharing integration, which had limitations on what operations were supported. It wasn’t broken, but it added friction and another dependency to manage.

    Apache Iceberg solves this cleanly. The table format is open. Snowflake, Spark, Flink, Trino, Athena, Dremio — they all read and write Iceberg natively. No connectors to manage. No format translation layer.

    Partition Management Was Getting Messy

    With Delta Lake, partitioning decisions are set at table creation. Changing a partition scheme means rewriting the table. At 100M+ rows, that’s not a quick operation.

    We had a table partitioned by event_date. Six months in, query patterns changed — analysts were filtering by event_date and region together. Repartitioning meant a full backfill job over a weekend, plus repointing all downstream dbt models.I wrote about a similar pain point in the problem with dbt incremental models — the pattern is the same.

    Iceberg’s partition evolution lets you change the partition spec without rewriting data. Old data stays as-is. New data uses the new scheme. Queries still work against both.

    Hidden Partitioning Changed How We Design Tables

    Iceberg supports hidden partitioning — you define partition transforms like days(event_timestamp) or bucket(user_id, 16) and Iceberg handles physical partitioning transparently. Your queries don’t need to know about partition columns. The engine prunes automatically.

    With Delta Lake, you need to explicitly filter on partition columns or you’ll scan everything. That’s fine when everyone knows the rules. It’s a problem when a new analyst writes a query without knowing which columns are partition keys.


    Where Delta Lake Is Still Better

    If you’re on Databricks, stay on Delta. The integration is tight, the tooling is mature, and Databricks has invested heavily in Delta’s performance.. Liquid Clustering makes partition management much more flexible. If Databricks is your primary compute layer, switching to Iceberg gives you marginal benefit for non-trivial migration cost.

    Delta’s MERGE performance on Spark is excellent. For high-frequency CDC workloads where you’re doing upserts at scale on Spark, Delta’s MERGE implementation is well-optimised. Iceberg’s MERGE has improved significantly but Delta still has an edge in some Spark-specific CDC patterns.

    Delta has simpler operational overhead for small teams. Delta’s transaction log is easier to reason about. The tooling for vacuum, optimize, and Z-ordering is well-documented and predictable.


    The Comparison You Actually Need

    Feature Delta Lake Apache Iceberg
    Engine supportSpark-native; connectors for othersTruly multi-engine (Spark, Flink, Trino, Snowflake, Athena)
    Partition evolutionRequires full table rewriteSchema-safe, no data rewrite needed
    Hidden partitioningNot supportedSupported — engines auto-prune
    MERGE / CDC performanceExcellent on SparkStrong, improving; slightly behind Delta on Spark CDC
    Vendor alignmentDatabricks ecosystemVendor-neutral, Apache foundation
    Operational toolingMature, well-documentedMaturing fast; strong in 2024–2025
    Multi-cloud flexibilityPossible but frictionFirst-class support across clouds
    Migration effortN/A (starting point)Non-trivial; plan 4–8 weeks

    THE MIGRATION: WHAT IT ACTUALLY COST US

    The Migration: What It Actually Cost Us

    I’ll be direct: the migration was harder than I expected. If you’ve read my piece on automation in data engineering, you’ll recognise the pattern — the technical part is rarely the hard part. It’s the downstream work nobody accounts for.

    The core work wasn’t the data conversion — we used the delta-iceberg migration utility and it handled most of the heavy lifting. The harder parts were everything else.

    Downstream dependency mapping.

     Every dbt model, every Airflow DAG, every Spark job that referenced a Delta table path needed updating. We had 40+ models. Two had hardcoded partition paths we didn’t catch until QA.

    Metadata catalog updates.

     We use AWS Glue Data Catalog. Every table needed its metadata updated to reflect the Iceberg format. Glue’s Iceberg support has improved, but it’s not frictionless.

    Testing the rollback plan. We kept Delta tables live for 30 days post-migration with a cutover switch in Airflow. That meant double-writing during the transition window — additional storage cost and added pipeline complexity.

    ⚠️ The migration trap: The data conversion tooling works. What catches teams off guard is the downstream mapping work — every pipeline, model, and job that references a table path. Budget more time for that than for the actual format conversion.

    Total elapsed time: six weeks. Two engineers. Not a weekend project.


    When to Choose Delta Lake

    • Your primary compute layer is Databricks
    • You’re a small team that wants simpler operations
    • You’re doing high-frequency CDC on Spark
    • You’re early stage — get something working first

    When to Choose Iceberg

    • You’re running multiple query engines (Spark + Snowflake, Trino + Flink)
    • You need partition evolution without full table rewrites
    • You’re building a vendor-neutral architecture
    • Your analysts query the lakehouse directly from Snowflake

    What I’d Do Differently

    Start with Iceberg if you’re greenfield. The setup is slightly more involved, but you avoid the migration cost entirely. The ecosystem has matured enough in 2024-2025 that “Iceberg is less mature” is no longer a strong argument.

    If you’re already on Delta and it’s working — don’t migrate for the sake of it. Migrate when you hit a specific limit: engine lock-in, partition inflexibility, or multi-cloud requirements.

    And if you do migrate, don’t underestimate the downstream mapping work. The data conversion is the easy part.


    Frequently Asked Questions

    What is the main difference between Delta Lake and Apache Iceberg?

    Delta Lake is a table format developed by Databricks, optimised for Spark workloads with strong Databricks integration. Apache Iceberg is an open table format designed for multi-engine environments — it works natively with Spark, Flink, Trino, Snowflake, and Athena without custom connectors. The core difference is engine flexibility.

    Is Apache Iceberg better than Delta Lake?

    It depends on your stack. Iceberg is better if you’re running multiple query engines or building a vendor-neutral architecture. Delta Lake is better if Databricks is your primary compute layer. Neither format is objectively superior.

    Can Snowflake read Delta Lake tables?

    Yes, through Delta Sharing or Snowflake’s Delta connector — but with limitations. Snowflake reads Iceberg tables natively as a first-class citizen, which is why multi-engine stacks tend to favour Iceberg.

    How hard is it to migrate from Delta Lake to Apache Iceberg?

    Harder than most blog posts suggest. The data conversion tooling handles the format migration, but remapping downstream pipelines, updating metadata catalogs, and testing rollback scenarios adds significant effort. Budget four to eight weeks for a production migration with 30–50 tables.

    Does dbt support Apache Iceberg?

    Yes. dbt supports Iceberg through the Spark and Athena adapters, and Snowflake’s Iceberg table support works with dbt models running on Snowflake. Production-ready as of 2024.

    What is hidden partitioning in Apache Iceberg?

    Hidden partitioning lets Iceberg manage partition logic transparently. You define partition transforms like days(event_timestamp) at the table level, and Iceberg handles physical file organisation and query pruning automatically — no need to filter on partition columns explicitly.

  • The Problem with Data Engineering Certifications That Nobody Talks About

    The Problem with Data Engineering Certifications That Nobody Talks About

    I passed the SnowPro Gen AI certification not too long ago. Within the same week I was back at my desk staring at a broken pipeline that no multiple-choice question had ever prepared me for. The cert looked great on my profile. It fixed exactly nothing about the actual problem in front of me.

    I’m not saying certifications are worthless. I’m saying the industry has developed a quietly dishonest relationship with them — one where vendors, hiring managers, and candidates all play along with a fiction that a passed exam means something it doesn’t. Nobody wants to be the one to say it out loud.

    So I will. Let me be direct about what’s actually going on.


    TL;DR

    • Certifications test what vendors want you to know about their products — not whether you can actually engineer data systems that work under real conditions
    • The exam content is often months or years behind the tools you’ll actually use in production
    • Hiring managers use certs as a filter because it’s easy — not because it’s accurate
    • You can pass most data engineering certs with two weeks of practice exams and zero production experience
    • The real signal employers should care about — and rarely do — is what you’ve built, what broke, and what you learned from it
    • Certifications have a specific, narrow value: they are a vocabulary test, not a competence test. Know what you’re paying for

    WHAT CERTIFICATIONS ACTUALLY TEST

    Let’s start with what’s literally on the exam. Take the Databricks Certified Data Engineer Associate . The exam covers Delta Lake concepts, basic Spark operations, Unity Catalog, Databricks workflows. Good things to know.

    But the exam tests your ability to identify the correct answer from four options in a controlled environment. It does not test whether you can debug a production Spark job that’s been running for six hours and slowly consuming memory. It doesn’t test whether you can diagnose why a Delta merge is creating file fragmentation degrading query performance. It doesn’t test whether you can architect a pipeline that recovers gracefully when an upstream API starts returning malformed JSON at 3am.

    Those are the problems data engineers actually face. None of them are in the certification.

    A certification tells you that someone understood the conceptual framework of a product well enough to pass a vendor-designed exam. It tells you almost nothing about their ability to operate that product under adversarial conditions. And production is always adversarial.

    This gap exists in the AWS Certified Data Engineer Associate ,the Google Professional Data Engineer ,the Azure Data Engineer Associate ,and every dbt or Snowflake certification available. They all test the vendor’s idealised scenario.

    Real pipelines are never idealised.


    THE VENDOR INCENTIVE PROBLEM

    Who designs these exams? The vendors. Who benefits when thousands of engineers study for, pay for, and pass these exams? The vendors. Certification programmes are not primarily educational products. They are marketing products that create a credentialled user base and deepen platform lock-in.

    When Snowflake designs its certification exams ,the goal is not to produce engineers who can evaluate whether Snowflake is the right tool. The goal is to produce engineers deeply familiar with Snowflake’s architecture, syntax, and product positioning — engineers who will advocate for Snowflake when tooling decisions come up at their company.

    The exam content is shaped by commercial interest, not by what data engineers actually need to know. The practical consequence: certifications optimise for breadth of product knowledge over depth of engineering judgment. You learn feature names, service limits, and recommended architectures. You don’t develop the instinct that tells you something is going to break before it breaks.


    THE HIRING MANAGER TRAP

    I’ve sat in hiring discussions where a candidate without certifications was dismissed faster than one with a string of logos after their name, despite the uncertified candidate having a demonstrably stronger GitHub portfolio and much more interesting answers about production incidents they’d owned.

    Certifications persist in job postings because they’re easy to verify and hard to argue with. A cert is binary. Either you have it or you don’t. Technical judgment, architecture instinct, debugging ability — these require effort to assess.

    ⚠️ The signal problem: If you can pass a data engineering certification with two weeks of practice exams and no production experience — and you can — then having the certification tells an interviewer almost nothing about whether you can do the job. It tells them you can study for a test. That’s useful. But it’s not the same thing.

    The engineers most dismissive of certifications are often the most experienced. The engineers who lean most heavily on cert lists are often the ones who haven’t done enough production work to know what the gap actually looks like.


    THE STALE CONTENT PROBLEM

    Data engineering moves fast. The tooling landscape in 2024 looks materially different from 2021. dbt Core has changed substantially. Apache Iceberg has gone from niche to mainstream. Lakehouse architecture has shifted from concept to default.

    Certification exams do not move at this speed. Exam content is updated infrequently — sometimes annually, sometimes less. You can hold an AWS Data Engineer cert that emphasises EMR and Glue in patterns most teams have replaced with more modern tooling. You can hold a Databricks cert that doesn’t reflect how Unity Catalog has fundamentally changed governance.

    The cert is not wrong. It’s just dated. And dated knowledge in data engineering isn’t neutral — it can actively mislead you about how things should be built.

    I wrote about a related version of this in “Why I Stopped Using Snowflake Tasks for Orchestration” — official documentation and certification content often lags behind what practitioners have already learned through trial and error in production.


    WHAT YOU ACTUALLY LEARN WHEN YOU STUDY FOR A CERT

    Here’s the part I want to be fair about. Studying for a data engineering certification isn’t worthless. It’s just worth something different from what most people think.

    When you study for the Google Professional Data Engineer exam, you learn the GCP data ecosystem — BigQuery, Dataflow, Pub/Sub, Cloud Composer, Dataproc — in a structured way. You develop a vocabulary. You understand how services relate to each other.

    What it doesn’t give you is judgment. Judgment about when to use Dataflow versus Dataproc. When BigQuery’s cost model makes it the wrong tool despite its performance. When a simple Cloud Function is a better answer than a fully orchestrated pipeline.

    The honest framing: a certification is a vocabulary test with a structured curriculum. If you’ve never worked on a platform and need to get up to speed quickly, studying for the cert is efficient. If you already have production experience, the cert adds limited signal beyond what’s already on your resume.


    THE PRACTICE EXAM LOOPHOLE NOBODY WANTS TO DISCUSS

    Most data engineering certifications can be passed with aggressive practice exam grinding and minimal practical experience. Platforms like Udemy , Whizlabs and ExamTopics sell practice exam bundles close enough to real questions that a disciplined studier can reverse-engineer most of the exam in two to three weeks.

    I’ve seen candidates with zero Snowflake production experience pass the SnowPro Core exam in a week of evening study. I’ve seen engineers memorise their way through the AWS Data Engineer Associate without writing a single Glue job. The credential is indistinguishable from someone who earned it through genuine depth.

    The vendors know this. They update exam content periodically to counter braindump culture, but it’s an arms race they’re perpetually losing.


    WHAT ACTUALLY SIGNALS ENGINEERING COMPETENCE

    If I’m hiring a data engineer, here’s what I actually want to see.

    Tell me about a pipeline that broke in production. Not a hypothetical. What broke, how you found out, what the root cause was, how you fixed it, what you changed to prevent recurrence. This conversation reveals more engineering judgment than any certification.

    Show me something you built. A GitHub repo .A dbt project. A pipeline architecture diagram with a written explanation. The work I’ve been documenting — from the problem with dbt incremental models to Snowflake zero-copy cloning gotchas — is far more useful signal than any certification I hold.

    Tell me about a technical decision you disagreed with. Engineering judgment includes knowing when to push back, when to compromise, how to argue for a position with evidence. No cert tests this.

    Walk me through how you’d approach this problem. Give them a real scenario — a data quality issue, a cost spike, a schema migration in a live system. Watch how they think, not just what they know.

    The gap between what certifications measure and what engineering competence looks like is large enough that I’d rather see zero certifications with a detailed post-mortem of a real incident than four certs with nothing to show for the work.


    WHEN CERTIFICATIONS ARE ACTUALLY WORTH PURSUING

    You’re breaking into the field. If you’re transitioning into data engineering, certifications serve a genuine purpose. They give you structured curriculum and a credential that signals seriousness to employers who don’t yet have anything else to evaluate you on.

    Your employer requires it. Many enterprise organisations and consulting firms have vendor partnership requirements mandating certified staff levels. In that case, the cert has real organisational value regardless of signal quality.

    You’re learning a new platform systematically. Using cert study as structured onboarding to a new tool is legitimate. The curriculum forces breadth coverage self-directed learning often misses. Just know that completing the cert doesn’t mean you know how to use the platform well.

    You’re in a market where it’s table stakes. In some geographies and sectors, certain certs are required to get an interview. Clear the gate, then demonstrate real depth in the room.

    The certification isn’t the problem. The mythology around it is. The idea that passing the exam means you can build reliable data systems — that’s the fiction that causes real damage.


    WHAT THE INDUSTRY SHOULD DO INSTEAD

    Portfolio-based evaluation. A documented data engineering project — architecture decisions, tradeoffs, failures encountered — tells a hiring team far more than an exam score. GitHub already supports this.

    Incident post-mortems as credentials. A well-written post-mortem demonstrates debugging methodology, systems thinking, and the ability to learn from failure. No certification tests these.

    Practical assessments over multiple choice. The Databricks Data Engineer Professional is harder than most — it has a coding component requiring actual proficiency. More exams should work this way.

    Open curriculum from neutral sources. The Data Engineering Handbook and open-source community resources are doing more for actual engineering capability than most vendor certification programmes.


    FREQUENTLY ASKED QUESTIONS

    Are data engineering certifications worth it in 2024?
    It depends on where you are in your career. For someone entering the field, certs provide structured curriculum and a credential that signals seriousness. For experienced engineers, your production track record carries far more weight with strong technical hiring teams. Certs are worth what they cost if you understand what they are: a vocabulary test, not a competence test.

    Which data engineering certification is the most respected?
    Among practitioners, the Databricks Data Engineer Professional is generally seen as harder and more meaningful because it includes a practical component. Google Professional Data Engineer has strong enterprise name recognition. AWS Certified Data Engineer Associate is widely recognised in cloud-native teams. But respected by whom matters — strong engineering teams care less about cert logos than about demonstrated ability.

    Can you become a data engineer without certifications?
    Absolutely. Many strong data engineers have no certifications at all. A track record of real work — systems built, incidents resolved, architectural decisions owned — is equally or more compelling to technical hiring teams worth impressing.

    How long does it take to pass data engineering certification exams?
    Most candidates report 2–6 weeks of focused study. With aggressive practice exam preparation, some pass in under two weeks — which is part of what makes the credentials less meaningful than they appear.

    Do data engineering certifications expire?
    Yes. AWS certifications expire after three years, Google Cloud after two, Databricks varies by level. Recertification tends to be easier than initial certification and often doesn’t reflect how dramatically the tooling has evolved.

    What should a data engineering portfolio include instead of certifications?
    End-to-end pipeline projects with documented architecture decisions. Written post-mortems of production incidents. Data quality testing approaches. dbt projects with meaningful transformation logic. Cost analyses or performance optimisations from real environments. Anything that shows how you think, not just what tools you’ve touched.


  • The Problem with Zero-Copy Cloning in Snowflake That Nobody Talks About

    The Problem with Zero-Copy Cloning in Snowflake That Nobody Talks About

    Every time I demo Snowflake to someone new, zero-copy cloning gets the biggest reaction. You type one line. You get an instant copy of a table — or an entire database — with no data duplication, no storage cost at the moment of creation. It feels like magic.

    And it is genuinely impressive engineering. I’m not here to tell you it’s a bad feature. It’s one of my favourite things about Snowflake and I use it constantly.

    But I’ve watched teams get badly surprised by it. A dev environment clone that started silently inflating the storage bill. A cloned database used for UAT that bypassed data masking policies on PII columns. A Time Travel query on a clone that returned nothing because the source table’s retention window had already expired.

    None of these are edge cases. They’re predictable consequences of how zero-copy cloning actually works — consequences that the marketing language around “instant, free copies” tends to obscure. Let me get into it.

    TL;DR

    • Zero-copy cloning is one of Snowflake’s best features — and one of the most misunderstood ones in production
    • Clones share micropartitions with the source — any modification to either side starts writing new storage, and that cost adds up fast in ways that aren’t visible upfront
    • Clones don’t inherit resource monitors, row-level security policies, or dynamic data masking by default — this is a compliance and governance trap waiting to happen
    • Time Travel on clones behaves differently from what most people expect, especially when the source table has already moved past its retention window
    • Clone sprawl is real — it’s invisible in the UI, expensive to audit, and teams rarely have a cleanup strategy until the bill arrives
    • This article covers what zero-copy cloning actually does under the hood, where it silently fails you, and how to use it without it becoming a liability

    HOW ZERO-COPY CLONING ACTUALLY WORKS

    When you clone a table in Snowflake, you’re not copying data. You’re creating a new metadata pointer that references the same underlying micropartitions as the source object.

    -- Instant. No data movement. No storage cost at this moment.
    CREATE TABLE orders_clone CLONE orders;
    
    -- Works for schemas too
    CREATE SCHEMA dev_schema CLONE prod_schema;
    
    -- And entire databases
    CREATE DATABASE dev_db CLONE prod_db;

    At the moment of creation, the clone costs you nothing in storage. Both the original and the clone point to the same micropartitions on disk. The moment either side changes, Snowflake uses copy-on-write. The modified micropartition gets written fresh for whichever side made the change.

    Think of it like a fork in a Git repo. At fork time, both repos share the same commit history. The moment either side commits, they diverge. The more divergence, the more independent storage you accumulate. Zero-copy cloning works exactly like this — except the “commits” are DML operations and the cost is real money.


    PROBLEM 1 — STORAGE COSTS THAT CREEP UP INVISIBLY

    A team clones production to create a development environment. The dev team runs experiments, updates records, backfills some columns. Six weeks later, storage is up 40%. Every modified micropartition in the dev database is now independent storage. Production kept its micropartitions too. You’re paying for both.

    The cost also compounds with Time Travel. If production has 90-day retention and you clone it for dev, that clone also starts with 90-day retention. DML operations in dev accumulate 90 days of write history.

    -- Clone with reduced Time Travel for non-production environments
    CREATE DATABASE dev_db CLONE prod_db;
    
    -- Immediately reduce Time Travel on the clone
    ALTER DATABASE dev_db SET DATA_RETENTION_TIME_IN_DAYS = 1;
    
    -- Or set it at schema level for finer control
    ALTER SCHEMA dev_db.analytics SET DATA_RETENTION_TIME_IN_DAYS = 0;

    Audit clone storage footprint regularly:

    -- Find clones and their storage footprint
    SELECT
        table_catalog,
        table_schema,
        table_name,
        clone_group_id,
        bytes / (1024 * 1024 * 1024)        AS size_gb,
        row_count,
        created                              AS clone_created_at
    FROM snowflake.account_usage.tables
    WHERE clone_group_id IS NOT NULL
      AND deleted IS NULL
    ORDER BY bytes DESC;

    This same class of invisible cost creep comes up a lot with Snowflake features that look free until you read the bill — similar to what I covered in “Why I Stopped Using Snowflake Tasks for Orchestration


    PROBLEM 2 — GOVERNANCE POLICIES DON’T FOLLOW THE CLONE

    Dynamic data masking policies are not automatically inherited by clones. The clone is a new object with no masking policies applied.

    -- On prod table — email is masked for non-PII roles
    SELECT email FROM prod_orders LIMIT 5;
    -- Result: ***@***.com (masked)
    
    -- On the clone without explicit policy assignment
    SELECT email FROM orders_clone LIMIT 5;
    -- Result: [email protected] (unmasked raw PII)

    Same problem with row access policies. A user restricted to one region in production can see all regions on the clone.

    The fix — make policy application part of your clone process:

    -- Step 1: Clone the table
    CREATE TABLE dev_db.analytics.orders CLONE prod_db.analytics.orders;
    
    -- Step 2: Re-apply masking policies immediately
    ALTER TABLE dev_db.analytics.orders
        MODIFY COLUMN email
        SET MASKING POLICY prod_db.security.email_mask;
    
    ALTER TABLE dev_db.analytics.orders
        MODIFY COLUMN phone_number
        SET MASKING POLICY prod_db.security.phone_mask;
    
    -- Step 3: Re-apply row access policy
    ALTER TABLE dev_db.analytics.orders
        ADD ROW ACCESS POLICY prod_db.security.region_access_policy
        ON (region_code);

    Better: wrap it in a stored procedure that enforces policy application as part of the clone operation:

    CREATE OR REPLACE PROCEDURE create_governed_clone(
        source_table      VARCHAR,
        target_table      VARCHAR,
        masking_policies  ARRAY
    )
    RETURNS STRING
    LANGUAGE JAVASCRIPT
    AS
    $$
        var clone_stmt = snowflake.execute({
            sqlText: `CREATE TABLE ${TARGET_TABLE} CLONE ${SOURCE_TABLE}`
        });
    
        for (var i = 0; i < MASKING_POLICIES.length; i++) {
            var policy = MASKING_POLICIES[i];
            snowflake.execute({
                sqlText: `ALTER TABLE ${TARGET_TABLE}
                          MODIFY COLUMN ${policy.column}
                          SET MASKING POLICY ${policy.policy_name}`
            });
        }
    
        return 'Clone created with governance policies applied: ' + TARGET_TABLE;
    $$;

    A clone that exists without its governance policies re-applied is a compliance gap, not a convenience feature. If you’re running dbt on top of Snowflake, the same mindset applies — see “The Problem with dbt Tests Nobody Talks About“:


    PROBLEM 3 — TIME TRAVEL ON CLONES ISN’T WHAT YOU THINK

    A clone’s Time Travel history starts from its creation date. You cannot go back to a point before the clone was created on the clone object.

    -- Source table: exists since 2024-01-01, 90-day retention
    -- Clone created: 2024-03-01
    
    -- This works — within the clone's own history
    SELECT * FROM orders_clone
    AT (TIMESTAMP => '2024-03-15 10:00:00'::TIMESTAMP_TZ);
    
    -- This FAILS — before the clone existed
    SELECT * FROM orders_clone
    AT (TIMESTAMP => '2024-02-01 10:00:00'::TIMESTAMP_TZ);
    -- Error: Statement time travel is not available for this object
    
    -- For pre-clone history, query the SOURCE table
    SELECT * FROM orders
    AT (TIMESTAMP => '2024-02-01 10:00:00'::TIMESTAMP_TZ);

    Also watch: if you clone a table that’s near the end of its retention window, any history that expires on the source is gone. The clone can’t access expired source history.


    PROBLEM 4 — CLONE SPRAWL AND THE INVISIBLE COST PROBLEM

    Zero-copy cloning is so easy that people create clones for everything — UAT, load testing, feature branches, one-off investigations that were supposed to be deleted on Friday. Three months later, nobody knows what exists or how diverged it’s become.

    Full clone audit query:

    SELECT
        t.table_catalog                                   AS database_name,
        t.table_schema                                    AS schema_name,
        t.table_name,
        t.clone_group_id,
        t.row_count,
        ROUND(t.bytes / POW(1024, 3), 3)                  AS size_gb,
        t.created                                         AS created_at,
        t.last_altered                                    AS last_modified_at,
        DATEDIFF('day', t.created, CURRENT_TIMESTAMP())   AS age_days,
        CASE
            WHEN DATEDIFF('day', t.last_altered, CURRENT_TIMESTAMP()) > 30
            THEN 'STALE — review for deletion'
            ELSE 'Active'
        END AS staleness_flag
    FROM snowflake.account_usage.tables t
    WHERE t.clone_group_id IS NOT NULL
      AND t.deleted IS NULL
    ORDER BY t.bytes DESC;

    Tag every clone at creation with expiry metadata:

    CREATE DATABASE uat_db CLONE prod_db
        COMMENT = '{"purpose": "UAT for v2.4 release", "owner": "[email protected]", "expires": "2024-04-30", "ticket": "JIRA-1234"}';
    
    -- Query clones past their expiry date
    SELECT
        table_catalog,
        table_schema,
        table_name,
        TRY_PARSE_JSON(comment):expires::DATE AS expiry_date,
        TRY_PARSE_JSON(comment):owner::STRING AS owner
    FROM snowflake.account_usage.tables
    WHERE clone_group_id IS NOT NULL
      AND deleted IS NULL
      AND TRY_PARSE_JSON(comment):expires::DATE < CURRENT_DATE();

    PROBLEM 5 — CLONING STREAMS AND TASKS DOESN’T WORK HOW YOU EXPECT

    Streams are not cloned when you clone a table or schema. The clone contains the data but has no streams attached.

    -- Prod table has a stream attached
    SHOW STREAMS ON TABLE prod_db.analytics.orders;
    -- Returns: orders_cdc_stream
    
    -- Clone the table
    CREATE TABLE dev_db.analytics.orders CLONE prod_db.analytics.orders;
    
    -- Check streams on clone
    SHOW STREAMS ON TABLE dev_db.analytics.orders;
    -- Returns: (empty)

    If you need CDC streams on cloned tables, create them explicitly after cloning:

    CREATE OR REPLACE STREAM dev_db.analytics.orders_cdc_stream
        ON TABLE dev_db.analytics.orders
        APPEND_ONLY = FALSE
        SHOW_INITIAL_ROWS = FALSE;
    
    CREATE OR REPLACE TASK dev_db.analytics.process_orders_changes
        WAREHOUSE = dev_wh
        SCHEDULE = '5 minute'
        WHEN SYSTEM$STREAM_HAS_DATA('dev_db.analytics.orders_cdc_stream')
    AS
        CALL dev_db.analytics.process_orders_sp();

    Tasks are cloned but start in a SUSPENDED state — they don’t auto-resume, which is correct behaviour (you don’t want dev tasks firing against prod targets), but it surprises teams expecting a live pipeline copy. If your pipeline relies on dbt incremental models consuming from those streams, the failure compounds further — see “The Problem with Incremental Models in dbt Nobody Talks About


    WHEN ZERO-COPY CLONING IS THE RIGHT TOOL

    Before risky migrations — clone first, get an instant rollback point:

    -- Before a risky migration
    CREATE TABLE orders_pre_migration CLONE orders;
    
    -- Run your migration
    ALTER TABLE orders ADD COLUMN new_column VARCHAR;
    UPDATE orders SET new_column = derive_value(existing_column);
    
    -- If something went wrong:
    -- DROP TABLE orders;
    -- ALTER TABLE orders_pre_migration RENAME TO orders;

    Instant dev environments, UAT cycles, zero-downtime data fixes — all excellent use cases. The feature is great. Using it without understanding the lifecycle is where teams get into trouble. If you want to go further on cost reduction for dev workloads, pairing clone strategy with DuckDB is worth exploring — “How to Query Snowflake in DuckDB and Cut Your Bill While Doing It


    FREQUENTLY ASKED QUESTIONS

    Q: Does zero-copy cloning in Snowflake really cost nothing?
    A: At creation: yes. The cost begins the moment either side is modified via copy-on-write. In active dev environments that are modified frequently, storage costs can grow significantly over weeks. Time Travel retention on the clone compounds this further.

    Q: Do data masking policies transfer when you clone a table?
    A: No. Masking policies are not inherited by clones. Sensitive columns are exposed in plaintext on the clone unless you explicitly re-apply policies after creation. Treat clone creation and policy application as a single atomic operation.

    Q: Can I use Time Travel on a clone to go back before it was created?
    A: No. A clone’s Time Travel history starts at its creation date. For history before the clone was created, query the source table directly.

    Q: Are Snowflake Streams copied when you clone a table?
    A: No. Streams are not part of the clone operation. Create them explicitly on the clone if your pipeline depends on CDC. Tasks are cloned but start suspended.

    Q: How do I audit all clones in my Snowflake account?
    A: Query snowflake.account_usage.tables filtering on clone_group_id IS NOT NULL. Tag clones at creation with JSON metadata in the COMMENT field — owner, expiry, purpose — to make audits actionable.

    Q: What’s the best practice for cloning production for dev?
    A: Clone, then immediately: reduce Time Travel retention to 0 or 1 day, re-apply all masking and row access policies, set a resource monitor on dev warehouses, and tag the clone with an expiry date in the COMMENT field.


    Related blogs

    → Snowflake official docs — cloning objects
    → Snowflake dynamic data masking docs
    → Snowflake Time Travel docs
    → Snowflake resource monitors docs
    → Snowflake Streams