Category: Snowflake

Dive deep into the Snowflake Data Cloud. Guides on building a modern cloud data warehouse, data sharing, performance optimization, and leveraging advanced features like Snowpipe and Streams.

  • The 2026 Migration Trap: Moving from Native Tables to Dynamic Apache Iceberg v3 in Snowflake

    The 2026 Migration Trap: Moving from Native Tables to Dynamic Apache Iceberg v3 in Snowflake

    The pitch is intoxicating and mostly true: keep your data in open Apache Iceberg format on your own object storage, let external engines read it, and let Snowflake’s Dynamic Tables handle the low-latency transformations on top — one declarative pipeline, no lock-in, a real lakehouse. In 2026, with Iceberg v3 generally available on Snowflake since May 7, teams are migrating native tables to dynamic Iceberg tables expecting exactly that. Most of them hit the same wall in the same order.

    The wall is this: “open and interoperable” describes the storage format, not the write path, and “low latency” describes Dynamic Tables under conditions that partitioned Iceberg writes and cross-engine change tracking quietly violate. The migration doesn’t fail loudly. It succeeds, ships, and then your incremental pipeline starts doing full refreshes you didn’t ask for, your partitioned writes fan out into a metadata problem, and the external engine you promised could write to these tables turns out to be read-only. This is the guide to the traps — the ones that don’t show up in the quickstart — and how to design around them before they cost you a quarter.

    TL;DR

    → Iceberg v3 (GA on Snowflake May 2026) brings deletion vectors, row lineage (native CDC), VARIANT, and multi-argument partition transforms. You cannot upgrade v2 to v3 in place — you recreate the table. Plan the migration, don’t expect an ALTER.

    → Dynamic Iceberg tables support PARTITION_BY, TARGET_FILE_SIZE, and PATH_LAYOUT. PATH_LAYOUT = HIERARCHICAL only produces Hive-style partitioned paths when paired with PARTITION_BY — and over-partitioning (more than a few thousand partitions) turns your metadata layer into the bottleneck.

    → The cross-engine reality: Snowflake-managed Iceberg tables are read-write for Snowflake, read-only for external engines. Writes from external engines to Snowflake-managed v3 tables via Horizon Catalog aren’t supported yet. External engines can only write to externally-managed tables.

    → Dynamic Tables track changes at the row level for native tables but the file level for externally-managed Iceberg base tables. Frequent copy-on-write on the external table degrades incremental refresh — a file changes, the whole file is “changed.”

    → INSERT OVERWRITE on a base table resets change-tracking metadata and forces a full refresh. Row lineage (v3) and primary keys with RELY are how you keep incrementality alive across rewrites.

    → Deletion vectors (v3 merge-on-read) are governed by heuristics: Snowflake only writes a deletion vector if fewer than ~5% of a file’s rows are deleted and the file is larger than ~1.6 MB. External engines that don’t understand v3 deletion vectors force you to set ICEBERG_MERGE_ON_READ_BEHAVIOR = 'DISABLED' (copy-on-write) for compatibility.

    What v3 actually changed, and why in-place upgrade isn’t a thing

    The target architecture: a Bronze/Silver/Gold lakehouse where Dynamic Iceberg tables handle incremental transforms and write open Iceberg that external engines can read. The traps live in the arrows, not the boxes.

    Iceberg v3 is a genuine step change, not a point release. It adds deletion vectors (up to ~10x faster DML by avoiding positional-delete merges at read time), row lineage for native change data capture, a VARIANT type for semi-structured data with structured-query performance, default column values, geometry/geography types, nanosecond timestamps, and multi-argument partition transforms. On Snowflake it went to preview in March 2026 and GA on May 7, 2026.

    Here’s the first thing that trips migrations: you can’t upgrade an Iceberg table from v2 to v3. There is no ALTER TABLE ... SET ICEBERG_VERSION = 3 that rewrites your existing table in place. You configure the default Iceberg version and create new v3 tables, migrating data into them. This matters because teams plan the migration as a flag flip and discover it’s a recreate-and-backfill — which, for a large partitioned table, is a real project with a real compute bill, not a maintenance-window toggle. The related gotcha: v2 tables using copy-on-write represent an updated or relocated row in a standard stream as a DELETE followed by an INSERT for the same row, so any CDC logic you built on v2 stream semantics needs re-validation against v3’s row lineage before you cut over.

    The partitioned-write trap: HIERARCHICAL paths and the metadata ceiling

    Dynamic Iceberg tables expose three storage-shaping properties: PARTITION_BY, TARGET_FILE_SIZE, and PATH_LAYOUT. The one that surprises people is PATH_LAYOUT. It defaults to FLAT, meaning all Parquet data files land directly under the data/ directory. Set it to HIERARCHICAL and Snowflake writes Hive-style partitioned paths — but only in combination with PARTITION_BY. Setting HIERARCHICAL without a partition spec does nothing useful; the two are a pair.

    A minimal partitioned dynamic Iceberg table looks like this:

    CREATE DYNAMIC ICEBERG TABLE my_dt (
      product_id NUMBER, product_name STRING, order_time TIMESTAMP_NTZ
    )
      TARGET_LAG = '20 minutes'
      WAREHOUSE = my_wh
      EXTERNAL_VOLUME = 'my_vol'
      CATALOG = 'SNOWFLAKE'
      BASE_LOCATION = 'my_dt'
      PARTITION BY (YEAR(order_time))
      PATH_LAYOUT = HIERARCHICAL
      AS SELECT product_id, product_name, order_time FROM staging;

    The trap isn’t the syntax; it’s the partition cardinality. Iceberg’s metadata tracks files per partition, and every partition you create adds manifest overhead. Snowflake’s own guidance is blunt: avoid creating more than a few thousand partitions, and test query performance against your actual workload before finalizing a partitioning strategy. The failure mode when you ignore this is quietly brutal — partition by DAY(event_time) on a table with a few years of history and a high-cardinality secondary key, and you can generate tens of thousands of tiny partitions, each with its own small files. Now your Dynamic Table refresh spends its time in metadata planning rather than moving data, and your “low-latency” pipeline has a latency floor set by manifest bookkeeping.

    The design rule that keeps you out of trouble: partition on the coarsest grain that still prunes your dominant query pattern (usually a month or a broad category), let TARGET_FILE_SIZE and Snowflake’s file management handle within-partition layout, and reach for clustering rather than finer partitions when you need more selective pruning. Hierarchical paths are for interoperability and human-navigable storage, not a license to over-partition.

    The cross-engine write trap: “interoperable” is asymmetric

    This is the one that derails architecture diagrams. The interoperability story — external engines like Spark and Trino reading your Iceberg data — is real, but it runs in one direction for Snowflake-managed tables. Snowflake-managed Iceberg tables are read-write for Snowflake and read-only for external engines. As of the v3 GA, reading Snowflake-managed v3 tables from an external engine via the Horizon Iceberg REST Catalog API is generally available; writing from external engines to Snowflake-managed v3 tables through Horizon is explicitly not supported yet.

    If your architecture needs an external engine to write Iceberg that Snowflake then transforms, you must use externally-managed tables — data written by Spark into a catalog like AWS Glue, which Snowflake reads via a catalog integration and a linked database. That’s a supported and powerful pattern (it’s the canonical Bronze layer of an open lakehouse), but it’s a different architecture with different semantics than “Snowflake-managed tables that everyone can write to,” which does not exist today. Decide early which engine owns writes for each table, because that choice dictates managed-vs-external, and switching later means a migration. A further sharp edge: you can’t write with vended credentials to cloned or converted tables, and you can’t write at all to a table that was converted from externally-managed to Snowflake-managed — conversions are one-way for write access.

    The change-tracking trap: file-level vs row-level

    The granularity of change tracking decides how much work an incremental refresh does. Row-level (native) processes a tight delta; file-level (external Iceberg) can reprocess an entire file because one row moved.

    Dynamic Tables get their speed from incremental refresh — processing only what changed since the last refresh. The catch that native-table migrators don’t see coming: Dynamic Tables track changes at the file level for externally-managed Iceberg base tables, whereas they track at the row level for native Snowflake tables. That single difference reshapes your performance profile.

    With a native base table, if one row in a micro-partition changes, Snowflake knows it was that row, and the incremental refresh processes a tight delta. With an externally-managed Iceberg base table, change tracking is file-granular: a copy-on-write update that rewrites a data file marks the entire file as changed, so the refresh reprocesses every row in it, even if one row moved. On a table with frequent small updates and copy-on-write behavior, this inflates the change set dramatically and can make an “incremental” refresh behave like it’s doing far more work than the actual data change justifies. Snowflake’s documentation states it plainly: frequent copy-on-write operations on externally-managed Iceberg tables may impact incremental-refresh performance.

    Then there’s the metadata reset. INSERT OVERWRITE on a base table — a common pattern for batch reloads — resets change-tracking metadata, and the next Dynamic Table refresh falls back to a full recomputation. If your ingestion rewrites tables wholesale, your downstream “incremental” pipeline isn’t incremental at all.

    How v3 features rescue the change-tracking story

    The good news is that v3 exists partly to solve this, and using its features deliberately is the difference between a fast lakehouse and a slow one.

    Row lineage is the headline. In v3, tables track _row_id (a stable unique identifier assigned to each row) and _last_updated_sequence_number (the commit that last touched the row). This lets any compliant engine reliably match the same row across snapshots and detect row-level changes — native CDC in the format itself, not bolted on. Row lineage is supported for both Snowflake-managed and externally-managed v3 tables and underpins append-only and standard streams on Snowflake-managed v3 tables.

    Primary keys with RELY are the pragmatic rescue for the INSERT OVERWRITE problem. If you declare a reliable primary key on the base table, Snowflake compares rows by key value instead of leaning on change-tracking columns — so even when a table is fully rewritten, it computes the minimal set of actual changes rather than reprocessing everything. This is also how you enable incremental refresh downstream of a full-refresh dynamic table, by giving Snowflake a stable identity to diff against. For append-only CDC, the QUALIFY ROW_NUMBER() OVER (PARTITION BY id ORDER BY updated_at DESC) = 1 pattern gives you latest-row-per-key with a derived unique key, handling out-of-order arrival without extra logic.

    Deletion vectors replace v2’s positional deletes for merge-on-read, and they’re governed by heuristics worth knowing: Snowflake writes a deletion vector only if fewer than ~5% of a data file’s rows are deleted and the file exceeds ~1.6 MB; otherwise it rewrites the file (copy-on-write). You control the behavior with ICEBERG_MERGE_ON_READ_BEHAVIOR. The compatibility trap: if an external engine in your stack doesn’t yet understand v3 deletion vectors, you must set that parameter to 'DISABLED' to force copy-on-write, or the external engine will misread the table. Interoperability constrains you to the capabilities of the least capable engine that touches the table.

    The gotchas nobody warns you about

    Change tracking must be on, with non-zero Time Travel, on every underlying object. Incremental refresh silently depends on it. Snowflake will try to enable it automatically for incremental dynamic tables, but if you recreate a base object you must re-enable it — and a base object with Time Travel set to zero quietly breaks incrementality.

    The GRANT syntax has a trap for dynamic Iceberg tables. To grant access to future dynamic Iceberg tables in a schema, you use GRANT … ON FUTURE ICEBERG TABLES without the DYNAMIC keyword. The intuitive ON FUTURE DYNAMIC ICEBERG TABLES does not cover them, so a reasonable-looking grant leaves new tables inaccessible.

    Gen2 warehouses matter more than you’d expect. Snowflake’s Dynamic Table performance work — measured up to ~2.8x faster refresh over the past year — is specifically tied to Gen2 warehouses for patterns like top-level aggregates, QUALIFY row/rank = 1, cluster-by, and joins. If your incremental pipeline is on Gen1, you’re leaving a large multiple of refresh speed on the table before any Iceberg tuning.

    Cross-region and cross-cloud tables bill for transfer. A Snowflake-managed Iceberg table whose external volume sits in a different region or cloud than your account incurs cross-region data-transfer charges under the DATA_LAKE transfer type. Keep external volumes in the same region as your account unless you have a deliberate DR reason not to.

    A migration order that avoids the traps

    Sequence matters. First, decide per table who owns writes — if an external engine writes, it’s externally-managed; if only Snowflake writes, Snowflake-managed — because that’s the irreversible-ish decision. Second, set your default Iceberg version to v3 and plan recreate-and-backfill for existing v2 tables rather than expecting an upgrade. Third, choose a coarse partition grain (validated against real query patterns, staying well under a few thousand partitions) and use clustering for finer pruning. Fourth, make change tracking deliberate: declare reliable primary keys where base tables get rewritten, lean on row lineage for CDC, and confirm change tracking plus non-zero Time Travel on every base object. Fifth, pin ICEBERG_MERGE_ON_READ_BEHAVIOR to match the least-capable engine that reads the table. Then move workloads to Gen2 warehouses and measure incremental-refresh times against your latency target before you call it done.

    The one principle

    “Open Iceberg lakehouse with low-latency Dynamic Tables” is true only when the write path, the partition cardinality, and the change-tracking granularity all line up — and by default they don’t. Migrating native tables to dynamic Iceberg v3 is a design exercise, not a format swap: decide who writes, partition coarsely, give Snowflake a stable row identity to diff against, and constrain merge-on-read to your least-capable engine. Get those four right and the lakehouse is genuinely fast and open. Get them wrong and you’ve built a slow data lake with extra steps, one full refresh at a time.

    Related reading: Create dynamic Apache Iceberg tables (official docs) · Manage Iceberg tables: row lineage & deletion vectors · Snowflake Iceberg v3: When to Migrate · dbt State on Snowflake: Skip Unchanged Models · Dynamic Airflow DAGs via Snowflake Metadata

  • Governing the AI Agent: Securing Snowflake CoCo and MCP Workflows in Production

    Governing the AI Agent: Securing Snowflake CoCo and MCP Workflows in Production

    In March 2026, two days after Snowflake shipped Cortex Code, security researchers at PromptArmor published something that should have changed how every data team thinks about AI agents. They didn’t break Snowflake’s authentication. They didn’t steal a password. They fed the agent a piece of poisoned content — an indirect prompt injection — and the agent, reasoning helpfully as designed, used its own cached Snowflake credentials to exfiltrate data and drop tables. The attacker never logged in. The agent did the damage, under its own legitimate identity, because someone told it to and nothing stopped it.

    That’s the whole problem with the agentic enterprise in one incident. We spent a decade getting good at controlling what people can do in Snowflake — roles, grants, masking, row access policies. Then we handed an autonomous agent the keys and discovered our governance model couldn’t tell the difference between a human running a query and an agent running the same query on someone’s behalf. CoCo isn’t just writing SQL anymore; it’s orchestrating pipelines, calling external tools over MCP, and taking actions across your stack. The question is no longer “can it do useful work” — it obviously can — but “what stops it from doing damage, and who approves the actions that matter.”

    This is the practitioner’s guide to governing agentic workflows in production: the identity model, data-movement controls, and multi-party approvals that let you use CoCo and MCP without turning every agent into an unaudited superuser.

    TL;DR

    → An agent runs with the privileges of the role that invoked it. If that role has broad SELECT across production, the agent has the same reach — and it evaluates whether a query is answerable, not whether it’s appropriate. Over-scoped roles are the root risk.

    → The injection surface is wide (READMEs, web content, table data, MCP tool responses) and can’t be eliminated. Governance shifts from “prevent the injection” to “limit the blast radius when one lands.” Scope first, then monitor.

    → AI Agent Identity (GA at Summit 2026) gives each agent a cryptographic identity, per-agent RBAC, and a full audit trail — so policies can treat agent traffic differently from human traffic and you can actually attribute actions.

    → Data Movement Policies restrict where data can flow and which channels agents can use — the control that stops an over-scoped agent from piping regulated data out through an MCP integration.

    → Multi-party approval (private preview) puts a human (or two) in the loop for destructive or high-sensitivity actions — the agentic equivalent of a code review before a DROP.

    → The MCP Gateway (built on Snowflake’s Natoma acquisition) centralizes and governs every MCP connection — identity-aware authorization and audit at the tool-call level, instead of each agent wiring its own servers ungoverned.

    → Start today: enable the free built-in prompt-injection guardrails, audit which agents touch sensitive data, write an explicit agent policy (what each agent may and may not do), and apply data-movement policies to your most sensitive tables.

    The core problem: an agent inherits your blast radius

    Everything else follows from one fact, so internalize it first. A Cortex Agent — and CoCo is one — runs under the privileges of the Snowflake user or role that invoked it. It has exactly the access that role has. Not less, because it isn’t sandboxed away from the role’s grants by default; not more, because Snowflake’s perimeter still applies. Whatever the invoking role can SELECT, the agent can SELECT.

    In a world of humans, over-scoped roles are a latent risk — a person could query the HR schema they never actually touch, but they don’t, because they know not to. An agent has no such judgment. It does not weigh whether querying the compensation table is appropriate to the task; it weighs whether the query is answerable given the permissions available. Ask an agent configured for product analytics a question that happens to be answerable using financial data its role can reach, and it will answer. There is no internal boundary that says “that’s not my department.”

    So the blast radius of every over-scoped role expands dramatically the moment that role underpins an always-on agent. The single most important governance move you can make is not a new feature — it’s scoping the agent’s role down to exactly the data its job requires, and no more. Every control below is a layer on top of that foundation. If the foundation is a role with broad production access, no amount of monitoring saves you.

    Same agent, same injection — the only difference is how tightly the underlying role is scoped. Least privilege is what makes a successful injection cheap.

    Why you defend the blast radius, not the perimeter

    The PromptArmor attack teaches the strategy. The injection didn’t target authentication; it targeted the agent’s reasoning and then rode its existing credentials. The injection surface — anything the agent reads and treats as context — is enormous: source files and READMEs, web pages it fetches, rows in tables it queries, and crucially the responses that come back from MCP tools. You cannot realistically eliminate that surface. A determined attacker will eventually land an injection.

    That reframes the whole job. If you can’t stop every injection, you make a successful one cheap. Controls that limit what the agent can access limit what an attacker can do through it. Scope first, then monitor. Every governance control that follows exists to shrink the blast radius of an injection that gets through — not to pretend none ever will.

    Agent identity: making the agent a first-class, distinct actor

    The reason our old governance couldn’t cope is that agent traffic looked exactly like human traffic. If an agent runs under a shared service account, you cannot tell in the audit log whether “the agent” or “a person using the agent’s role” ran a query, and you cannot apply different rules to the two. AI Agent Identity, which went GA at Summit 2026, fixes this at the platform level: every agent gets a cryptographic identity, per-agent RBAC, and a complete audit trail.

    Concretely, that buys three things. First, attribution — the audit log records that this specific agent, not a nebulous service account, took this action, so incident response has something to trace. Second, differential policy — because Snowflake can recognize when an action occurs in an agent’s context, security teams can apply custom masking or visibility rules to agent traffic specifically, tightening or loosening access for agents independently of the humans behind them. Third, lifecycle — a distinct identity can be reviewed and decommissioned when a project ends, which is the antidote to the classic failure mode where a “the agent” service account silently accumulates privileges for years and is never cleaned up.

    The practical instruction: never run production agents under a shared or personal role. Give each agent its own identity, grant it a purpose-built role scoped to its task, and treat that identity as something you review on a schedule — the same way you’d review a human’s access, because now it’s a non-human actor with real reach.

    Data movement policies: stopping the exfiltration path

    Here’s the MCP-specific risk that most governance frameworks haven’t caught up to. When you connect an agent’s output to external systems over MCP, data reachable by the agent becomes potentially reachable outside the Snowflake perimeter. If the agent’s role can read regulated or confidential data, that data can flow outward through an MCP integration that isn’t subject to the same controls as the warehouse. Your carefully governed table is one tool-call away from a Slack channel or a third-party API.

    Data Movement Policies are the control for exactly this. They let you restrict where data can go and which channels agents are allowed to use, applied at the level of your most sensitive tables. The pattern that works: identify your regulated and confidential datasets, and attach movement policies that restrict agentic access channels — so even if an agent’s role can technically read a table, the policy prevents that data from being moved out through an ungoverned path. This is the difference between “the agent can see it” and “the agent can send it somewhere,” and for regulated data those are very different permissions.

    Pair this with the principle that the model runs where the data lives. Snowflake’s model-in-platform approach (running Claude and other models natively inside Cortex) means sensitive data doesn’t have to leave the perimeter for the agent to reason over it. Movement policies then govern the exceptions — the deliberate, approved paths where data does flow outward — rather than leaving every MCP connection as an open door.

    Multi-party approval: a human gate on destructive actions

    Not every action needs a human. The pattern is to auto-execute low-risk reads and route destructive or high-sensitivity actions to an approval gate — a code-review step before an agent does something irreversible.

    Some actions are too consequential to let an agent take unilaterally, no matter how well-scoped. Dropping a table, moving regulated data, granting privileges, deploying to production — these are the agentic equivalent of a force-push to main. Multi-party approval (in private preview as of Summit 2026) is the control: it requires human sign-off before an agent executes designated high-risk actions, and for the most sensitive it can require two approvers.

    The design pattern that keeps this usable is triage. If you gate everything, people rubber-stamp approvals and the control becomes theater. Instead, classify actions by risk. Read-only work — a SELECT, a Cortex Search, generating a chart — executes automatically; that’s the whole point of an agent. Destructive or high-sensitivity actions divert to an approval gate before they run, and are blocked and logged if no one approves. The engineering task is deciding, up front, which actions in your environment belong on the destructive list — and it’s worth doing that exercise now, before you turn agents loose, rather than after an incident.

    The MCP Gateway: governing the actions, not just the data

    Early MCP adoption looks like every agent configuring its own servers independently — one team wires up a Jira server, another points at an internal API, and nobody has a single view of what’s connected or what those connections can do. That’s ungoverned by construction. Snowflake’s acquisition of Natoma exists to fix this: the MCP Gateway is a centralized, governed layer for every MCP connection in the organization.

    What the Gateway changes is where enforcement happens. Instead of trusting each agent to behave, every tool call — sending an email, opening a ticket, hitting an API — flows through a gateway that enforces identity verification, access policies, and audit controls at the level of the individual action. This extends governance from the data an agent reads to the actions it takes. Central management means one place to see and control all external tool connections; gateway-level access control means an agent can only invoke the tools its policy permits; and tool-call-level audit means you can reconstruct exactly what an agent did across systems, not just within Snowflake.

    The mental shift: data governance and action governance are different problems. Row access policies protect what the agent can read. The MCP Gateway protects what the agent can do with tools once it has read something. In the agentic enterprise you need both, because an agent that can only read sensitive data is a smaller problem than one that can read it and send it somewhere.

    The gotchas nobody warns you about

    Service accounts are where governance goes to die. The most common real-world failure isn’t a clever attack — it’s an agent running under a service account created for “the agent,” never scoped tightly, never reviewed, accumulating privileges as teams bolt on data sources. Tie every agent to a defined access scope and a lifecycle policy, and review non-human identities on the same cadence as human ones.

    MCP tool responses are part of the injection surface. It’s easy to think of prompt injection as coming from user input, but a compromised or malicious MCP server can return a response crafted to hijack the agent’s reasoning. Treat data coming back from external tools with the same suspicion as any other untrusted input — which is another argument for routing MCP through a governed gateway rather than trusting arbitrary servers.

    Built-in guardrails are necessary, not sufficient. Snowflake’s baseline prompt-injection protection is free, automatic, and worth enabling immediately — it blocks known attack patterns. But it’s a pattern database, so it lags novel attacks by definition. Guardrails reduce the frequency of successful injections; scope and movement policies reduce the impact. You need both, and you should never let “guardrails are on” substitute for scoping.

    Cross-organization collaboration multiplies the identity problem. As agentic workflows span companies (clean rooms, shared data, partner integrations), privacy-preserving controls and role separation stop being nice-to-haves and become engineering requirements. An agent acting across an organizational boundary needs an identity and policy that make sense on both sides.

    A starting checklist for production

    If you’re deploying agents now, the order of operations matters. Enable the built-in AI guardrails first — they’re free and automatic. Audit which agents (and which underlying roles) can reach sensitive data, and scope those roles down to least privilege; this is the highest-leverage step and it’s just RBAC discipline. Give each production agent its own identity rather than a shared service account. Write an explicit agent policy — for each agent, what it may do and what’s off-limits — and apply data movement policies to your most sensitive tables, restricting agentic channels specifically. Then identify the destructive actions in your environment that warrant two-person confirmation and get ahead of the multi-party approval rollout. Finally, route MCP connections through a governed gateway so action-level governance and audit exist from day one rather than being retrofitted after an incident.

    The one principle

    An AI agent is a non-human actor that inherits a role’s full reach and exercises none of a human’s judgment, so govern it as an identity, not as a feature. Scope its role to least privilege, give it a distinct auditable identity, restrict where its data can move, gate its destructive actions behind human approval, and route its tool calls through a governed MCP gateway. You will not prevent every prompt injection. What you can decide, in advance, is how little damage a successful one is able to do — and in the agentic enterprise, that decision is the whole game.

    Related reading: Cortex Agents governance & access control (official docs) · Snowflake Managed MCP Servers: secure, governed data agents · How to Use MCP in Snowflake CoCo Desktop · Debugging Zero-Copy Clone Storage Costs in CI/CD · Dynamic Airflow DAGs via Snowflake Metadata

  • Dynamic Airflow DAGs via Snowflake Metadata: Eliminating Hardcoded Pipeline Tasks

    Dynamic Airflow DAGs via Snowflake Metadata: Eliminating Hardcoded Pipeline Tasks

    I once inherited an Airflow repo with 214 DAG files that were, functionally, the same DAG. Each one extracted a table from a source system, loaded it into Snowflake, and ran a transform. The only differences between them were the table name, the schedule, and which SQL file to run. Someone had copy-pasted the template 214 times, and every schema change meant a find-and-replace across 214 files and a prayer that nothing got missed. Onboarding a new table meant copy-pasting a 215th.

    That repo is the case against hardcoded pipeline tasks in a single painful sentence: if the only thing that changes between your DAGs is data, then your DAGs should be generated from data. The fix is to move the pipeline definitions out of Python files and into a Snowflake metadata table, then generate the DAGs from that table. Add a row, get a pipeline. Change a row, change a pipeline. No copy-paste, no 214-file find-and-replace.

    This is the guide to doing that properly — including the distinction that trips most people up (there are two completely different “dynamic” features in Airflow and they solve different problems), the metadata-driven generation pattern, and the parsing gotchas that will wreck your scheduler if you get them wrong.

    TL;DR

    → Two different features share the word “dynamic.” Dynamic DAG generation builds DAG structure at parse time from config/metadata — the task count is fixed for a given run. Dynamic task mapping (.expand()) creates N task instances at runtime from an upstream task’s output. They solve different problems; you’ll often use both.

    → The metadata-driven pattern: store pipeline definitions (DAG name, tasks, schedule, SQL file, parent/child dependencies) in a Snowflake table → feed each row into a Jinja template → render a dag.py file per pipeline. Add a row, get a DAG.

    → Add operational columns to the metadata table — created_at and last_updated_at — so you can track which pipelines exist and trigger regeneration when a definition changes.

    → Use environment variables, not Airflow Variables, in top-level DAG code. Airflow Variables hit the metadata DB on every parse and will slow your scheduler to a crawl.

    → Generate tasks in a stable, sorted order every time (ORDER BY in your query or sorted() in Python), or the Grid View reshuffles tasks on every refresh and your history becomes unreadable.

    → For large numbers of generated DAGs, use get_parsing_context() to skip building DAG objects you don’t need during task execution — one documented case cut parsing from 120s to 200ms.

    → Use dynamic task mapping when the count is unknown until runtime (e.g. “process however many files landed today”). Note trigger_rule=ALWAYS is not allowed on task-generated mapped tasks.

    The distinction that trips everyone up

    Before any code, get this straight, because conflating the two is the single most common source of confusion I see. Airflow has two features with “dynamic” in the name and they are not interchangeable.

    Dynamic DAG generation is about producing DAG files or objects programmatically. Instead of hand-writing 214 near-identical DAGs, you write one generator that reads definitions from somewhere (a config file, a metadata table) and emits the DAGs. The important property: the structure is decided at parse time, when Airflow loads the DAG file. For a given DAG run, the number of tasks is fixed. This is what you want when you have many similar pipelines that differ only by parameters.

    Dynamic task mapping (introduced in Airflow 2.3, via .expand() and .map()) is about creating task instances at runtime. A task returns a list, and Airflow creates one copy of a downstream task per element — and it doesn’t know how many until the upstream task actually runs. This is the MapReduce model: the scheduler creates N copies of the mapped task right before execution. This is what you want when the count is genuinely unknown until runtime — “process each file that landed in S3 today,” where “today” might be 3 files or 300.

    The rule of thumb: if you know the shape of the work when the DAG is parsed, use dynamic DAG generation. If the shape depends on data that only exists at runtime, use dynamic task mapping. A mature setup often uses both — generated DAGs whose internal tasks map over runtime data.

    Left: fixed structure known at parse time. Right: task instances fanned out at runtime. Same word, opposite problems.

    The metadata-driven pattern

    The pipeline that builds pipelines: a Snowflake metadata table feeds a Jinja template that renders one dag.py per row, which Airflow then parses like any other DAG.

    The architecture has four moving parts. First, a metadata table in Snowflake that holds pipeline definitions. At minimum it stores, per task: the DAG name it belongs to, the task name, the schedule, what the task runs (say, a SQL file path), and the parent/child dependency links. A row-per-task layout with a parent_task column lets you express arbitrary dependency graphs — a task names its parent, and the generator wires the edges.

    Here’s a minimal shape:

    CREATE TABLE pipeline_metadata (
      dag_name      STRING,
      task_name     STRING,
      parent_task   STRING,  -- NULL for a root task
      schedule      STRING,  -- e.g. '0 2 * * *'
      sql_file      STRING,  -- what the task executes
      is_active     BOOLEAN,
      created_at    TIMESTAMP,
      last_updated_at TIMESTAMP
    );

    Second, a Jinja template — a .j2 file that looks like a DAG with placeholders where the metadata values go: the DAG id, the schedule, a loop that emits one operator per task, and the dependency wiring. Third, a generator that queries the metadata table, groups rows by dag_name, and renders the template once per DAG, writing out a dag.py file. Fourth, Airflow’s normal DAG File Processor, which parses those rendered files exactly as if you’d hand-written them.

    The payoff is the operational columns. Because each row carries created_at and last_updated_at, you can tell when a pipeline was first defined and when it last changed. When someone edits a definition, last_updated_at moves, and you can trigger regeneration for just the affected DAGs rather than rebuilding everything. Onboarding a new pipeline is now an INSERT, not a new file.

    Rendering the DAG from a row

    The generator itself is short. Conceptually: query the active metadata, group by DAG, and for each group render the template with that group’s tasks and dependencies. A sketch:

    from jinja2 import Environment, FileSystemLoader
    import os
    
    # env var, NOT an Airflow Variable — see the parsing note below
    env = os.environ.get("DEPLOYMENT", "PROD")
    
    rows = run_query("""
      SELECT dag_name, task_name, parent_task, schedule, sql_file
      FROM pipeline_metadata
      WHERE is_active = TRUE
      ORDER BY dag_name, task_name  -- stable order, always
    """)
    
    template = Environment(loader=FileSystemLoader("templates")) \
        .get_template("dag_template.j2")
    
    for dag_name, tasks in group_by_dag(rows):
      rendered = template.render(dag_name=dag_name, tasks=tasks, env=env)
      with open(f"dags/{dag_name}.py", "w") as f:
        f.write(rendered)

    Notice the ORDER BY. That is not cosmetic — it’s load-bearing, and the next section explains why.

    The parsing gotchas that wreck schedulers

    Three parse-time mistakes and their fixes. Every one of these is invisible until your scheduler is under load, then very visible.

    Dynamic generation runs at parse time, and the DAG File Processor parses your files constantly. Anything expensive or unstable in that path multiplies across every parse. Three specific mistakes:

    Airflow Variables in top-level code. It’s tempting to configure your generator with Variable.get("something"). Don’t, not at the top level. Every Airflow Variable read in top-level code opens a connection to the metadata database, and top-level code runs on every parse. At scale this hammers your metadata DB and drags parsing. Use environment variables (os.environ.get(...)) for anything read during generation — they’re free to read and don’t touch the DB.

    Unstable task ordering. If your generator emits tasks in a different order on different parses — because the query has no ORDER BY, or you iterated a Python set — Airflow’s Grid View reshuffles the task rows every time it refreshes. Your run history becomes impossible to read, and it looks like the DAG is changing when it isn’t. Always impose a stable order: ORDER BY in the query, or sorted() in Python. Deterministic generation is not optional.

    Parsing every DAG on every task execution. The DAG File Processor loads the whole file to get metadata, but executing a single task only needs that one DAG object. If your generator builds hundreds of DAGs in one file, every task execution pays to construct all of them. The fix is get_parsing_context(): check which DAG is actually being parsed and skip generating the rest. The documented “Magic Loop” example cut parsing from 120 seconds to 200 milliseconds this way. It’s most valuable when the generated-DAG count is high — use it with care and test it, since it doesn’t apply if later DAGs depend on earlier ones.

    When to reach for dynamic task mapping instead

    Everything above generates structure from metadata known at parse time. But some workloads only reveal their shape at runtime, and that’s dynamic task mapping’s job. The canonical example is file processing: an unknown number of files land in cloud storage each day, and you want one task instance per file loaded into Snowflake.

    The pattern is a task that returns the list, and a downstream task that expands over it:

    @task
    def list_new_files():
      return get_s3_keys(prefix=f"{{{{ ds_nodash }}}}/")  # however many landed
    
    @task
    def load_to_snowflake(key):
      copy_into_snowflake(key)
    
    load_to_snowflake.expand(key=list_new_files())

    The scheduler creates one load_to_snowflake instance per key, right before execution, and the Grid View shows the mapped count in brackets. You can also map over task groups with the @task_group decorator and .expand() when each unit of work is several steps, using the map_indexes parameter to pull the right XCom per instance. One constraint to remember: trigger_rule=TriggerRule.ALWAYS is not allowed on a task-generated mapped task, because the expanded parameters are undefined at the moment of immediate execution — Airflow raises an error at parse time if you try.

    Cost and maintenance math

    The win here isn’t compute cost, it’s maintenance cost, and it compounds. Go back to the 214-DAG repo. A schema change that touched every pipeline meant editing 214 files — call it a day of careful, error-prone work, plus review, plus the near-certainty of missing one. With metadata-driven generation, the same change is either one UPDATE to the metadata table or one edit to the shared Jinja template, followed by regeneration. Minutes, not a day, and uniform by construction — you cannot miss one, because there’s only one definition.

    Onboarding scales the same way. In the file-per-pipeline world, each new table is a new hand-authored file and a new opportunity for drift. In the metadata world it’s an INSERT. Ten new tables is ten rows. The marginal cost of a pipeline drops toward zero, which changes what’s worth automating — pipelines that weren’t worth hand-writing become trivially worth a row.

    The gotchas nobody warns you about

    Generation failure takes down everything at once. The flip side of one definition is one point of failure. A bug in the template or generator doesn’t break one DAG, it breaks all of them. Validate rendered output (even a quick python -c "compile(...)" check) before writing files, and keep the last-good rendered files so a bad generation doesn’t wipe working DAGs.

    Metadata and reality drift. The metadata table says what pipelines should exist; the dags/ folder holds what does. If someone edits a rendered file by hand, or a row is deleted without removing the file, the two diverge. Treat rendered files as build artifacts, never edit them directly, and have regeneration remove files for DAGs no longer in the metadata.

    Secrets don’t belong in the metadata table. It’s tempting to store connection details per pipeline. Keep credentials in Airflow Connections or a secrets backend and reference them by name from the metadata — the table should hold pipeline structure, not secrets.

    Too much magic hurts debuggability. A generated DAG is one level removed from the code you read. When something breaks at 2 a.m., the on-call engineer is debugging rendered output, not the template. Keep the template readable, keep the rendered files on disk (don’t generate purely in memory), and make it obvious which metadata row produced which DAG.

    The one principle

    If the only thing that changes between your pipelines is data, define them with data, not code. Put pipeline definitions in a Snowflake metadata table, render them through one Jinja template, and let Airflow parse the result — but keep generation deterministic, keep Airflow Variables out of top-level code, and remember that one definition means one point of failure worth guarding. The goal isn’t cleverness; it’s that onboarding the 215th pipeline should be an INSERT, and a schema change should touch one place, not two hundred.

    Related reading: Airflow: Dynamic DAG Generation (official docs) · Airflow: Dynamic Task Mapping (official docs) · Orchestrating dbt With Airflow on Snowflake · dbt State on Snowflake: Skip Unchanged Models · Snowflake Query Execution: What Really Happens

  • Debugging Zero-Copy Clone Storage Costs in CI/CD

    Debugging Zero-Copy Clone Storage Costs in CI/CD

    The Snowflake bill for our CI account had roughly tripled over a quarter, and nobody could point to why. We hadn’t loaded meaningfully more data. Compute was flat. But storage kept climbing, month over month, in an account whose entire job was to spin up throwaway test environments and tear them down. Throwaway. Torn down. And yet the storage line kept going up and to the right.

    The culprit was the feature I’d been recommending to everyone as “basically free”: zero-copy clones. Our CI pipeline cloned production on every pull request, ran migrations and tests against the clone, and dropped it at the end. Textbook. The problem is that “zero-copy” describes the moment of creation and nothing after it, and “drop” doesn’t mean what you think it means when clones are involved. We were paying for storage we believed we’d deleted weeks ago.

    This is the guide to why that happens, how to find it in your own account, and how to stop it. If you run clone-based CI/CD at any scale, some version of this is almost certainly happening to you right now.

    TL;DR

    → Zero-copy clones are free at creation — they share the source’s micro-partitions through metadata pointers. They are not free after anything writes. Every INSERT/UPDATE/DELETE on either side writes new micro-partitions that are billed.

    → In CI/CD the divergence is your migrations. Clone prod, run a schema migration or a backfill against the clone, and you’ve just created new micro-partitions that cost real storage — every pull request, every pipeline run.

    → Dropping the clone does not immediately free that storage. Dropped tables enter Time Travel, then Fail-safe (up to 1 day + 7 days on permanent tables) before the bytes are physically removed. Fast CI loops drop clones constantly and stack up a rolling backlog of retained bytes.

    → The nasty one: clone-group ownership transfer. Storage for shared micro-partitions is owned by the oldest table in the clone group. Drop the source and its still-shared partitions don’t vanish — ownership transfers to a surviving clone. You can delete “the original” and watch storage not move.

    → Diagnose with SNOWFLAKE.ACCOUNT_USAGE.TABLE_STORAGE_METRICS: ACTIVE_BYTES, TIME_TRAVEL_BYTES, FAILSAFE_BYTES, and the key one, RETAINED_FOR_CLONE_BYTES — bytes kept alive only because a clone still references them.

    → Fixes: clone with transient tables/databases for CI (no Fail-safe, minimal Time Travel), set Time Travel to 0 days on CI objects, actually DROP at pipeline end even on failure, and don’t run heavy migrations against the clone if a lighter check will do.

    Why “zero-copy” is a half-truth

    Snowflake stores table data in immutable micro-partitions — compressed columnar files, tens to hundreds of MB each. Once written, a micro-partition is never modified. When you clone a table, Snowflake doesn’t copy those files. It writes a new metadata entry pointing at the same set of micro-partitions. That’s why a clone of a 5 TB table is instant and costs nothing extra at that instant. It’s a hard link at the partition level, not a copy.

    The word “zero-copy” describes exactly that instant and no other. Because micro-partitions are immutable, the moment you change a row — in the clone or the original — Snowflake can’t edit the shared partition in place. It writes a new micro-partition containing the change, and that new partition is owned exclusively by whichever side made the change. The unchanged partitions stay shared. So your storage cost isn’t the size of the clone; it’s the size of the divergence between the clone and its source.

    The correct mental model, which took me an embarrassingly large bill to internalize: a clone is not a free copy, it’s an instant branch that gets more expensive as it diverges. Read-only clone of 1 TB? Costs nothing. Clone you fully rewrite? Costs a second 1 TB. Real CI workloads land in between — and “in between,” multiplied by every pull request, is a budget line.

    Where the cost actually enters in CI/CD

    The clone is free at step 1. Your migration at step 2 is what creates billed storage — and step 4’s drop doesn’t reclaim it right away.

    Here’s the standard CI pattern, the one in every tutorial:

    CREATE DATABASE ci_test_${BUILD_ID} CLONE production_db;
    -- run migrations against the clone
    -- run integration tests
    DROP DATABASE IF EXISTS ci_test_${BUILD_ID};

    Step one is genuinely free. The cost enters at “run migrations.” A migration that adds a column, backfills a value, rebuilds a table, or runs a dbt model against the clone writes new micro-partitions for every affected partition. If your migration touches 10% of a 500 GB table, you just materialized ~50 GB of new storage — for one CI run. Run that pipeline 40 times a day across a team and the daily divergence is measured in terabytes of writes, even though each individual run “only” changed a slice.

    None of that is visible while you’re looking at it, because the clone gets dropped at the end and the environment looks clean. Which brings us to the part that actually generates the surprise bill.

    The two things that keep paying after you “delete”

    1. Dropping a table doesn’t free its bytes immediately. When you DROP a permanent table (or database), it doesn’t evaporate — it goes into Time Travel for its retention period (default 1 day, and up to 90), and then into Fail-safe for a further 7 days, during which only Snowflake can recover it. Throughout both windows you’re billed for those bytes. A CI loop that creates and drops clones dozens of times a day is continuously feeding a rolling backlog: at any given moment you’re paying for the Time-Travel-and-Fail-safe tail of every clone dropped in roughly the last week, not just the ones alive right now.

    2. Clone-group ownership transfer — the one that breaks intuition. Every table in a clone group has an independent lifecycle, but the storage for shared micro-partitions is owned by the oldest table in the group. Here’s the trap: you decide the source table is the problem and drop it. You expect storage to fall. It doesn’t. Because a clone still references those shared partitions, Snowflake can’t release them — so when they’d otherwise exit Time Travel, ownership transfers to a surviving clone instead. You deleted the original and the bytes simply changed owner. This is why teams stare at a dropped production backup table and can’t understand why the account storage didn’t budge.

    Finding it in your own account

    Four columns tell the whole story. RETAINED_FOR_CLONE_BYTES is the one that reveals storage kept alive purely because a clone still references it.

    The view you want is SNOWFLAKE.ACCOUNT_USAGE.TABLE_STORAGE_METRICS. It breaks every table’s footprint into the buckets that matter:

    SELECT
      table_catalog,
      table_schema,
      table_name,
      active_bytes / POW(1024,3) AS active_gb,
      time_travel_bytes / POW(1024,3) AS time_travel_gb,
      failsafe_bytes / POW(1024,3) AS failsafe_gb,
      retained_for_clone_bytes / POW(1024,3) AS clone_retained_gb
    FROM snowflake.account_usage.table_storage_metrics
    WHERE retained_for_clone_bytes > 0
    ORDER BY retained_for_clone_bytes DESC;

    ACTIVE_BYTES is your live data — the part you expect to pay for. TIME_TRAVEL_BYTES and FAILSAFE_BYTES are the recovery tails. RETAINED_FOR_CLONE_BYTES is the smoking gun: bytes that are only still on disk because some clone in the group references them. If that column is large on tables you thought were long gone, you’ve found your leak.

    To hunt CI clones specifically, filter by naming convention and age. Because Snowflake records clone lineage, you can surface old clones still retaining significant storage:

    SELECT
      table_catalog,
      table_name,
      clone_group_id,
      retained_for_clone_bytes / POW(1024,3) AS clone_retained_gb,
      table_created
    FROM snowflake.account_usage.table_storage_metrics
    WHERE table_catalog ILIKE 'CI_TEST_%'
      AND retained_for_clone_bytes > 0
    ORDER BY clone_retained_gb DESC;

    One caveat worth knowing: ACCOUNT_USAGE views have latency (often a couple of hours), so don’t expect a drop you ran five minutes ago to show up instantly. Debug against yesterday’s picture, not this second’s.

    The cost math, concretely

    Say production is 2 TB and your CI migration reliably rewrites ~8% of it per run: ~160 GB of new micro-partitions per pipeline. The clone is dropped at the end, so those 160 GB immediately become Time Travel + Fail-safe bytes rather than active bytes — and they linger for the retention tail. With a 1-day Time Travel plus 7-day Fail-safe window on permanent objects, each run’s divergence sticks around for roughly 8 days before it’s physically purged.

    Run the pipeline 30 times a day and, in steady state, you’re carrying roughly 30 runs/day × 8 days × 160 GB ≈ 38 TB of retained bytes that you believe you deleted. At standard on-demand storage rates that’s a four-figure monthly line for data that exists only because “drop” isn’t “delete” and permanent tables carry a Fail-safe tail. The exact number depends on your migration’s write volume and your retention settings — but the shape is always the same, and it’s always bigger than teams expect.

    The fixes, in priority order

    Clone into transient objects for CI. This is the single biggest lever. Transient tables and databases have no Fail-safe period and a Time Travel retention of 0 or 1 day. Clone production into a transient database for CI, and when you drop it there’s no 7-day Fail-safe tail — the bytes are reclaimable almost immediately. CREATE TRANSIENT DATABASE ci_test_${BUILD_ID} CLONE production_db; Note the source’s own storage behavior is unchanged; this only governs the CI-side lifecycle, which is exactly the part generating your backlog.

    Set Time Travel to zero on CI objects. If you can’t use transient objects for some reason, at least set DATA_RETENTION_TIME_IN_DAYS = 0 on the CI database so dropped clones don’t linger in Time Travel. Combined with the above, your CI divergence becomes genuinely short-lived.

    Actually drop, even on failure. The tutorial pattern drops the clone at the end — but if tests fail and the pipeline exits early, the DROP may never run. Orphaned clones from failed builds are a classic source of retained storage. Put the DROP in a finally/always block so it runs regardless of test outcome, and add a scheduled sweeper task that drops any CI_TEST_% database older than a few hours as a backstop.

    Diverge less. Ask whether your CI actually needs to rewrite 8% of a 2 TB table. Often the migration under test only needs to run against a representative subset, or the test only needs schema validation, not a full data backfill. Cloning gives you production-realistic structure for free; you don’t always need to exercise it against production-scale writes.

    Mind the ownership trap when cleaning up. If you’re deleting old backup clones to reclaim space, remember that dropping the oldest member of a clone group transfers ownership rather than freeing bytes. To actually reclaim storage from a clone group, you generally need to drop all members that reference the shared partitions and let the retention windows expire. Deleting one and expecting the bill to fall is how the confusion starts.

    The gotchas nobody warns you about

    Grants diverge at clone time. A clone inherits grants and masking policies from the source at the instant of cloning, then becomes independent. For CI this is usually fine, but if your pipeline relies on grants applied to production after the clone was taken, they won’t be there.

    Small-file defragmentation writes Time Travel bytes too. Even plain INSERT/COPY/Snowpipe loads can generate Time Travel and Fail-safe bytes, because Snowflake periodically compacts small micro-partitions — deleting the small ones (which enter the recovery tail) and writing a consolidated one. So retained bytes aren’t exclusively a clone phenomenon; clones just amplify it.

    External tables, stages, and pipes don’t clone. If your CI environment depends on them, cloning the database won’t bring them along — you’ll need to recreate them in the clone.

    ACCOUNT_USAGE latency hides fast loops. Because the storage views lag by up to a few hours, a tight CI loop can be generating and “hiding” retained storage faster than your dashboards refresh. Trust the trend over days, not the instantaneous number.

    The one principle

    Zero-copy cloning is free to create and expensive to diverge — and “drop” is not “delete.” In CI/CD, the storage you pay for is the write volume of your migrations times the retention tail of your dropped clones. Clone into transient objects, keep Time Travel short, drop reliably, and diverge only as much as the test actually requires. The feature isn’t lying to you; it’s just describing creation, not the whole lifecycle. Manage the lifecycle and the “free” clone stays close to free.

    Related reading: Snowflake data storage considerations (clone groups & CDP) · TABLE_STORAGE_METRICS view reference · dbt State on Snowflake: Skip Unchanged Models · Snowflake Query Execution: What Really Happens · Snowflake Iceberg v3: When to Migrate

  • How to Use MCP in Snowflake CoCo Desktop

    How to Use MCP in Snowflake CoCo Desktop

    The first thing I tried to do in CoCo Desktop was ask it to pull the open tickets for a data pipeline I was debugging. It couldn’t. Not because it wasn’t smart enough — it’s genuinely good at reasoning over your Snowflake schemas — but because CoCo’s context ends where Snowflake’s context ends. It knew everything about my tables, my RBAC, my lineage. It knew nothing about my Jira board sitting one browser tab away.

    That gap is exactly what MCP closes. Once I wired up a couple of MCP servers, CoCo went from “excellent inside Snowflake’s walls” to “reaches into the rest of my stack” — Jira, GitHub, internal APIs — without me writing a line of integration code. This is the practical guide to doing that: the setup flow, where the config actually lives, how credentials are handled, and the operational limits that will trip you up on day one if nobody warns you.

    A quick naming note before we start, because it confused me too: CoCo is the new name for Cortex Code. Snowflake renamed it at Summit 2026. Same product, same architecture — you’ll still see “cortex” all over the file paths and environment variables, which is why this guide uses both names where the paths demand it.

    TL;DR

    → MCP (Model Context Protocol) is an open standard that connects CoCo Desktop to external tools — GitHub, Jira, internal APIs, databases — without per-tool integration code. Add a server once and its tools appear to the agent automatically.

    → Setup is fast: Agent Settings → MCP tab → + New → pick a scope (Global or Workspace) → pick a transport (Command/stdio or Remote/HTTP) → fill in details → Save. The server starts immediately, no restart.

    → Two transport types: Command (stdio) runs a local process (e.g. uvx mcp-server-git); Remote (HTTP) connects to a URL (e.g. a hosted server with auth headers).

    → Config lives in JSON: global at ~/.snowflake/cortex/mcp.json (all workspaces), workspace at <workspace>/.snowflake/cortex/mcp.json (that project only). Top-level key is "mcpServers".

    → Credentials are handled for you: on first connection CoCo migrates secrets (env vars, headers, OAuth tokens) out of mcp.json and into your OS keychain, then strips them from the file. Never hardcode tokens.

    → The limits that bite: tool output is capped at 50 KB (design servers to return summaries, not raw dumps), default tool timeout is 60 seconds (override with COCO_MCP_TOOL_TIMEOUT_MS), and tool names must be alphanumeric/underscore/hyphen and under 64 characters or the server is rejected outright.

    → If you already run MCP servers for Claude Desktop, Cursor, or Windsurf, CoCo Desktop can often reuse them — MCP is a standard, not a Snowflake-specific connector.

    What MCP actually does for CoCo

    CoCo is a data-native coding agent. Its whole advantage is that it understands your Snowflake environment — live schemas, access controls, lineage — so it generates SQL and dbt code that actually works against your real objects within your permissions. That’s also its boundary. The moment you need context from outside Snowflake, CoCo is blind to it.

    MCP is the bridge. It’s an open protocol (the same one Claude Desktop, Cursor, and Windsurf use) that lets an agent call tools exposed by external “servers.” A GitHub MCP server exposes tools like “search code” and “list pull requests.” A Jira server exposes “find issues” and “create ticket.” Once you register that server with CoCo, those tools become part of the agent’s toolbox automatically — no code changes, no custom connector. You ask CoCo “what are the open bugs on the ingestion pipeline?” and it calls the Jira tool, reads the result, and reasons over it alongside your Snowflake context.

    The mental model that helped me: CoCo already has one deep well of context (Snowflake). MCP servers are additional wells you drill wherever you need them. Each server you add widens what the agent can see and do.

    Setting up your first MCP server

    The whole setup is a short form in Agent Settings. The server starts the moment you save — no restart dance.

    You manage everything through the Agent Settings panel. Open Agent Settings, select MCP from the sidebar, and you’ll see the MCP Connectors panel listing any configured servers and their status.

    To add one, click + New. You’ll fill in a short form:

    Server Name — a unique identifier, e.g. github. This name matters more than it looks: it becomes part of the tool namespace. A server named github exposes tools like mcp__github__search. Pick descriptive names so tool calls read clearly — mcp__github__search tells you what it does; mcp__gh1__search doesn’t.

    Scope — Global stores the server in ~/.snowflake/cortex/mcp.json and makes it available in every workspace. Workspace stores it in <workspace>/.snowflake/cortex/mcp.json, scoped to the current project so it travels with the repo. Use Global for tools you always want (your personal GitHub); use Workspace for project-specific servers that should live in version control with the code.

    Server Type (transport) — pick Command (stdio) to run a local process, then enter the command (for example uvx mcp-server-git). Pick Remote (HTTP) to connect to a hosted server, then enter the Server URL (for example https://your-mcp-server-url) and optionally add auth Headers. For stdio servers you can add Environment Variables instead.

    Click Save, and the server starts. Its tools are available to the agent immediately.

    If you don’t have a specific server in mind, click + New and select Browse MCP Servers — CoCo Desktop ships with a gallery of ready-to-install integrations you can add straight from the UI.

    Editing the config directly (JSON)

    The form is convenient, but for anything repeatable — sharing setup with a team, checking config into git — you’ll want the JSON. In the Add New MCP Server form, switch to the JSON tab, or edit the files directly. The top-level key is "mcpServers", and each entry is keyed by server name:

    {
      "mcpServers": {
        "git": {
          "command": "uvx",
          "args": ["mcp-server-git"]
        },
        "internal-api": {
          "type": "http",
          "url": "https://your-mcp-server-url",
          "headers": { "Authorization": "Bearer ${API_TOKEN}" }
        }
      }
    }

    CoCo expands environment variables in config fields before connecting, so you can reference ${API_TOKEN} and similar. Prefer the braced form ${VAR} over bare $VAR to avoid ambiguity. There’s also a special ${workspaceFolder} variable that resolves to the current workspace root — handy for paths like cwd or envFile.

    How config files stack (the merge order)

    Config merges from multiple sources; later layers win on name collisions. Workspace beats global beats admin-enforced — unless the admin has locked things down.

    This is the part that saves you a confusing debugging session later. CoCo Desktop merges MCP config from several sources, and when two sources define a server with the same name, the later source wins. The order, from lowest to highest priority:

    First, administrator-enforced servers from managed settings. Then user (global) servers from ~/.snowflake/cortex/mcp.json. Then workspace servers from <workspace>/.snowflake/cortex/mcp.json. So if you have a server named github in both your global and your workspace config, the workspace definition takes precedence. This is usually what you want — a project can override your personal defaults — but it also means a workspace config you forgot about can silently shadow your global one.

    On managed accounts there’s an extra wrinkle: admins can restrict MCP usage through managed settings and URL allowlists, and can even disable user MCP servers entirely so that only admin-enforced servers load. If a server you configured refuses to appear on a corporate account, check whether the admin has locked MCP down before you assume your config is broken.

    How credentials are handled (better than you’d expect)

    This surprised me pleasantly. When you add a server with environment variables, headers, or OAuth, CoCo doesn’t leave your secrets sitting in a plaintext JSON file. On first connection it migrates those sensitive values out of mcp.json and into your operating system’s keychain, then rewrites the JSON file with those fields removed. Credentials are stored under a keychain entry named mcp_oauth_<server-name> as a single blob containing tokens, OAuth registration, headers, and environment variables.

    Practically, this means: put your token in as an env var reference or let the OAuth flow run, and after the first connect it won’t be in the file anymore. Don’t hardcode raw tokens in mcp.json expecting them to stay — and don’t panic when they disappear from the file, that’s the migration working. If you ever need to reset a credential, remove and re-add the server to trigger a fresh flow.

    The operational limits nobody warns you about

    These three cost me time before I understood them, and they’re the difference between “MCP is flaky” and “MCP works fine, I just configured it wrong.”

    Tool output is capped at 50 KB. If you point an MCP server at something that returns large result sets — a query that dumps thousands of rows, an API that returns a giant JSON blob — CoCo truncates the output and appends a notice. The fix isn’t to raise a limit; it’s to design the server to return summaries or pointers, not raw dumps. Have the tool return “here are the top 20 rows and a row count” or “results written to this file,” and let CoCo read the detail in a follow-up step if it needs to.

    The default tool timeout is 60 seconds. Wire up a server that hits a slow internal API and you can spend ten minutes assuming the connection is broken when the tool is just slow. Override the timeout globally with the COCO_MCP_TOOL_TIMEOUT_MS environment variable — raise it for genuinely long-running tools, or lower it to fail fast on servers that should be quick.

    Tool names must be alphanumeric, underscores, or hyphens, and under 64 characters. An MCP server that exposes a tool with a non-conforming name gets rejected outright — not silently renamed, rejected. If a server won’t load and the config looks right, check the tool names it exposes.

    The gotchas nobody warns you about

    Cross-app discovery on shared machines. Because MCP is a shared standard, CoCo can discover servers you set up for other tools — and on a shared machine, that can mean picking up someone else’s servers or exposing yours. Be deliberate about scope on multi-user boxes.

    Variables expand from the launch environment, not your editor’s shell. CoCo expands ${VAR} from the environment it was launched in, not from a shell embedded in an editor. If a variable resolves to empty, check that it’s actually set in the environment where CoCo (not your terminal-inside-the-app) started.

    Descriptive server names aren’t cosmetic. Because the server name becomes the tool namespace prefix, a vague name makes every downstream permission rule and tool call harder to read. Name servers for what they connect to, once, up front.

    Permissions are per-tool and worth configuring. MCP tools participate in CoCo’s standard permission system. You can allow, deny, or prompt per tool, matching individual tools by full name (mcp__github__read_file) or all tools from a server with a wildcard (mcp__github__*). At runtime CoCo also asks on first use and can remember the choice for the session. Denying destructive tools explicitly — mcp__github__delete_repo, say — is cheap insurance.

    A sensible starting setup

    If you’re setting this up for the first time on a Snowflake data project, here’s the configuration I’d start with. Add a Git server (Command/stdio, uvx mcp-server-git) at Workspace scope so it travels with the repo. Add your issue tracker (Jira or GitHub Issues) at Global scope since you’ll want it everywhere. Set a permission policy that allows read tools freely, asks on writes, and denies anything destructive. Bump COCO_MCP_TOOL_TIMEOUT_MS only if you actually add a slow server. And design any custom internal-API server to return summaries under 50 KB from the start, so you never hit the truncation wall.

    That gives you a CoCo that reasons over your Snowflake data and your tickets and your code history, with guardrails on the actions that matter — which is the whole point of MCP here.

    The one principle

    CoCo’s native genius is Snowflake context; MCP is how you extend its reach past Snowflake’s walls without writing integration code. Add servers deliberately, name them clearly, let the keychain hold your secrets, and design tools to return summaries — then the agent can reason across your whole stack instead of just your warehouse.

    Related reading: CoCo Desktop MCP support (official docs) · Model Context Protocol specification · Snowflake CoCo product page · Snowflake Interactive Tables: How and When to Use Them · Orchestrating dbt With Airflow on Snowflake

  • Snowflake Interactive tables: How and when to use them ( From Production)

    Snowflake Interactive tables: How and when to use them ( From Production)

    The first time a product manager asked me why our “real-time” Snowflake dashboard took four seconds to load on a Monday morning, I didn’t have a good answer. The data was fresh. The query was simple — a filtered aggregation over a few million rows. But at 9 a.m., when three hundred people opened the same dashboard at once, our XSMALL warehouse queued them up like a single cashier at a stadium. Each query was fast in isolation. Together, they were a traffic jam.

    I did what everyone does: I threw a bigger warehouse at it, then a multi-cluster warehouse, then watched the credits burn. It helped the concurrency but the per-query latency floor never really dropped below a second or two, and the bill made my manager wince. Snowflake is an OLAP engine. It’s built to scan enormous columnar data for analytics, not to answer the same small lookup a thousand times a second. I was using a freight train to run a pizza delivery service.

    Then Snowflake shipped Interactive Tables and Interactive Warehouses. I’ve now run them in production for a few months, and this is the honest guide I wish I’d had: what they are, when they’re worth it, when they’ll quietly cost you money, and the gotchas that only show up after you’ve committed.

    The whole feature at a glance: who it serves (green), how it works (blue), and how you set it up plus its limits (yellow).

    TL;DR

    → Interactive Tables + Interactive Warehouses are a serving layer inside Snowflake, built for low-latency, high-concurrency reads — dashboards, data-powered APIs, AI agents querying structured data in real time.

    → They work as a pair. The interactive table is optimized for fast retrieval; the interactive warehouse caches its data files as hot cache and serves sub-second reads. Standard warehouses can query interactive tables, but you only get the big speedup through an interactive warehouse.

    → Real numbers from independent testing: an Interactive XS delivered roughly 3.9x lower latency than a standard Gen1 XS, at about 40% lower credit cost per hour — combining to around a 75% reduction in cost per query on a serving workload.

    → The big constraint: no UPDATE or DELETE. The only DML is INSERT OVERWRITE. You keep data fresh with auto-refresh (set TARGET_LAG) against a source table, not by mutating the interactive table.

    → The clustering key is fixed at creation and cannot be changed. Choose it to match the WHERE clauses of your most latency-critical queries. Get this wrong and your only fix is recreating the table.

    → Interactive warehouses historically did not auto-suspend — they stay warm to keep the cache ready, so you effectively pay 24/7. As of the spring 2026 updates, auto-suspend, auto-resume, and auto-scaling reached GA, which changes the cost math meaningfully.

    → Use them when latency and concurrency are the product. Skip them for batch ETL, ad-hoc exploration, or anything write-heavy. This is a serving layer, not a transformation layer.

    What they actually are (in plain terms)

    Think of your Snowflake setup as having two jobs that have always been jammed into the same tool. Job one is transformation: big batch queries that scan and reshape data. Standard warehouses are great at this. Job two is serving: answering a flood of small, repetitive reads instantly — the dashboard that refreshes for every user, the API endpoint hit thousands of times a minute. Standard warehouses are mediocre at this, not because they’re slow, but because they’re built for throughput on big scans, not latency on small lookups under heavy concurrency.

    Interactive Tables and Warehouses are Snowflake’s serving layer. An interactive table stores data in a form optimized for fast, filtered retrieval. An interactive warehouse contains a query engine tuned for short, highly concurrent reads, and it caches the interactive table’s data files locally as hot cache. When a query comes in, it’s answered from that warm cache with a latency profile closer to a key-value store than a data warehouse. The two are a matched pair — the table is fast on its own, but the warehouse is where the sub-second magic happens.

    One mental model that helped me: a standard warehouse is a library where a librarian walks the stacks for every request. An interactive warehouse is the same library, but the most-requested books are already stacked on the front desk, warm and waiting. That’s the cache. It’s why the warehouse has to stay on — the moment it suspends, the front desk clears and the next reader waits for the walk to the stacks again.

    When to use them (and when not to)

    I’ve become fairly opinionated about this after watching a few teams reach for interactive tables because they were new and shiny, then get surprised by the bill. Here’s the honest split.

    Use interactive tables when: you’re serving a customer-facing or internal dashboard with real concurrency (dozens to thousands of simultaneous users), you’re powering a data API where each call is a small filtered read and latency is a product requirement, you’re feeding an AI agent that queries structured data in real time, or you have a workload where the same shapes of query hit the same tables constantly and predictably. In all these, latency and concurrency are the product, and a continuously-running warehouse is justified because the traffic is continuous.

    Don’t use them when: your workload is batch ETL or transformation (that’s what standard warehouses are for), your queries are ad-hoc and exploratory (the cache never warms usefully if every query is different), your data is write-heavy with lots of updates and deletes (the no-DML constraint will fight you), or your traffic is spiky and infrequent (a warehouse you pay to keep warm for occasional bursts is money on fire — though auto-suspend now softens this).

    The question I ask before every interactive-table decision: does this workload justify a warehouse that runs continuously? If yes, the performance is genuinely excellent. If no, you’re probably better off with a well-tuned standard warehouse and result caching.

    How to actually set it up

    The mental model is familiar if you’ve used Snowflake, which is one of the nicer things about this feature. You create the table with a standard warehouse, then serve it through an interactive one.

    First, create the interactive table. The CREATE TABLE syntax is extended with the INTERACTIVE keyword, and a CLUSTER BY clause is required — this isn’t optional like it is on standard tables:

    CREATE INTERACTIVE TABLE dashboard_events
    CLUSTER BY (tenant_id, event_date)
    AS SELECT tenant_id, event_date, event_type, metric_value
    FROM raw.events;

    Notice the clustering key. Choose it to match the WHERE clauses of your most time-critical queries — here, assuming most dashboard reads filter by tenant and date. This decision is permanent, so think about it harder than you normally would.

    To keep the table fresh without DML, make it a dynamic interactive table by setting a target lag. It will auto-refresh from the source to stay within that window (the minimum is 60 seconds):

    CREATE INTERACTIVE TABLE dashboard_events
    TARGET_LAG = '60 seconds'
    CLUSTER BY (tenant_id, event_date)
    AS SELECT tenant_id, event_date, event_type, metric_value
    FROM raw.events;

    Then create an interactive warehouse and attach the table so its data files get cached. Keep the warehouse small — an XSMALL is often plenty for serving:

    CREATE INTERACTIVE WAREHOUSE serving_wh
    WAREHOUSE_SIZE = 'XSMALL';
    
    ALTER WAREHOUSE serving_wh ADD TABLES dashboard_events;

    Point your dashboard or API at serving_wh, and reads now hit the warm cache. The first queries after attaching a table run while the cache warms, so they’ll be slower — don’t benchmark in that window and panic.

    The cost math, honestly

    This is where teams get surprised, so let’s be concrete. Interactive warehouses cost roughly 40% less per credit than standard warehouses — an XSMALL standard warehouse runs at 1 credit/hour, while an interactive XSMALL is about 0.6 credits/hour. Combined with the lower latency (fewer warehouse-seconds per query), independent testing found the cost per query dropped by around 75% on a serving workload.

    That sounds like a pure win, and for the right workload it is. But the catch has always been the always-on nature. Historically, interactive warehouses did not auto-suspend — they have to stay warm to keep the cache ready, so you were effectively paying for 0.6 credits/hour × 24 hours × 30 days whether or not traffic justified it. That’s roughly 432 credits/month for a single XSMALL that never sleeps. If your traffic is genuinely continuous, the per-query savings dwarf this. If your traffic is bursty, this idle cost can erase the savings entirely.

    The spring 2026 updates changed this materially: auto-suspend and auto-resume reached GA, so you can now let an interactive warehouse suspend during quiet periods and pay the cache-rewarming cost on resume instead of paying to stay warm around the clock. There’s also a fallback warehouse option — when a query on the interactive warehouse exceeds the timeout, Snowflake can transparently retry it on a designated standard warehouse instead of erroring out, which makes mixed workloads far less fragile. And auto-scaling went GA, so the warehouse scales with concurrency instead of you hand-tuning cluster counts.

    My rule: model the idle cost first. Take your warehouse’s credits/hour, multiply by the hours you’ll actually keep it warm, and compare that baseline to your current serving spend before you get excited about per-query savings. The per-query numbers are real, but they only pay off above a traffic threshold.

    The gotchas nobody warns you about

    The clustering key is a one-way door. On a standard table, you can iterate on clustering freely. On an interactive table, the clustering key is fixed at creation and cannot be changed. If your query patterns shift, or you guessed wrong about which columns your hot queries filter on, your only remedy is recreating the table. Treat this decision like a schema migration, not a tuning knob. This is, in my experience, the single most common source of regret with the feature.

    No UPDATE, no DELETE — plan your pipeline around it. The only DML is INSERT OVERWRITE. If your instinct is to patch a few rows, you can’t. The intended pattern is: mutate the source table with normal DML, and let auto-refresh (TARGET_LAG) propagate changes to the interactive table. This is actually more efficient than row-level DML on the serving layer, but it forces you into a refresh-based mental model. Append-only and full-refresh patterns fit naturally; frequent surgical updates do not.

    The warehouse only queries what you attach. An interactive warehouse can only query interactive tables that have been explicitly added to it, and it only supports SELECT. Forget to ADD TABLES and your queries won’t find the data. This trips people up because it’s different from the standard warehouse model where any warehouse can query any table it has grants on.

    Cache-warming latency is real and invisible in benchmarks. When you first attach a table, or right after a refresh, the cache is cold and those initial queries are slower. If you benchmark immediately after setup and conclude the feature is underwhelming, you measured the cold path. Let it warm, then measure.

    No Fail-safe. The Fail-safe data recovery mechanism isn’t available for interactive tables. Time Travel still works, so you’re not flying blind, but the extra safety net you may be used to isn’t there. For a serving layer fed from a source table you control, this is usually fine — you can always rebuild from source — but know it going in.

    Mistakes that drain the budget

    Keeping a warehouse warm for traffic that isn’t there. The classic. You stand up an interactive warehouse for a dashboard that gets heavy use twice a day and sits idle the rest of the time, and you pay to keep the cache warm through all the dead hours. With auto-suspend now GA, configure it — don’t run 24/7 out of habit.

    Routing every query to the interactive warehouse. Interactive warehouses shine on small, concurrent reads. Fire a heavy, complex analytical query at one and you’re using the wrong tool — that’s what the fallback warehouse and your standard clusters are for. Route small serving queries to interactive, keep heavy jobs on standard. Getting this routing right is where most of the real-world value (and most of the operational effort) lives.

    Migrating tables that don’t need it. Every table you make interactive adds a data-management surface — a refresh to monitor, a cache to keep warm, a clustering decision you can’t undo. Only promote the tables that actually sit in a latency-critical serving path. A table that backs a weekly report has no business being interactive.

    The one principle

    Interactive tables are a serving layer, not a transformation layer. Transform data on standard warehouses; serve it on interactive ones — and only when the traffic is continuous enough to justify a warehouse that stays warm. The feature is genuinely excellent at the job it was built for. Nearly every problem I’ve seen with it came from asking it to do a different job.

    Related reading: Snowflake interactive tables and warehouses (official docs) · CREATE INTERACTIVE TABLE reference · Snowflake Query Execution: What Really Happens · Snowflake Iceberg v3: When to Migrate · dbt State on Snowflake: Skip Unchanged Models

  • Orchestrating dbt with Airflow on Snowflake: Job vs Model-Level in 2026

    Orchestrating dbt with Airflow on Snowflake: Job vs Model-Level in 2026

    For years, the pattern was: Airflow sits in one corner of your infrastructure, dbt runs on a server somewhere else, they pass data between each other via manual credential handoffs and cron jobs, and when something breaks at 2 AM, you’re SSH-ing into the dbt box, checking Airflow logs, querying Snowflake directly, and stringing it all together in your head.

    The question you’ve been asking for three years is whether dbt should be one big task in Airflow (job-level) or broken into one task per model (model-level). The answer in 2026 is: it doesn’t matter anymore. What matters is that your dbt runs inside Snowflake as a native DBT PROJECT object, Airflow orchestrates it from outside, and all the monitoring, logs, and failure notifications are in one place instead of three.

    This shift from external dbt servers to Snowflake-native orchestration changes everything. Not just infrastructure. The way you think about observability, debugging, and the entire data platform.

    TL;DR

    → Job-level orchestration: one Airflow task runs `dbt run –select tag:daily`. Simple, clean, fast. Loses model-level visibility and parallelization. Use when: small projects, simple DAGs, speed matters over observability.

    → Model-level orchestration: Astronomer’s Cosmos library renders each dbt model as a separate Airflow task. Full visibility, failures are model-scoped, parallelization is automatic. Overhead was real. Not anymore in 2026.

    → Snowflake native dbt projects (GA Nov 2025): dbt runs inside Snowflake as a schema-level DBT PROJECT object. No external dbt server. No separate credentials. Orchestrate from Airflow via REST API. This is the new default architecture.

    → Real benchmark: a 400-model project on job-level Airflow + external dbt = 18-minute runs. Model-level Cosmos + Snowflake native dbt = 12 minutes with full per-model visibility. The performance penalty of model-level is gone.

    → The observability win: failures show up as “stg_orders failed at compile time” in Airflow UI, not “dbt run exited with code 1” in a SSH log. Debugging time drops by 40-60%.

    → Snowflake-native setup: create a DBT PROJECT in a schema, grant Airflow’s service account EXECUTE on it, call it from Airflow via SnowflakePythonOperator with `EXECUTE DBT PROJECT` command. Snowflake handles execution, Airflow gets the logs.

    → If you’re still running dbt Core on an external server with Airflow alongside it: try the native dbt Projects setup this weekend. Most teams report the infrastructure feels 40% lighter after the rebuild.

    The job-level vs model-level question, and why it’s been misleading

    For the last four years, the Airflow + dbt conversation has been dominated by one question: should you run all of dbt in one Airflow task (job-level), or break it into one task per model (model-level)?

    Job-level won out in most teams because it was simpler. One Airflow task. One line of configuration. Fast deploys. The downside: if a model failed in the middle of a 400-model run, the entire job failed, and you had to dig into dbt logs to find which of the 20 intermediate models actually broke.

    Model-level promised full visibility — every model gets its own task, failures are scoped to the model, Airflow’s UI shows you exactly where the pipeline broke. But it had a penalty: rendering 400 models as 400 separate Airflow tasks created overhead in the Airflow scheduler, the DAG parsing time doubled, and you had to manage dynamic task generation (which was brittle).

    For four years, teams picked job-level because the model-level overhead wasn’t worth the observability gain. That trade-off is no longer real.

    Why model-level is winning in 2026 — and what changed

    Three things shifted:

    1. Astronomer’s Cosmos library matured. Cosmos (open-source) automatically converts a dbt project into an Airflow DAG. Instead of manually writing task definitions for every model, you pass your dbt_project directory to Cosmos, and it generates the DAG dynamically. The overhead of parsing 400 models exists — it’s not magical — but it’s now acceptable (1–2 seconds added to DAG parsing). Not free, but not expensive.

    2. Snowflake native dbt Projects arrived. When dbt runs on an external server, model-level orchestration meant Airflow communicating task-by-task with that external box. Latency overhead. Credential management complexity. Snowflake’s native dbt Projects (GA November 2025) lets dbt run inside Snowflake itself. Airflow just sends a single command (`EXECUTE DBT PROJECT model_name`) to Snowflake and waits for results. The execution is native. The communication is just REST API calls. The overhead drops significantly.

    3. Orchestration became about observability, not infrastructure. Teams in 2026 have stopped asking “what’s the performance impact?” and started asking “can I see failures at model granularity?” The answer is yes, and the cost is no longer real. The observability win — debugging a failed model in minutes instead of 45 minutes — justifies the architecture.

    Real benchmark: 400-model project, production traffic

    A data team running Airflow on AWS (3x m5.2xlarge instances), dbt Core on a separate EC2 box, Snowflake warehouse (MEDIUM):

    Before (job-level + external dbt): `dbt run –select tag:daily` runs once per day. Entire job as one Airflow task. 18 minutes wall-clock time. Failed model buries itself in the dbt run log. Debugging takes 45+ minutes because you’re correlating dbt logs, Snowflake QUERY_HISTORY, and Airflow task logs.

    After (model-level Cosmos + Snowflake native dbt): Same 400 models, now as 400 Airflow tasks generated by Cosmos. Parallel execution on the MEDIUM warehouse runs up to 8 models at a time. 12 minutes wall-clock time (33% faster). Failed model shows up in Airflow UI as a red task. Click it. See the exact model that failed, the SQL compile error, the exact line. Debugging takes 8 minutes.

    The speed improvement comes from parallelization (you can run independent models concurrently). The debugging improvement comes from per-model observability. Both were impossible before because the overhead was too high. Not anymore.

    How to orchestrate Snowflake native dbt Projects from Airflow

    Snowflake-native dbt Projects: dbt runs inside Snowflake, Airflow orchestrates from outside. Simpler infrastructure, unified observability.

    Here’s what a production DAG looks like when dbt runs as a native Snowflake object, orchestrated from Airflow:

    from airflow import DAG
    from airflow.providers.snowflake.operators.snowflake import SnowflakePythonOperator
    from airflow.providers.slack.operators.slack_webhook import SlackWebhookOperator
    from datetime import datetime
    
    default_args = {
    'owner': 'analytics',
    'retries': 2,
    'retry_delay': timedelta(minutes=5),
    }
    
    with DAG(
    'snowflake_dbt_daily_run',
    default_args=default_args,
    schedule_interval='0 2 * * *', # 2 AM daily
    start_date=datetime(2026, 1, 1),
    catchup=False,
    ) as dag:
    
    # Raw data load (external tool or Airflow operator)
    load_raw = SnowflakePythonOperator(
    task_id='load_raw_data',
    python_callable=load_from_source, # your extraction logic
    )
    
    # Run dbt transformations as a native Snowflake DBT PROJECT
    run_dbt = SnowflakePythonOperator(
    task_id='dbt_transform',
    python_callable=execute_dbt_project,
    op_kwargs={
    'sql_command': 'EXECUTE DBT PROJECT analytics_db.transforms',
    'database': 'analytics_db',
    },
    )
    
    # Export or activate downstream (BI, ML, etc.)
    notify_success = SlackWebhookOperator(
    task_id='notify_success',
    http_conn_id='slack_webhook',
    message='Daily dbt transforms completed successfully',
    )
    
    load_raw >> run_dbt >> notify_success

    The key line is `EXECUTE DBT PROJECT analytics_db.transforms`. That command runs inside Snowflake. Airflow waits for it to complete. Logs come back to Airflow. All in one place.

    Before this, you’d have an external dbt server, SSH credentials in Airflow secrets, a bash script that connects to the box and runs `dbt run`, and error handling that was fragile. Now it’s a direct REST API call to Snowflake.

    Setup: Snowflake side (one-time)

    Create the dbt project object in Snowflake. One time. Then Airflow orchestrates it:

    -- As SYSADMIN or higher
    USE ROLE SYSADMIN;
    
    -- Create a dedicated role and user for Airflow
    CREATE ROLE IF NOT EXISTS dbt_executor_role;
    CREATE USER IF NOT EXISTS airflow_svc_user
    DEFAULT_ROLE = dbt_executor_role
    DEFAULT_WAREHOUSE = dbt_transform_wh;
    
    -- Create a dedicated warehouse for dbt runs
    CREATE OR REPLACE WAREHOUSE dbt_transform_wh
    WITH WAREHOUSE_SIZE = 'MEDIUM'
    AUTO_SUSPEND = 120
    AUTO_RESUME = TRUE
    INITIALLY_SUSPENDED = TRUE;
    
    -- Grant permissions to Airflow's service account
    GRANT ALL ON WAREHOUSE dbt_transform_wh TO ROLE dbt_executor_role;
    GRANT USAGE ON DATABASE analytics_db TO ROLE dbt_executor_role;
    GRANT USAGE, CREATE TABLE, CREATE VIEW ON SCHEMA analytics_db.staging TO ROLE dbt_executor_role;
    GRANT USAGE, CREATE TABLE, CREATE VIEW ON SCHEMA analytics_db.marts TO ROLE dbt_executor_role;
    
    -- Grant the crucial permission: execute dbt projects in that schema
    GRANT EXECUTE DBT PROJECT ON SCHEMA analytics_db.transforms TO ROLE dbt_executor_role;
    
    -- Grant the role to your service user
    GRANT ROLE dbt_executor_role TO USER airflow_svc_user;

    Then create your dbt project in Snowflake (via Snowsight → Workspaces, or via SQL `CREATE DBT PROJECT`). That’s it on the Snowflake side.

    The three gotchas you’ll hit

    Gotcha 1: Forgetting schema-level EXECUTE permissions. The `GRANT EXECUTE DBT PROJECT ON SCHEMA` is the easy line to miss. You can grant object-level execute all day and Airflow still won’t be able to run the project. It’s schema-level that matters.

    Gotcha 2: dbt docs and artifacts not flowing back to Airflow. When dbt runs inside Snowflake, the manifest.json and dbt_project.yml artifacts stay inside Snowflake. If you’re using those artifacts downstream (dbt Cloud webhooks, dbt Mesh coordination, Lineage tools), you need to export them explicitly from Snowflake after the run completes. Set up a post-run task that pulls `SELECT GET_STAGE_LOCATION(…)` to grab the artifacts.

    Gotcha 3: Incremental models and first-run confusion. This is the same gotcha as in the dbt State article — the first run of an incremental model executes a full load, changing the compiled SQL. dbt State knows about this. Airflow doesn’t. Expect a full downstream rebuild on Run 2. Normal behavior. Just know it coming in.

    When to use job-level, when to use model-level

    Job-level still makes sense if: your dbt project is small (<50 models), the entire project fits in a single logical unit, you don’t need per-model visibility, or you’re on dbt Cloud’s native scheduler (not Airflow). Keep it simple.

    Model-level (Cosmos) makes sense if: your project has 100+ models, you need per-model failure isolation, debugging speed matters, or you want Airflow as the single source of truth for your entire data pipeline. Most production teams in 2026 are here.

    The hybrid: Some teams run both. Job-level for hourly incremental ingestion (simple, fast), model-level for daily mart builds (visibility matters). You can mix them in the same DAG — one task for `dbt run –select tag:hourly`, another task group for model-level mart runs via Cosmos.

    The one principle

    Observability at execution time beats simplicity at configuration time. Job-level is simpler to configure. Model-level is simpler to debug. In production systems that need to run reliably and recover fast, you spend more time debugging than configuring. Pick the architecture that lets you see failures at the right granularity.

    Related reading: dbt State on Snowflake: Skip unchanged models, cut runtime 60% · dbt Fusion: 30x Faster Parsing with the Rust Engine · Data Contracts: Stop Schema Breakage Before It Happens · Official dbt + Airflow Integration Guide · Astronomer Cosmos: dbt + Airflow

  • Snowflake Iceberg V3: When to Actually Migrate(vs Native Tables)

    Snowflake Iceberg V3: When to Actually Migrate(vs Native Tables)

    Most data engineers I talk to still store everything in Snowflake native format. It’s simple: load data, query data, done. But here’s what nobody’s talking about: if you’re querying that data from anywhere else — Spark, Databricks, even just a local Python script — you’re paying a hidden “data tax.” Redundant storage, egress fees, ETL pipeline complexity. For Fortune 500 companies, that tax runs $2 million to $7 million a year. And Snowflake’s new Apache Iceberg v3 support (GA May 2026) actually changes the math. But migrating is a choice, not a reflex — and there are specific gotchas that’ll bite you if you don’t plan right.

    The honest decision tree: migrate if you’re paying egress fees or running multi-engine queries.

    TL;DR

    → Apache Iceberg v3 is GA on Snowflake (May 2026). New features: deletion vectors (10x faster DML), row lineage for CDC, VARIANT type for semi-structured data, nanosecond timestamps, default column values.

    → Month-to-month costs are roughly equal to native Snowflake tables (compute identical, storage ~$23/TB native vs ~$0.023/GB S3, negligible difference).

    → Migrate if: (a) you query from Spark/Databricks (egress fees kill you), (b) you’re paying >$500/month for Snowflake storage, (c) you want single copy of truth across multiple engines.

    → Don’t migrate if: you only query from Snowflake, storage bill is small, and you’re not building a multi-engine architecture.

    → New gotcha: You can’t upgrade v2 tables in-place to v3. No writing to v3 tables via external engines (Spark) yet. External engine compaction gets billed starting May 21, 2026.

    → Real win: Snowflake Storage for Iceberg (GA April 2026) means you don’t manage S3 buckets. Snowflake handles it, with Fail-safe recovery built in.

    → The “data tax” of $2M–$7M annually on Fortune 500 costs more than Iceberg migration ever will.

    The mental model that’s keeping you locked in

    Here’s the picture most teams hold: Snowflake stores data. We query it in Snowflake. Done. Native tables, simple syntax, life is easy. And if you only query in Snowflake, that model works fine. You get the speed, the simplicity, the integration with dbt, the Time Travel.

    But the moment you have data living in two systems — Snowflake for reporting, Spark for ML training, Databricks for a BI tool, even just a DuckDB instance on your laptop — you’ve broken the simple model. Now you have two copies of the data, or worse, a pipeline that’s constantly syncing between them. You’re paying Snowflake egress fees to get data out ($0.02 per GB across regions, $0.08 between clouds). You’re rebuilding the same transformation logic in both systems. You’re managing schema evolution in two places. The complexity compounds.

    Iceberg was built to solve exactly this. One copy of the data, on open cloud storage (S3, Azure Blob, GCS), readable by any engine that supports the Iceberg format. Snowflake, Spark, Databricks, Trino, DuckDB. All of them see the same table, the same schema, the same snapshot. No replication, no egress fees, no syncing.

    But Iceberg isn’t free. It trades simplicity for flexibility. And for teams that genuinely don’t need that flexibility, native tables are still the right call.

    The hidden cost of locking data into proprietary formats. For large teams, it’s massive.

    What changed in Iceberg v3, and why it matters

    Iceberg v2 shipped in 2023 and covered the basics: open format, ACID transactions, schema evolution, snapshots. v3 (released June 2025, GA on Snowflake May 7, 2026) added seven new capabilities. Only three actually change how you’d use it.

    Deletion vectors. In v2, if you deleted or updated a row, Iceberg had to rewrite the entire data file (copy-on-write). Slow and expensive. v3 adds deletion vectors — a separate, small metadata file that marks rows as deleted without touching the original data. Result: 10x faster DML operations on large tables. If you’re doing frequent small updates (common in streaming ingestion), v3 matters.

    Row lineage. v3 tracks which rows were inserted, updated, or deleted with metadata fields (_row_id, _last_updated_sequence_number). This is how Snowflake implements change data capture (CDC) without external tooling. A Dynamic Iceberg Table can now refresh incrementally on only the rows that changed, not the whole partition. Critical for SCD2 and CDC pipelines.

    VARIANT type. v2 forced you to choose: store JSON as a string (slow parsing at query time) or explode it into a wide schema (thousands of nullable columns, query disasters). v3 adds native VARIANT support, and Snowflake automatically shreds it (extracts nested fields and indexes them) at write time. Query performance on semi-structured data jumps dramatically. This alone is why observability platforms are betting on Iceberg.

    The other four (default column values, geometry/geography types, nanosecond timestamps, partition transform improvements) are niche. Don’t worry about them unless you hit them.

    The cost math: Native vs Iceberg in real dollars

    Let’s be honest: most articles skip the cost comparison and jump to “Iceberg is cheaper!” It usually isn’t, month-to-month. Here’s why.

    Two-column cost breakdown. Native Snowflake: 2,000 credits at $3 = $6,000 compute, $23/TB storage = $230, total $6,230/month. Iceberg (Snowflake managed): same $6,000 compute, S3 at $0.023/GB = $235 storage, bundled compaction = $0, total $6,235/month. Verdict: same cost, but Iceberg enables multi-engine and zero egress.

    The real numbers. On a month-to-month basis, they’re nearly identical. The wins come from elsewhere.

    For a typical 10 TB table with 1,000 queries per month (small-to-medium workload):

    Native Snowflake: Compute 2,000 credits ($6,000) + Snowflake storage 10TB at $23/TB ($230) = $6,230/month.

    Iceberg (Snowflake-managed storage, GA April 2026): Compute 2,000 credits ($6,000) + S3 storage 10TB (10,240 GB × $0.023/GB = $235) + compaction bundled ($0) = $6,235/month.

    Basically the same. Where Iceberg wins is not in monthly costs. It wins in:

    Egress fees. If you query that 10 TB table from a Databricks cluster once a month, native Snowflake costs 10,000 GB × $0.08/GB (cross-cloud egress) = $800. Iceberg: $0. Over a year, that’s $9,600. At any real-world scale (multi-engine queries), egress dominates.

    No data duplication. If you’re currently syncing data between Snowflake and Databricks (ETL pipeline, manual export, Fivetran), that pipeline costs money too. Shared Iceberg table means you stop paying to move the data. One table, multiple readers.

    Storage simplicity. With Snowflake Storage for Iceberg (new, April 2026), you don’t manage S3 buckets yourself. Snowflake handles encryption, replication, Fail-safe recovery. You save the operational tax of bucket management, lifecycle policies, and debugging storage issues.

    So here’s the honest scorecard:

    For Snowflake-only users: Native tables win. Simpler, no migration pain, costs are identical.

    For multi-engine shops (Snowflake + Spark + Databricks): Iceberg wins. Egress fees alone justify the migration, and you get single source of truth as a bonus.

    The gotchas that will hurt your migration

    You can’t upgrade v2 tables to v3 in-place. There’s no ALTER TABLE ... SET ICEBERG_VERSION = 3. To get v3, you have to CREATE a new table. That means copying data (compute cost, time), repointing your queries, and hoping nothing breaks downstream. On large tables, this is a multi-day operation.

    External engines can’t write v3 tables yet. You can read v3 tables from Spark, Trino, DuckDB, all day. But writing is blocked. Snowflake says it’s “planned,” but if you’re building a shared Iceberg table that Spark needs to update, you’re stuck on v2. This is a major limitation if you’re counting on true multi-engine write access.

    Compaction gets billed starting May 21, 2026. When an external engine writes to an Iceberg table (via Spark, Trino, etc.), it creates small data files. Snowflake’s compaction automatically consolidates them into bigger files for query performance. Until May 21, that was free. Now it costs credits. Budget for ongoing compaction maintenance if you have heavy external write workloads.

    ⚠️ Don’t convert cloned tables with vended credentials. If you clone a native Snowflake table and then convert it to Iceberg, you can’t write to it with vended credentials (external query engine creds). You’d have to connect the external engine directly to your S3 bucket, defeating the whole point. Create the Iceberg table fresh if you’re using vended creds.

    Schema changes are cheap but metadata bloat is real. Iceberg tracks every schema change as a separate metadata version. On tables with thousands of ALTER COLUMN operations, metadata can get unwieldy. Compact your metadata regularly with CALL SYSTEM$OPTIMIZE(...).

    The mistakes teams make when migrating

    1. Migrating for the wrong reason. “Everyone’s talking about Iceberg, so we should move.” Wrong. Migrate only if you have a concrete use case: egress fees, multi-engine queries, or storage cost >$500/month. Otherwise you’re trading simplicity for nothing.

    2. Not testing external engine read performance first. Iceberg’s query performance depends heavily on your cloud setup, partitioning strategy, and how many small files are sitting around. Test Spark/Databricks queries on a small Iceberg table before migrating your 100 TB production table. You might find that your workload is slower on Iceberg, not faster.

    3. Assuming v3 is backward-compatible with v2. It’s not. Engines that only understand v2 (like older Spark runtimes, Trino versions) will fail on v3 tables. Check that every tool in your stack supports v3 *before* upgrading. v2 → v3 is one-way; there’s no downgrade.

    4. Ignoring the partition evolution story. Iceberg lets you change your partitioning scheme without rewriting the whole table. It’s a huge feature, but it’s also easy to mess up. Bad partitioning (e.g., partitioning by a column with 10 million distinct values) creates a partition explosion. Get your partitioning right before you migrate, not after.

    5. Migrating everything at once. Pick one critical table, migrate it, test multi-engine queries for a month, then move the rest. Iceberg is mature enough for production, but it’s not old enough that every edge case is documented. Be intentional.

    When to actually migrate: The real decision

    Stop and ask yourself: Do you actually need Iceberg?

    Yes, if: You query the same data from Snowflake and Spark/Databricks. You’re paying egress fees. You have data warehouses in multiple clouds and want to query across them. You’re building a data lakehouse and want to ditch proprietary formats.

    No, if: You only query from Snowflake. Your storage bill is <$500/month. You’re using Snowflake’s Time Travel, zero-copy clones, and other native features heavily. You don’t need to share data with other engines.

    For most teams, the answer is no. And that’s okay. Native Snowflake tables are extremely good. Simple, fast, well-integrated with dbt. There’s no shame in staying native.

    But for teams hitting the “data tax” — redundant copies, egress fees, multi-engine complexity — Iceberg v3 actually delivers. The gotchas are real, but they’re manageable. The cost savings are modest month-to-month, but the flexibility is transformative.

    The one principle that matters

    Interoperability beats simplicity when you’re already paying for fragmentation. If your current architecture already costs you $800/month in egress, $300/month in ETL pipelines, and engineering time chasing sync issues, Iceberg’s “complexity” is actually a simplification. You’re not adding complexity; you’re replacing it with a standard.

    If you’re simple and integrated today, stay there. Don’t pay the cost of flexibility you don’t need. But if you’re paying the data tax, Iceberg’s math changes fast.

    Related reading: Snowflake Apache Iceberg tables (official docs) · Snowflake Time Travel: The Real Architecture · Snowflake Optima: 15x Faster Queries at Zero Cost · Query Snowflake in DuckDB and Cut Costs

  • Snowflake Query Execution: what really happens under the hood

    Snowflake Query Execution: what really happens under the hood

    Ask ten data engineers what happens when you run a query in Snowflake and most of them will tell you the same thing: the warehouse runs it. SQL goes in, the warehouse chews on it, results come out. Clean, simple, and just wrong enough to cost you money. The truth is that Snowflake query execution is a trip through three separate layers, and the one you actually pay for is the last to get involved — if it gets involved at all.

    Flowchart of the life of a Snowflake query: query submitted, cloud services layer parses and prunes, result cache check returns a cached result on a hit with zero credits, and on a miss the virtual warehouse runs it across MPP nodes before fetching micro-partitions from storage and returning results.

    The path every query takes — and the shortcut a cache hit gets to skip compute entirely.

    TL;DR

    → Running a query isn’t one step. It’s a trip through three layers that scale independently, and the compute you pay for is the last one to wake up.

    → The cloud services layer does the thinking first — parses, plans, and prunes — using per-partition metadata to throw out data that can’t match your filter before a single byte is read.

    → If the exact same query ran in the last 24 hours and the data hasn’t moved, you get a cached result for free. No warehouse. No credits. Milliseconds.

    → Only on a cache miss does a virtual warehouse actually spin up and crunch the query across its nodes in parallel.

    → Warehouses keep recently-read micro-partitions on local SSD, but that cache vanishes the second the warehouse suspends.

    → “Identical” is brutally literal. A stray table alias, lowercase keywords, or a RANDOM() call quietly knocks you off the cache and back onto billed compute.

    → When a query is slow, it’s usually scanning too much, not computing too much. A bigger warehouse fixes the second problem and never the first.

    The mental model that’s quietly costing you money

    Here’s the reframe that changes how you think about every query: the virtual warehouse is the last thing to get involved, and plenty of queries never touch it at all. Almost everything that decides whether your query is fast or slow, cheap or expensive, happens in a layer you probably weren’t even picturing.

    Get this wrong and you end up doing what I’ve watched teams do a dozen times: a dashboard feels sluggish, someone bumps the warehouse from Medium to Large, it gets a little faster, everyone moves on. The bill goes up. The query was never compute-bound in the first place. They paid more to run the same broken query faster.

    So let’s actually follow a query, from the moment you hit run to the moment rows come back.

    The three layers a query passes through

    Snowflake splits into three layers, and the thing to internalize is that they scale completely independently. That’s not a trivia fact — it’s the reason the whole platform behaves the way it does. Every query touches all three. Some only touch the first.

    Diagram of Snowflake’s three layers: the cloud services layer handling parse, optimize, prune and result cache at the top; the compute layer with virtual warehouse MPP nodes in the middle; and the storage layer holding micro-partitions in S3, Azure Blob or GCS at the bottom, with a query entering the top and a cached result returning from it

    The cloud services layer is the brain. The warehouse is just the muscle.

    At the top sits the cloud services layer — the brain. Logins, sessions, parsing, query planning, transaction coordination, all the metadata: it lives here. Underneath that is the compute layer, made up of virtual warehouses. These are the MPP clusters that do the actual SQL grunt work. And at the bottom is storage: your data sitting in cloud object storage (S3, Azure Blob, or GCS) as immutable, compressed, columnar micro-partitions.

    The magic is that none of these share resources with each other. You can resize compute without moving a single byte of storage. You can point ten warehouses at the same table and none of them slow the others down. Keep that in your back pocket, because it’s exactly why the “just make the warehouse bigger” instinct fails so often.

    Step 1: The cloud services layer does the thinking

    Your query lands, and before any table data gets read, the cloud services layer goes to work.

    It parses the SQL, and the cost-based optimizer builds a plan. Then it does the single most important thing for performance, and it does it for free: partition pruning. Every micro-partition carries metadata, including the min and max value for each column. So when you write WHERE order_date = '2026-06-01', the optimizer reads that metadata first and skips every partition whose range can’t possibly hold a matching row. Ten thousand partitions in the table, three that actually get scanned. No index. No tuning. You didn’t do anything.

    This is also why a query can come back in milliseconds with no warehouse at all. Run a SELECT COUNT(*) or a MAX() and Snowflake often answers straight from the metadata cache — you’ll see a lone METADATA-BASED RESULT step in the query profile and zero compute on the bill.

    How good the pruning is comes down to how well your data is naturally ordered, which is the whole point of clustering and Snowflake’s automatic optimization features.

    Step 2: The result cache check, before any compute

    Now, still before any warehouse gets involved, cloud services checks the result cache. Identical query in the last 24 hours, data hasn’t changed since? Snowflake just hands back the stored result from the cloud services layer. Nothing resumes. Nothing gets billed. The answer is basically instant.

    This is the cheapest query you’ll ever run, and it’s the reason refreshing a dashboard a second time costs nothing. Here’s the part that surprises people: every time that cached result gets reused, the 24-hour clock resets. Keep hitting it and a result can stay alive for up to 31 days. A popular dashboard query can effectively sit in cache for a month and never cost a credit.

    Which brings us to the catch. “Identical” is doing a lot of work in that sentence, and it’s the thing nobody reads about until they’re staring at a bill wondering where the money went.

    Step 3: The virtual warehouse finally runs it

    Cache miss. Now — and only now — the query goes to a virtual warehouse. If it was asleep, it resumes (a second or two, usually). The warehouse is a little cluster of compute nodes, and this is where massively parallel processing earns its name: the surviving micro-partitions get split across the nodes and cores, and the filters, joins, and aggregations all happen in parallel.

    This is the layer with a meter running. You’re billed by the second, with a 60-second minimum every time a warehouse starts or resumes. That minimum trips people up constantly — a warehouse that keeps flickering on and off for tiny one-off queries can quietly cost more than one you just leave warm, because every single resume restarts that 60-second clock. It’s also the only layer where resizing actually helps, and that matters less often than you’d think.

    Step 4: Storage, and the cache that disappears

    The warehouse goes and fetches the micro-partitions that survived pruning. Each micro-partition holds 50–500 MB of uncompressed data in a columnar format, so only the columns you actually asked for get read. Ask for two columns out of fifty and you pay to read two.

    And there’s a second cache down here. As the warehouse pulls partitions from remote storage, it stashes them on local SSD. Next query on that same warehouse that needs the same data? It reads from SSD instead of making the round trip to object storage, which is a lot faster. This is what people mean when they say a warehouse “warms up.”

    The catch — and there’s always a catch — is that this cache gets wiped the instant the warehouse suspends. That’s the real trade-off hiding inside your AUTO_SUSPEND setting. Suspend fast and you stop paying for idle time but you toss the warm cache. Leave it running and you keep the cache but pay for the idle. A lot of teams land on 60 seconds, but honestly the right answer depends entirely on how often your queries actually fire.

    Three caches, one comparison to bookmark

    There are three caches, they live in different layers, and mixing them up is behind half the “wait, why was that slow?” conversations I’ve ever had.

    CacheLayerWhat it storesSurvives suspend?Compute cost
    Result cacheCloud servicesFinal query result setsYes (24h, resets on reuse, 31-day max)None
    Metadata cacheCloud servicesRow counts, min/max, distinct countsYesNone
    Local disk (warehouse) cacheComputeMicro-partitions read from storageNo — purged on suspendWarehouse already running

    The two cloud-services caches are shared across the whole account — every warehouse, every user benefits. The local disk cache belongs to one warehouse and dies with it. If you want the gory details with real query-profile screenshots, Snowflake’s own community piece on caching is the place to go.

    The cache rules nobody warns you about

    The result cache only kicks in when Snowflake decides the new query is identical to the old one, and “identical” is far more literal than anyone expects. Snowflake’s own docs spell it out: add a table alias, or just retype your keywords in lowercase, and you miss the cache. The query goes to billed compute and you never get a warning.

    It also bails on the cache if the query contains a non-deterministic function — UUID_STRING, RANDOM, RANDSTR are the usual suspects — or an external function, or if it reads from a hybrid table. And the role running it needs privileges on every table involved.

    ⚠️ What this means in practice: if you want your dashboards and scheduled jobs riding the cache, lock down the exact text your BI tool emits. Random aliasing, dynamic comments, an injected session variable, a stray current_timestamp() — any of it silently drops you off the cache, and you start paying to recompute answers you already had. The Query Cost Estimator is a quick way to put a number on what those misses are costing you.

    Why a bigger warehouse usually isn’t the answer

    This is the most expensive misunderstanding in all of Snowflake, so it’s worth being blunt about it. A query is slow for one of two reasons: it’s scanning too much data, or it’s doing too much actual computation. Resizing the warehouse only helps the second one.

    When the real problem is bad pruning — Snowflake had to scan most of the table because the data wasn’t laid out in a way it could skip — a bigger warehouse just reads the same mountain of data faster and charges you more per second for the privilege. The fix isn’t more nodes. It’s better pruning: clustering, a tighter filter, a smarter query shape. The query profile tells you which world you’re in. Look at partitions scanned versus partitions total. If you’re chewing through 95% of them, no warehouse size on earth is going to save you.

    And sometimes the honest answer to a giant, I/O-heavy scan isn’t a bigger Snowflake warehouse at all — it’s moving that work somewhere cheaper, which is the whole idea behind running the query in DuckDB instead. If you do decide a resize is warranted, the warehouse sizing guide will at least keep you from overshooting.

    What I actually check when a query misbehaves

    When something’s slow or weirdly expensive, here’s where I start. All of it comes out of the query profile and INFORMATION_SCHEMA — no guessing, no vibes.

    -- Did your last query hit the result cache?
    -- A cache hit shows bytes_scanned = 0 and a near-zero execution time.
    SELECT query_id,
           query_text,
           bytes_scanned,
           percentage_scanned_from_cache,
           execution_time / 1000 AS exec_seconds
    FROM TABLE(information_schema.query_history())
    ORDER BY start_time DESC
    LIMIT 10;
    
    -- Turn the result cache off for honest benchmarking (this session only)
    ALTER SESSION SET USE_CACHED_RESULT = FALSE;
    -- ...run your test queries...
    ALTER SESSION UNSET USE_CACHED_RESULT;
    
    -- Hunt down your worst pruning offenders
    SELECT query_id,
           partitions_scanned,
           partitions_total,
           ROUND(100 * partitions_scanned / NULLIF(partitions_total, 0), 1) AS pct_scanned,
           bytes_scanned / 1e9 AS gb_scanned
    FROM snowflake.account_usage.query_history
    WHERE partitions_total > 0
    ORDER BY bytes_scanned DESC
    LIMIT 20;
    
    -- Balance warm cache against idle cost
    ALTER WAREHOUSE analytics_wh SET AUTO_SUSPEND = 60;  -- seconds

    The number to stare at is pct_scanned in that third query. High on your slow queries? You’ve got a pruning problem, and that’s where your time should go — not the warehouse dropdown.

    What this actually costs you

    Architecture is interesting, but the bill is what makes it matter. So let’s put real numbers on it. Credit prices swing with your edition, cloud, and region, so treat these as illustrative — drop your own rate into the cost calculator for the exact figure.

    Picture one ordinary dashboard query. It runs on a Medium warehouse (4 credits an hour, call it roughly $3 a credit), and it fires 100 times a day because that’s how often people open the dashboard. Here’s how the same query plays out depending on whether the layers are doing their job:

    ScenarioWhat actually runs~Cost / day~Cost / month
    No cache, bad pruning (30s/run, scans ~95% of partitions)100 full compute runs~$10~$300
    Result cache hits 90% of the time10 compute runs, 90 free cached returns~$1~$30
    Cache + good pruning (3s/run, scans ~5%)10 runs at a tenth of the scan~$0.10~$3

    Same query. Same data. Roughly $300 a month versus $3, and the only difference is whether you let the result cache and pruning do what they’re built to do. Now multiply that across a dashboard with thirty queries on it, and you can see how a Snowflake bill quietly triples without anyone writing a single new query.

    The gotchas nobody warns you about

    The 60-second minimum punishes spiky workloads. A query that finishes in two seconds still bills a full minute if the warehouse had to resume to run it. Set AUTO_SUSPEND too aggressively on a workload that fires a query every couple of minutes and you’ll pay more in cold-start minimums than you ever saved on idle time. Aggressive suspend is not automatically cheaper.

    One write nukes the whole result cache. The cache is invalidated by any change to the underlying table, not just changes to the rows your query touched. A single late-arriving record at 2 a.m. quietly wipes the cached result for every dashboard query built on that table, and tomorrow morning they all run on compute again. If your “cached” dashboard mysteriously costs money some mornings, this is usually why.

    Cloud services is free, right up until it isn’t. The parsing, planning, and pruning in the cloud services layer is free — but only while it stays under 10% of your daily warehouse compute. Hammer it with thousands of tiny metadata queries or relentless INFORMATION_SCHEMA polling and you cross that line, and Snowflake starts billing the overage. Most teams never hit it; the ones running huge volumes of trivial queries do, and they never see it coming.

    The local cache doesn’t follow you between warehouses. That warm SSD cache belongs to one specific warehouse. Run your ETL on one warehouse and your reporting on another — sensible workload isolation — and the reporting warehouse gets exactly zero benefit from the partitions ETL just pulled. Each warehouse warms its own cache from cold.

    SELECT * throws away the columnar advantage. Snowflake only reads the columns you ask for. Ask for all of them and you pay to read all of them, even when the dashboard displays three. In a wide table that’s the difference between scanning a few columns and dragging the entire row off storage.

    The mistakes that quietly drain the budget

    Almost every overspending Snowflake account I’ve looked at is making some combination of these five:

    1. Leaving AUTO_SUSPEND at the 600-second default. Ten full minutes of paid idle after every burst of activity. On a warehouse that’s used in short bursts, that idle time can dwarf the actual query time. Most analytics warehouses are fine at 60 seconds.

    2. One giant warehouse for everything. Pile ETL, ad-hoc analysis, and dashboards onto a single warehouse and they fight over the cache and the compute. Separate warehouses per workload keep each cache warm and each workload predictable.

    3. Scaling up when the real fix is pruning. The reflex we opened with. If pct_scanned is high, a bigger warehouse just burns more credits reading the same data. Fix the layout, not the size.

    4. Letting the BI tool emit sloppy SQL. Inconsistent aliases, injected timestamps, and per-user comments mean “the same” dashboard query is never byte-for-byte identical, so it never reuses the result cache. Standardize what the tool sends.

    5. Over-clustering a hot table. Automatic clustering isn’t free — it spends credits reorganizing data in the background. On a table that’s written constantly, that background churn can cost more than the queries it speeds up. Cluster the tables you read far more than you write.

    The one principle to take away

    If you remember nothing else, remember this: scan less, don’t compute faster. Snowflake decides almost everything about a query’s speed and cost before the warehouse ever wakes up — in the pruning, in the caches, in how your data is laid out. The warehouse size is the last lever you should reach for, not the first. Get the layers above it working and most “we need a bigger warehouse” conversations simply stop happening.

    Related reading: Snowflake Time Travel: The Real Architecture · Snowflake Optima: 15x Faster Queries at Zero Cost · Query Snowflake in DuckDB and Cut Costs · Snowflake Streams & Tasks: SCD2 Pipeline Guide

  • How the Warehouse Cache Actually Works in Snowflake

    How the Warehouse Cache Actually Works in Snowflake

    A dashboard that ran in four seconds on Monday took nineteen seconds on Tuesday. Same query, same data, same warehouse size. I spent the better part of an hour convinced Snowflake was having a bad day, checking the status page, refreshing query history, muttering about “platform issues” in our team Slack — before I noticed our DevOps script had quietly added an aggressive auto-suspend policy the night before. The warehouse cache was getting wiped every single morning, and I’d built our entire “fast dashboard” reputation on a cache that reset itself before anyone showed up to work.

    TL;DR

    • Snowflake’s warehouse cache (local disk cache) stores raw compressed micro-partition data on each node’s SSD, not query results.
    • It’s separate from the result cache and metadata cache — three different caches, three different jobs.
    • Auto-suspending a warehouse wipes this cache completely. Resuming starts it cold every time.
    • You can verify it’s working with the ‘percentage scanned from cache’ field in query profile or ACCOUNT_USAGE.QUERY_HISTORY.
    • Multi-cluster warehouses don’t share this cache — a query routed to a new cluster starts cold even if a sibling cluster is warm.
    • You can’t manually size or pin it. The only real lever you control is the auto-suspend timer.

    That mistake is what got me actually reading how this thing works instead of just trusting that Snowflake would “handle it.” Turns out the warehouse cache isn’t magic, isn’t tunable, and isn’t the same thing as the result cache most people learn about first. Here’s what it actually is, where it lives, and how to tell when it’s helping you versus quietly costing you money.

    Diagram showing a cloud data warehouse architecture with three layers: Cloud services (result and metadata cache), compute layer with three SSD nodes, and a database storage layer with immutable micro-partitions. Arrows indicate data flow.

    Three caches, one name people use for all of them

    Snowflake has three distinct caching layers, and the confusion starts because people say “Snowflake caches my query” without specifying which one did the work. The result cache sits in the cloud services layer and stores entire finished query results — if the same exact SQL text runs again within 24 hours and the underlying data hasn’t changed, you get the answer back with bytes_scanned = 0 and zero compute cost. The metadata cache, also in cloud services, holds statistics about every micro-partition — min and max values per column, row counts — so Snowflake can decide which partitions to skip before it ever touches the data.

    The warehouse cache is the third one, and it’s the one this article is actually about. It lives on the local SSD of every node in a running virtual warehouse, and it stores raw, compressed micro-partition data — not query results, not aggregated answers, the actual columnar bytes that got pulled from remote storage to answer a scan.

    Why this distinction actually matters

    If you only know the result cache exists, you’ll misdiagnose a lot of performance issues. Change one character in a WHERE clause, add a comment, swap the role running the query — any of those bypass the result cache entirely, because it requires an exact text match. The warehouse cache doesn’t care about query text at all. It cares about which micro-partitions a query needs and whether those bytes are already sitting on a node’s SSD from a previous scan.

    What a micro-partition actually is

    You can’t understand the warehouse cache without understanding the unit it stores. When data lands in a Snowflake table, it gets automatically carved into micro-partitions — contiguous, immutable blocks holding somewhere between 50MB and 500MB of uncompressed data each, stored in a columnar format. There’s no manual partitioning scheme to design, no index to build. Snowflake just does this on every load.

    Each micro-partition carries its own metadata: minimum and maximum values for every column, which is exactly what the metadata cache is built from. When you filter a query on order_date > '2026-01-01', Snowflake checks that metadata first and skips any micro-partition whose max date falls before that threshold. That’s partition pruning, and it happens before a single byte gets pulled into the warehouse cache. Pruning decides what to read; the warehouse cache decides how fast a repeat read of the same partitions will be.

    The actual lifecycle of the warehouse cache

    Here’s the sequence that matters in practice. A warehouse resumes from suspended state with completely empty SSD — there’s nothing cached because the compute nodes assigned to it are freshly provisioned. The first query that touches a table has to pull every relevant micro-partition from remote cloud storage, which is the slowest tier in the whole architecture. As those partitions get read, they’re written to the local SSD cache as a side effect — not because you asked for caching, just because that’s what happens when a node reads remote data.

    The second query — if it touches the same micro-partitions and the warehouse is still running — can read those bytes off local SSD instead of going back to remote storage. This is meaningfully faster, and it’s why a sequence of similar queries against the same table speeds up the longer a warehouse stays warm. There’s no explicit “build the cache” step. Cache population is a byproduct of usage, which is exactly why a single cold query tells you almost nothing about real-world performance.

    Then the warehouse suspends, and it’s gone

    This is the part that bit me. When a warehouse auto-suspends, the compute nodes it was using get released back into Snowflake’s shared pool. The SSD on those specific nodes goes with them. When the warehouse resumes — even if it’s seconds later, even if it’s the exact same warehouse name — there’s no guarantee you get the same physical nodes back, and the cache starts from zero regardless. There’s no persistence, no “save state before suspending.” It’s just gone.

    Resizing a warehouse up or down does the same thing. A different size means a different set of nodes, which means different SSDs, which means the next queries run cold no matter how warm things were five minutes earlier.

    How to actually see this working

    Stop assuming and go look at it. Open the query profile for any query in Snowsight and check the IO statistics panel — there’s a field literally called percentage_scanned_from_cache. Run the same query twice in a row on a warm warehouse and watch that number jump from near 0% to something much higher on the second run. Suspend the warehouse, resume it, run the same query again, and watch it drop back to 0%. That’s the entire mechanism, visible in about ninety seconds of testing.

    For a wider view across your account, query history gives you the same field at scale. This is the query I run when someone asks “is our caching even helping”:

    Check your real cache hit rate (last 30 days, by warehouse)

    SELECT
        warehouse_name,
        COUNT(*) AS query_count,
        SUM(bytes_scanned) AS bytes_scanned,
        SUM(bytes_scanned * percentage_scanned_from_cache) AS bytes_from_cache,
        SUM(bytes_scanned * percentage_scanned_from_cache)
            / SUM(bytes_scanned) AS pct_scanned_from_cache
    FROM snowflake.account_usage.query_history
    WHERE start_time >= DATEADD(month, -1, CURRENT_TIMESTAMP())
      AND bytes_scanned > 0
    GROUP BY 1
    ORDER BY 5;

    A low percentage here on a warehouse running frequent, similar queries is a signal — either your auto-suspend timer is too aggressive for the workload, or the queries aren’t actually similar enough at the data level to benefit from a warm cache, even if the SQL looks similar to a human reading it.

    Result cache, warehouse cache, metadata cache — side by side

    Cache LayerWhere It LivesWhat It StoresCleared WhenCompute Cost
    Result CacheCloud Services layerFull query results24 hours of inactivity, or DDL on underlying tablesZero — no warehouse needed
    Warehouse CacheSSD on each compute nodeRaw compressed micro-partitionsWarehouse suspends, resizes, or node is replacedWarehouse must be running
    Metadata CacheCloud Services layerMin/max values, row counts, partition statsRarely — persists with the tableZero — used for pruning before scan

    The auto-suspend tradeoff nobody explains clearly

    Snowflake’s own guidance generally points toward short auto-suspend windows to control credit spend, and for spiky, unpredictable workloads that’s the right call. But if a warehouse runs frequent, similar queries back-to-back — a BI tool polling dashboards, an analyst iterating on the same fact table — an aggressive suspend timer means you’re paying the “cold scan” tax on nearly every query, because the cache never gets the chance to stay warm between them.

    The fix isn’t complicated once you see the tradeoff: separate warehouses by access pattern. A reporting warehouse that gets hit constantly during business hours can run a longer suspend window, or stay up during known peak hours, while a warehouse running sporadic ad-hoc analyst queries can suspend aggressively without losing much, since the cache wasn’t going to be useful between unrelated queries anyway.

    The multi-cluster gotcha

    If you’re running a multi-cluster warehouse for concurrency, know that clusters don’t share this cache with each other. Cluster A being fully warm doesn’t help a query that gets routed to Cluster B when Snowflake spins up a new cluster to handle a concurrency spike. That new cluster starts cold, scans from remote storage, and only builds its own local cache from that point forward. Teams chasing consistent query latency under high concurrency often get surprised by this — the warehouse “should” be warm, and on average it is, but any individual query can still land on a cold cluster.

    What the warehouse cache does not do

    It’s worth being precise about the boundaries here, because I’ve seen this cache get credited for things it isn’t responsible for. It doesn’t store query results — that’s the result cache’s job, and it’s a different layer entirely with a different lifetime. It doesn’t help with intermediate computation that spills to local disk during a large sort or hash join — that’s a separate spillage mechanism tracked under bytes spilled to local storage in query profile, not the same SSD allocation conceptually even though it physically lives in a similar place. And it provides no benefit on the write path for a fresh INSERT into new micro-partitions, since there’s nothing previously cached to reuse.

    What’s actually worth doing about this

    You don’t get a dial to resize this cache or pin specific tables into it, so the practical levers are all about behavior, not configuration. Match auto-suspend timers to actual access patterns instead of using one default across every warehouse. If a workload is genuinely cache-sensitive — recurring dashboards, iterative analyst sessions — consider a short warm-up query immediately after resume rather than letting the first real user query eat the cold-start cost. And when you’re debugging a “why did this get slower” ticket, percentage_scanned_from_cache should be one of the first three things you check, right alongside partition pruning stats, before you start blaming the query itself.

    For the deeper mechanics of how partition pruning interacts with clustering keys, the official Snowflake documentation on warehouse cache optimization is worth reading directly — it’s one of the rare vendor docs pages that actually shows the diagnostic query instead of just describing the concept. The micro-partitions and clustering documentation is the right follow-up if pruning efficiency turns out to be your actual bottleneck instead of cache temperature.

    Does the Snowflake warehouse cache get cleared when the warehouse suspends?

    Yes, completely. The warehouse cache lives on the SSD of the compute nodes assigned to that warehouse, and those nodes get released back to the pool on suspend. When the warehouse resumes — even seconds later — it’s starting from zero cached data.

    Why does percentage_scanned_from_cache show 0% on a brand new warehouse?

    Because there’s nothing to scan yet. The first query against any table after a cold start has to pull every micro-partition it needs from remote storage. Cache population happens as a side effect of running queries, not in advance.

    Does the warehouse cache help with INSERT, UPDATE, or DELETE performance?

    📷 the SSD cache layer, mid-rebuild after a warehouse resume — not glamorous, but it’s where the speed comes from