Category: Snowflake

Dive deep into the Snowflake Data Cloud. Guides on building a modern cloud data warehouse, data sharing, performance optimization, and leveraging advanced features like Snowpipe and Streams.

  • The Hidden Architecture Behind Snowflake Time Travel: Why It’s Not Really a Backup Feature

    The Hidden Architecture Behind Snowflake Time Travel: Why It’s Not Really a Backup Feature

    TL;DR

    → Time Travel is not a backup — it’s a versioned metadata pointer to immutable micro-partitions you already paid to store
    → Snowflake never overwrites data in place. Every UPDATE or DELETE creates new micro-partitions and marks old ones as expired
    → Standard edition gives you 1 day. Enterprise gives up to 90 days — but storage costs multiply fast on high-churn tables
    → After Time Travel expires, data moves to Fail-safe for 7 more days — but only Snowflake support can retrieve it
    → Zero-copy clones use the same micro-partition pointers — no extra storage until you diverge from the source
    → High-churn tables on 90-day retention can silently balloon your Snowflake bill by 10x


    The misconception that costs people money

    Most engineers who discover Time Travel think: “Great, we have backups.” That’s the wrong mental model — and it’s the one that leads to both security gaps and surprise storage bills. Time Travel is not a backup. It’s a metadata feature built on top of something Snowflake was already doing.

    Understanding why requires understanding how Snowflake actually stores data under the hood.

    How Snowflake stores data: micro-partitions

    Snowflake doesn’t store your tables as traditional database files. It stores them as micro-partitions — small, immutable, columnar files in cloud object storage (S3, Azure Blob, GCS), typically 50–500MB compressed each.

    The word immutable is the key. Snowflake never modifies a micro-partition once it’s written. Every micro-partition is a read-only snapshot of the data at the moment it was created. So what happens when you UPDATE a row? Snowflake writes a new micro-partition with the updated data and marks the old one as expired.

    Diagram showing data partitions before and after an update: before, partitions A and B are active; after, A is expired (for Time Travel), A* contains the updated row, and B remains active and unchanged.

    The old data doesn’t go anywhere immediately — it just gets a metadata flag saying ‘this version is no longer current.’ This is Copy-on-Write, and it’s the architectural foundation that makes Time Travel possible essentially for free.

    Time Travel isn’t a feature Snowflake built on top of backups. It’s a feature Snowflake built on top of an immutable storage model they were already using. The retained partitions are a side effect of how writes work — Time Travel just decides how long to keep them.

    What Time Travel actually is

    Time Travel is Snowflake’s metadata layer keeping pointers to those expired micro-partitions, instead of immediately flagging them for deletion. When you query with AT(TIMESTAMP => ...) or BEFORE, you’re not restoring from a backup. You’re asking Snowflake’s metadata layer to temporarily re-point to the expired partitions. The data was always there — you’re just re-routing the query to read older versions.

    This is why Time Travel queries are fast. There’s no restore process. No data movement. Snowflake reads directly from the older partitions.

    The three-zone model: Active, Time Travel, Fail-safe

    Understanding the full picture requires knowing all three zones data passes through after it’s written and then changed.

    A flowchart illustrates the three-zone data lifecycle: Active (current data, query anytime), Time Travel (1–90 days, billed, SQL access), Fail-safe (7 days, support only, Snowflake cost), and Gone (permanent, no recovery).

    Active data is what your current queries see — the live micro-partitions. Time Travel holds expired micro-partitions for your configured retention window. You can query this with SQL, clone from it, and UNDROP tables dropped within the window. Fail-safe activates when Time Travel expires — Snowflake keeps those partitions for 7 more days, but only Snowflake support can retrieve them. After that, data is permanently gone.

    Time Travel vs Fail-safe — the comparison you need

    FeatureTime TravelFail-safe
    Duration0–90 days (edition dependent)7 days (fixed, non-configurable)
    Who can accessYou — via SQL queriesSnowflake support only
    Query directlyYes — AT / BEFORE syntaxNo — support ticket required
    Clone fromYes — zero-copy clonesNo
    Storage costYes — counts against your billNo additional charge
    ConfigurableYes — per table/schema/databaseNo — always 7 days
    Best forOperational recovery, auditingLast-resort disaster recovery

    The storage cost nobody warns you about

    Every expired micro-partition kept for Time Travel counts against your Snowflake storage bill. The formula is brutal: a table with 90-day retention that sees 100% of its rows updated daily is storing 91 versions of itself simultaneously.

    Bar chart comparing storage multipliers for low churn (green) and high churn (orange) data over different retention periods (1, 7, 30, 90 days), showing high churn increases storage costs, especially at 90 days (30x).

    Most teams set 90-day retention on everything because the docs say Enterprise supports up to 90 days and more seems better. Then they get their first monthly storage invoice and start asking questions.

    ⚠️ The fix for high-churn tables: Set DATA_RETENTION_TIME_IN_DAYS = 0 on transient staging tables, session event tables, or any table where Time Travel has no operational value. You lose time travel on those tables, but you stop paying for micro-partitions you’ll never query.

    Zero-copy clones: same architecture, surprising implications

    Zero-copy clones work through the same micro-partition pointer mechanism. When you CREATE TABLE clone CLONE source, Snowflake doesn’t copy any data. It creates a new table object whose metadata points to the same micro-partitions as the source. Storage only diverges when you write new data to either the source or the clone.

    This is why ‘create a clone before a dangerous operation’ is nearly free — until you start modifying the clone. It’s also why clones on Time Travel windows are powerful: you can clone a table as it existed 7 days ago with zero storage cost at creation time.

    Why Time Travel is not a backup

    Account-level events affect everything. If your Snowflake account is compromised at the account level, or you accidentally drop the entire database, Time Travel data is in the same account. It’s not in a separate system.

    Cloud storage failure. Time Travel data lives in the same cloud storage as your active data. A regional disaster that takes out your Snowflake data takes out Time Travel with it.

    It expires. A backup you can restore from in 6 months is a backup. Time Travel data that’s gone after 90 days is a version history, not a backup. For genuine disaster recovery, you need cross-region replication or dedicated exports.

    Practical SQL patterns


    Here are the patterns I use most in production — from basic time travel queries to monitoring which tables are inflating your storage bill:

    -- Query a table as it existed yesterday
    SELECT * FROM orders
      AT(TIMESTAMP => DATEADD(DAY, -1, CURRENT_TIMESTAMP()));
    
    -- Query using a specific offset in seconds
    SELECT * FROM orders
      AT(OFFSET => -3600);  -- 1 hour ago
    
    -- Restore a dropped table
    UNDROP TABLE orders;
    
    -- Clone a table from 7 days ago (zero-copy, no extra storage)
    CREATE TABLE orders_snapshot
      CLONE orders
      AT(TIMESTAMP => DATEADD(DAY, -7, CURRENT_TIMESTAMP()));
    
    -- Check Time Travel storage usage by table
    SELECT table_name,
           active_bytes / 1e9         AS active_gb,
           time_travel_bytes / 1e9    AS time_travel_gb,
           failsafe_bytes / 1e9       AS failsafe_gb
    FROM information_schema.table_storage_metrics
    WHERE time_travel_bytes > 0
    ORDER BY time_travel_bytes DESC;
    
    -- Set retention to 0 for high-churn tables you don't need to travel
    ALTER TABLE session_events
      SET DATA_RETENTION_TIME_IN_DAYS = 0;

    Frequently Asked Questions

    Q: How does Snowflake Time Travel actually work?
    A: Time Travel works by retaining expired micro-partitions rather than deleting them. When you run UPDATE or DELETE, Snowflake writes new micro-partitions and marks old ones as expired but keeps them for your retention window. When you query with AT or BEFORE, Snowflake re-points to those expired partitions at the metadata level. No data is copied or moved — it’s a metadata operation.

    Q: Is Snowflake Time Travel the same as a backup?
    A: No. Time Travel is not a backup. It’s access to older versions of data in the same system. If your Snowflake account is deleted, compromised at account level, or if cloud storage fails, Time Travel disappears with it. For true disaster recovery you need cross-region replication or separate exports.

    Q: How long does Snowflake Time Travel last?
    A: Standard edition: maximum 1 day. Enterprise and higher: up to 90 days, configurable per table, schema, or database. Transient and temporary tables max out at 1 day regardless of edition.

    Q: What happens after Time Travel expires?
    A: Expired micro-partitions move to Fail-safe — a non-configurable 7-day window managed by Snowflake. You cannot query Fail-safe data yourself. Only Snowflake support can recover it, and recovery is not guaranteed. After Fail-safe expires, data is permanently deleted.

    Q: Does Time Travel affect storage costs?
    A: Yes, significantly. Every expired micro-partition counts toward your storage bill. High-churn tables on 90-day retention can cost 10x more storage than the active data alone. Set DATA_RETENTION_TIME_IN_DAYS = 0 on staging tables or high-churn tables where Time Travel has no operational value.

    Q: What’s the difference between Time Travel and Fail-safe?
    A: Time Travel is user-controlled — you query it with SQL, configure its duration, and clone from it. Fail-safe is Snowflake-controlled — only support can access it, it’s always exactly 7 days, and it exists for Snowflake’s disaster recovery, not yours.

  • Snowflake CoCo Desktop — What It Is, How It Works, and Whether It’s Worth It

    Snowflake CoCo Desktop — What It Is, How It Works, and Whether It’s Worth It

    Snowflake just announced a lot at Summit 2026. Most of it was the usual conference noise. CoCo Desktop isn’t.

    I’ve been following Cortex Code — now officially rebranded as CoCo — since it first shipped in Snowsight. The Summit 2026 announcement on June 2 changed the scope significantly. A native desktop IDE, Cloud Agents that run async without keeping your machine on, a Slackbot, mobile app, and integrations with VS Code, Excel, and Claude Code. That’s not a feature update. That’s a platform play.

    Here’s my honest take on what CoCo Desktop actually is, how it compares to what you’re probably already using, and whether data engineers should care.


    TL;DR

    → Snowflake CoCo is the official rebrand of Cortex Code — same product, bigger vision, launched at Summit 2026 on June 2
    → CoCo Desktop is a native IDE that reads your Snowflake schemas, RBAC policies, and lineage before generating any code
    → It scored 72.1% on dbt’s ADE-Bench vs 65.1% for Claude Code — but benchmarks and production are different things
    → New at Summit: Cloud Agents run tasks async in Snowflake’s cloud, Automations handle recurring workflows, Skill Catalog shares reusable flows
    → It integrates with VS Code, Slack, Excel, and Claude Code — so you don’t have to abandon your existing tools
    → Worth evaluating if your team lives in Snowflake — not worth migrating to if you’re happy with Claude Code or Cursor


    What CoCo Actually Is (And What Changed at Summit 2026)

    CoCo (formerly Cortex Code) is Snowflake’s data-native AI coding agent. The key word is data-native. Unlike general-purpose coding assistants, CoCo reads your live Snowflake environment — schemas, RBAC policies, lineage — before generating anything. It doesn’t generate SQL and hope it matches your tables. It knows your tables.

    That’s been true since Cortex Code. What changed at Summit 2026:

    CoCo Desktop — a native desktop IDE, not just a Snowsight panel. Full agentic development, local environment, MCP integrations.

    Cloud Agents — launch tasks that run async in Snowflake’s cloud. Your machine doesn’t need to stay on. This is the biggest shift.

    Automations — recurring, event-driven workflows. CoCo monitors, validates, and triggers without manual intervention.

    Skill Catalog — teams can share and reuse proven CoCo workflows. Institutional knowledge, codified.

    The rebranding from Cortex Code to CoCo also signals something: Snowflake is positioning this as a platform, not a feature. The Cortex branding was burying it inside a product umbrella. CoCo gets its own identity, its own downloads page, and its own development roadmap.


    The Data-Native Context Advantage

    This is where CoCo genuinely differentiates from Claude Code, Cursor, or GitHub Copilot.

    Every general-purpose coding assistant has the same fundamental problem with data engineering work: it doesn’t know your data. You ask it to write a dbt model and it writes syntactically valid SQL against tables that may or may not exist in your warehouse. You have to paste schema context, explain relationships, describe RBAC constraints. It’s friction.

    CoCo eliminates that friction for Snowflake environments. Before generating anything, it reads your live schema, understands which tables your role can access, and traces lineage to understand dependencies. The output is grounded in your actual environment.

    Here’s a real example of the difference — a natural language prompt to CoCo and what it generates knowing your schema:

    CoCo Desktop — Schema-Grounded dbt Model Generation

    -- Example: What you say to CoCo Desktop
    "Create an incremental dbt model that calculates 
    7-day rolling revenue by customer segment, 
    partitioned by signup_date"
    
    -- What CoCo generates (reads your actual schema first):
    {{ config(
        materialized='incremental',
        unique_key='customer_segment || date_trunc(day, event_date)',
        on_schema_change='sync_all_columns'
    ) }}
    
    WITH revenue_base AS (
        SELECT
            c.segment                          AS customer_segment,
            DATE_TRUNC('DAY', o.created_at)    AS event_date,
            SUM(o.revenue_usd)                 AS daily_revenue
        FROM {{ ref('orders') }} o
        JOIN {{ ref('customers') }} c ON o.customer_id = c.id
        {% if is_incremental() %}
        WHERE o.created_at >= (
            SELECT DATEADD(DAY, -8, MAX(event_date)) FROM {{ this }}
        )
        {% endif %}
        GROUP BY 1, 2
    )
    SELECT
        customer_segment,
        event_date,
        daily_revenue,
        AVG(daily_revenue) OVER (
            PARTITION BY customer_segment
            ORDER BY event_date
            ROWS BETWEEN 6 PRECEDING AND CURRENT ROW
        ) AS rolling_7d_revenue
    FROM revenue_base

    The output references your actual table names, your actual columns, and respects your actual partitioning strategy — because CoCo read your schema before writing a single line.

    Cloud Agents — Async Scheduled Tasks

    The second genuinely new capability is Cloud Agents. Here’s what that looks like in practice:

    -- Cloud Agent example: schedule a validation job
    -- that runs without keeping your laptop open
    
    -- In CoCo Desktop → New Cloud Agent:
    {
      "name": "daily_revenue_validation",
      "trigger": "schedule",
      "cron": "0 6 * * *",
      "task": "Run data quality checks on orders table,
               flag anomalies > 2 std deviations,
               post summary to #data-alerts Slack channel",
      "context": ["orders", "customers", "revenue_daily"],
      "on_failure": "notify_slack"
    }
    
    -- CoCo generates, schedules, and monitors this
    -- entirely within Snowflake's governed environment

    This is the shift from coding assistant to autonomous agent. CoCo doesn’t just help you write the job — it runs the job, in Snowflake’s governed environment, on a schedule, and reports back.


    The Benchmark Reality Check

    Snowflake claims CoCo scored 72.1% on dbt’s ADE-Bench versus 65.1% for Claude Code. That’s a real benchmark on real analytics engineering tasks — 145 queries, statistically significant.

    I want to be honest about what this means and doesn’t mean.

    It means CoCo is genuinely better at Snowflake-specific SQL and dbt model generation than Claude Code in a controlled evaluation. That’s not surprising — CoCo has live schema context and was purpose-built for this use case.

    It doesn’t mean CoCo is better for all the work you actually do. ADE-Bench measures analytics engineering tasks specifically. It doesn’t measure debugging Python pipeline errors, writing Airflow DAGs, reviewing infrastructure-as-code, or any of the other things Claude Code or Cursor handle in a typical data engineering workday.

    If 80% of your coding work is Snowflake SQL and dbt models, CoCo’s benchmark advantage is real and production-relevant. If you’re a generalist data engineer working across multiple systems, that 7-point advantage on analytics SQL is a smaller part of your actual workflow.


    Where CoCo Desktop Has Limits

    No offline mode. CoCo Desktop requires a Snowflake account connection. If you’re working without internet access or in an environment where outbound connections are restricted, it doesn’t work.

    Snowflake-only context. CoCo understands your Snowflake environment deeply. It doesn’t understand your Postgres database, your Kafka topics, or your Airflow DAG structure unless you give it that context manually — at which point you’ve lost the data-native advantage.

    Token-based pricing. Cloud Agents and Automations consume Snowflake credits. For high-frequency automation workflows, the cost model needs evaluation before you commit. This is a brand new product — pricing behaviour at scale is unknown.

    MCP ecosystem is smaller than Claude Code’s. CoCo supports GitHub, Jira, Google Workspace via MCP. Claude Code’s MCP ecosystem is broader. If your workflow relies on specific MCP integrations, check the current list before assuming coverage.


    The Comparison You Actually Need

    FeatureCoCo DesktopClaude Code / Cursor
    Data contextReads live Snowflake schema, RBAC, lineage automaticallyNo native warehouse context — you provide manually
    SQL generation72.1% ADE-Bench — purpose-built for analytics SQL65.1% ADE-Bench — strong general coding
    dbt supportNative — reads dbt project structure and modelsGood — but no automatic schema grounding
    Pipeline authoringSnowflake-native — Snowpark, Streams, TasksGeneral Python — works but no Snowflake operators
    Cloud AgentsRun tasks async in Snowflake cloudLocal execution only
    MCP integrationsGitHub, Jira, Google WorkspaceBroader third-party connector ecosystem
    Slack / mobileSlackbot and mobile app coming soonNo native Slack or mobile interface
    GovernanceRBAC-aware — won’t violate access policiesNo governance layer — manual enforcement
    Best forTeams fully on SnowflakeGeneral data engineering, polyglot stacks

    How I’d Actually Use This

    I wouldn’t replace Claude Code with CoCo. I’d use them for different things.

    CoCo Desktop for: writing dbt models, generating Snowpark pipelines, setting up Cloud Agents for recurring validation jobs, anything where Snowflake schema context is the difference between useful output and generic SQL.

    Claude Code for: debugging Python pipeline errors, writing Airflow DAGs, reviewing infrastructure code, cross-system work, anything outside the Snowflake context boundary.

    The Skill Catalog is the feature I’m most interested in practically. Codifying proven CoCo workflows — a data quality check pattern, a standard incremental model template, a Snowflake Stream processing pattern — and sharing them across the team is where the real leverage is. That’s institutional knowledge made reusable. I wrote about a similar pattern in Delta Lake vs Iceberg — the tools that win long-term are the ones that compound team knowledge, not just individual productivity.


    When to Evaluate CoCo Desktop

    Your team is primarily on Snowflake. If 70%+ of your data work is in Snowflake, CoCo’s context advantage is real and compounding. The time saved not pasting schema context into Claude Code adds up fast.

    You need governed AI development. CoCo’s RBAC awareness means it won’t generate queries that violate access policies. For compliance-heavy environments, that’s not a nice-to-have.

    You want async agentic workflows. Cloud Agents are genuinely new. If you want to describe a monitoring job in natural language and have it run on a schedule without babysitting it, CoCo is currently the only tool that does this inside a governed Snowflake environment.

    When to Stick With What You Have

    You’re on a polyglot stack. Snowflake is one of several systems. CoCo’s advantage disappears outside the Snowflake context boundary.

    You’re happy with Claude Code or Cursor. The 7-point ADE-Bench gap doesn’t justify a tool switch if your current workflow is working and your team is productive.

    You want to wait for GA. CoCo Desktop is very new — announced June 2, 2026. Production edge cases, pricing at scale, and Cloud Agent reliability are unknown quantities. Evaluating in staging is smart. Full production adoption before GA carries risk.


    What I’d Do Right Now

    Download CoCo Desktop and run it against one real project — ideally a dbt model you’ve been meaning to refactor or a validation job you’ve been doing manually. That’s the fastest way to evaluate whether the schema-grounding advantage is worth it for your specific workflow.

    Don’t make a team-wide decision based on benchmarks alone. ADE-Bench is a good signal, but your specific schema complexity, RBAC structure, and workflow patterns will determine whether the 7-point advantage is meaningful in practice.

    Watch the Cloud Agents closely. That’s where the real competitive moat is if Snowflake executes. An AI agent that runs governed, async, schema-aware tasks without manual intervention is a different category from a coding assistant.


    Frequently Asked Questions

    Q: What is Snowflake CoCo Desktop?
    A: CoCo Desktop is a native desktop IDE from Snowflake that connects directly to your Snowflake account and uses AI to generate SQL, dbt models, and pipelines from natural language. It reads your live schema, RBAC policies, and data lineage before generating any code — meaning it understands your actual data environment, not just generic SQL syntax. It was announced at Snowflake Summit 2026 on June 2 as the rebrand of Cortex Code.

    Q: Is Snowflake CoCo the same as Cortex Code?
    A: Yes. CoCo is the official rebrand of Cortex Code, announced at Snowflake Summit 2026. The product functionality and architecture are the same — the rename reflects Snowflake’s broader vision for AI-powered development. If you were using Cortex Code, nothing changes in your existing workflows.

    Q: How does CoCo Desktop compare to Claude Code?
    A: CoCo scored 72.1% on dbt’s ADE-Bench versus 65.1% for Claude Code on analytics engineering tasks — but the more important difference is context. CoCo reads your Snowflake schema, RBAC, and lineage automatically. Claude Code needs you to provide that context manually. For teams fully on Snowflake, CoCo’s data-native context is a real advantage. For polyglot stacks or non-Snowflake work, Claude Code is still stronger.

    Q: What are CoCo Cloud Agents?
    A: Cloud Agents let you launch tasks in Snowsight that run async in Snowflake’s cloud — without your laptop staying open. You describe the task in natural language, CoCo generates and schedules it, and it runs in a governed Snowflake environment. This is the key difference from a coding assistant — Cloud Agents turn CoCo into an autonomous development platform, not just an autocomplete tool.

    Q: What tools does CoCo Desktop integrate with?
    A: CoCo integrates with VS Code, Slack (Slackbot, with mobile app coming soon), Microsoft Excel, and Anthropic’s Claude Code via MCP. It also supports MCP servers for GitHub, Jira, and Google Workspace. You don’t have to abandon your existing tools — CoCo is designed to work alongside them.

    Q: Is CoCo Desktop free?
    A: CoCo Desktop requires a Snowflake account with Cortex Code enabled and is billed based on token consumption. Snowflake offers trial access with free credits for new users. Costs depend on query volume and token usage — check Snowflake’s pricing page for the latest details since this launched at Summit 2026 in June.

  • Delta Lake vs Apache Iceberg — Why I Chose Iceberg for Our Data Lakehouse

    Delta Lake vs Apache Iceberg — Why I Chose Iceberg for Our Data Lakehouse

    TL;DR
    → Delta Lake is easier to start with, especially if you’re already on Databricks
    → Iceberg wins on engine flexibility — works natively with Spark, Flink, Trino, Snowflake, and more without custom connectors
    → Delta Lake’s vendor coupling with Databricks is a real cost if you’re multi-cloud or multi-engine
    → Iceberg’s partition evolution lets you change partition schemes without rewriting data — that feature alone saved us a full weekend of migration work
    → Migration from Delta to Iceberg is harder than most blog posts suggest — budget four to eight weeks, not a weekend
    → If you’re greenfield, start with Iceberg. If Delta is working, don’t migrate until you hit a specific limit


    I didn’t choose Iceberg because I read a benchmark blog post. I chose it after six months of hitting Delta Lake’s limits in ways that weren’t obvious until they were expensive.

    We were running a mid-sized data lakehouse — S3-backed, Spark for processing, Snowflake for consumption, dbt for transformation. Delta Lake was the default choice. Everyone on the team had used it before. The documentation was solid. It worked — until it didn’t.

    This isn’t a “here are the specs” comparison. You can get that from the docs. This is what actually happened when I ran both in production, why I made the switch, and what I’d tell you before you pick one.


    What We Were Actually Trying to Solve

    Before I get into the comparison, context matters. Our stack at the time: raw data landing in S3, Apache Spark for heavy transformation, Snowflake as the consumption layer for analysts, dbt for modeling, and Apache Airflow for orchestration.

    We needed ACID transactions on S3, time travel for debugging, and the ability to do incremental loads without full partition rewrites. Delta Lake checked all those boxes — initially. The problems showed up at scale and at the edges.


    Where Delta Lake Started Hurting Us

    Engine Lock-In Was a Real Problem

    Delta Lake works great if Spark is your only compute engine. The moment we tried to query Delta tables directly from Snowflake or Trino, things got complicated. Delta’s transaction log format is proprietary. You need the Delta connector — and not every engine has a first-class one.

    We wanted analysts to query raw lakehouse tables directly from Snowflake without going through Spark first. With Delta, that required Snowflake’s Delta Sharing integration, which had limitations on what operations were supported. It wasn’t broken, but it added friction and another dependency to manage.

    Apache Iceberg solves this cleanly. The table format is open. Snowflake, Spark, Flink, Trino, Athena, Dremio — they all read and write Iceberg natively. No connectors to manage. No format translation layer.

    Partition Management Was Getting Messy

    With Delta Lake, partitioning decisions are set at table creation. Changing a partition scheme means rewriting the table. At 100M+ rows, that’s not a quick operation.

    We had a table partitioned by event_date. Six months in, query patterns changed — analysts were filtering by event_date and region together. Repartitioning meant a full backfill job over a weekend, plus repointing all downstream dbt models.I wrote about a similar pain point in the problem with dbt incremental models — the pattern is the same.

    Iceberg’s partition evolution lets you change the partition spec without rewriting data. Old data stays as-is. New data uses the new scheme. Queries still work against both.

    Hidden Partitioning Changed How We Design Tables

    Iceberg supports hidden partitioning — you define partition transforms like days(event_timestamp) or bucket(user_id, 16) and Iceberg handles physical partitioning transparently. Your queries don’t need to know about partition columns. The engine prunes automatically.

    With Delta Lake, you need to explicitly filter on partition columns or you’ll scan everything. That’s fine when everyone knows the rules. It’s a problem when a new analyst writes a query without knowing which columns are partition keys.


    Where Delta Lake Is Still Better

    If you’re on Databricks, stay on Delta. The integration is tight, the tooling is mature, and Databricks has invested heavily in Delta’s performance.. Liquid Clustering makes partition management much more flexible. If Databricks is your primary compute layer, switching to Iceberg gives you marginal benefit for non-trivial migration cost.

    Delta’s MERGE performance on Spark is excellent. For high-frequency CDC workloads where you’re doing upserts at scale on Spark, Delta’s MERGE implementation is well-optimised. Iceberg’s MERGE has improved significantly but Delta still has an edge in some Spark-specific CDC patterns.

    Delta has simpler operational overhead for small teams. Delta’s transaction log is easier to reason about. The tooling for vacuum, optimize, and Z-ordering is well-documented and predictable.


    The Comparison You Actually Need

    Feature Delta Lake Apache Iceberg
    Engine supportSpark-native; connectors for othersTruly multi-engine (Spark, Flink, Trino, Snowflake, Athena)
    Partition evolutionRequires full table rewriteSchema-safe, no data rewrite needed
    Hidden partitioningNot supportedSupported — engines auto-prune
    MERGE / CDC performanceExcellent on SparkStrong, improving; slightly behind Delta on Spark CDC
    Vendor alignmentDatabricks ecosystemVendor-neutral, Apache foundation
    Operational toolingMature, well-documentedMaturing fast; strong in 2024–2025
    Multi-cloud flexibilityPossible but frictionFirst-class support across clouds
    Migration effortN/A (starting point)Non-trivial; plan 4–8 weeks

    THE MIGRATION: WHAT IT ACTUALLY COST US

    The Migration: What It Actually Cost Us

    I’ll be direct: the migration was harder than I expected. If you’ve read my piece on automation in data engineering, you’ll recognise the pattern — the technical part is rarely the hard part. It’s the downstream work nobody accounts for.

    The core work wasn’t the data conversion — we used the delta-iceberg migration utility and it handled most of the heavy lifting. The harder parts were everything else.

    Downstream dependency mapping.

     Every dbt model, every Airflow DAG, every Spark job that referenced a Delta table path needed updating. We had 40+ models. Two had hardcoded partition paths we didn’t catch until QA.

    Metadata catalog updates.

     We use AWS Glue Data Catalog. Every table needed its metadata updated to reflect the Iceberg format. Glue’s Iceberg support has improved, but it’s not frictionless.

    Testing the rollback plan. We kept Delta tables live for 30 days post-migration with a cutover switch in Airflow. That meant double-writing during the transition window — additional storage cost and added pipeline complexity.

    ⚠️ The migration trap: The data conversion tooling works. What catches teams off guard is the downstream mapping work — every pipeline, model, and job that references a table path. Budget more time for that than for the actual format conversion.

    Total elapsed time: six weeks. Two engineers. Not a weekend project.


    When to Choose Delta Lake

    • Your primary compute layer is Databricks
    • You’re a small team that wants simpler operations
    • You’re doing high-frequency CDC on Spark
    • You’re early stage — get something working first

    When to Choose Iceberg

    • You’re running multiple query engines (Spark + Snowflake, Trino + Flink)
    • You need partition evolution without full table rewrites
    • You’re building a vendor-neutral architecture
    • Your analysts query the lakehouse directly from Snowflake

    What I’d Do Differently

    Start with Iceberg if you’re greenfield. The setup is slightly more involved, but you avoid the migration cost entirely. The ecosystem has matured enough in 2024-2025 that “Iceberg is less mature” is no longer a strong argument.

    If you’re already on Delta and it’s working — don’t migrate for the sake of it. Migrate when you hit a specific limit: engine lock-in, partition inflexibility, or multi-cloud requirements.

    And if you do migrate, don’t underestimate the downstream mapping work. The data conversion is the easy part.


    Frequently Asked Questions

    What is the main difference between Delta Lake and Apache Iceberg?

    Delta Lake is a table format developed by Databricks, optimised for Spark workloads with strong Databricks integration. Apache Iceberg is an open table format designed for multi-engine environments — it works natively with Spark, Flink, Trino, Snowflake, and Athena without custom connectors. The core difference is engine flexibility.

    Is Apache Iceberg better than Delta Lake?

    It depends on your stack. Iceberg is better if you’re running multiple query engines or building a vendor-neutral architecture. Delta Lake is better if Databricks is your primary compute layer. Neither format is objectively superior.

    Can Snowflake read Delta Lake tables?

    Yes, through Delta Sharing or Snowflake’s Delta connector — but with limitations. Snowflake reads Iceberg tables natively as a first-class citizen, which is why multi-engine stacks tend to favour Iceberg.

    How hard is it to migrate from Delta Lake to Apache Iceberg?

    Harder than most blog posts suggest. The data conversion tooling handles the format migration, but remapping downstream pipelines, updating metadata catalogs, and testing rollback scenarios adds significant effort. Budget four to eight weeks for a production migration with 30–50 tables.

    Does dbt support Apache Iceberg?

    Yes. dbt supports Iceberg through the Spark and Athena adapters, and Snowflake’s Iceberg table support works with dbt models running on Snowflake. Production-ready as of 2024.

    What is hidden partitioning in Apache Iceberg?

    Hidden partitioning lets Iceberg manage partition logic transparently. You define partition transforms like days(event_timestamp) at the table level, and Iceberg handles physical file organisation and query pruning automatically — no need to filter on partition columns explicitly.

  • The Problem with Data Engineering Certifications That Nobody Talks About

    The Problem with Data Engineering Certifications That Nobody Talks About

    I passed the SnowPro Gen AI certification not too long ago. Within the same week I was back at my desk staring at a broken pipeline that no multiple-choice question had ever prepared me for. The cert looked great on my profile. It fixed exactly nothing about the actual problem in front of me.

    I’m not saying certifications are worthless. I’m saying the industry has developed a quietly dishonest relationship with them — one where vendors, hiring managers, and candidates all play along with a fiction that a passed exam means something it doesn’t. Nobody wants to be the one to say it out loud.

    So I will. Let me be direct about what’s actually going on.


    TL;DR

    • Certifications test what vendors want you to know about their products — not whether you can actually engineer data systems that work under real conditions
    • The exam content is often months or years behind the tools you’ll actually use in production
    • Hiring managers use certs as a filter because it’s easy — not because it’s accurate
    • You can pass most data engineering certs with two weeks of practice exams and zero production experience
    • The real signal employers should care about — and rarely do — is what you’ve built, what broke, and what you learned from it
    • Certifications have a specific, narrow value: they are a vocabulary test, not a competence test. Know what you’re paying for

    WHAT CERTIFICATIONS ACTUALLY TEST

    Let’s start with what’s literally on the exam. Take the Databricks Certified Data Engineer Associate . The exam covers Delta Lake concepts, basic Spark operations, Unity Catalog, Databricks workflows. Good things to know.

    But the exam tests your ability to identify the correct answer from four options in a controlled environment. It does not test whether you can debug a production Spark job that’s been running for six hours and slowly consuming memory. It doesn’t test whether you can diagnose why a Delta merge is creating file fragmentation degrading query performance. It doesn’t test whether you can architect a pipeline that recovers gracefully when an upstream API starts returning malformed JSON at 3am.

    Those are the problems data engineers actually face. None of them are in the certification.

    A certification tells you that someone understood the conceptual framework of a product well enough to pass a vendor-designed exam. It tells you almost nothing about their ability to operate that product under adversarial conditions. And production is always adversarial.

    This gap exists in the AWS Certified Data Engineer Associate ,the Google Professional Data Engineer ,the Azure Data Engineer Associate ,and every dbt or Snowflake certification available. They all test the vendor’s idealised scenario.

    Real pipelines are never idealised.


    THE VENDOR INCENTIVE PROBLEM

    Who designs these exams? The vendors. Who benefits when thousands of engineers study for, pay for, and pass these exams? The vendors. Certification programmes are not primarily educational products. They are marketing products that create a credentialled user base and deepen platform lock-in.

    When Snowflake designs its certification exams ,the goal is not to produce engineers who can evaluate whether Snowflake is the right tool. The goal is to produce engineers deeply familiar with Snowflake’s architecture, syntax, and product positioning — engineers who will advocate for Snowflake when tooling decisions come up at their company.

    The exam content is shaped by commercial interest, not by what data engineers actually need to know. The practical consequence: certifications optimise for breadth of product knowledge over depth of engineering judgment. You learn feature names, service limits, and recommended architectures. You don’t develop the instinct that tells you something is going to break before it breaks.


    THE HIRING MANAGER TRAP

    I’ve sat in hiring discussions where a candidate without certifications was dismissed faster than one with a string of logos after their name, despite the uncertified candidate having a demonstrably stronger GitHub portfolio and much more interesting answers about production incidents they’d owned.

    Certifications persist in job postings because they’re easy to verify and hard to argue with. A cert is binary. Either you have it or you don’t. Technical judgment, architecture instinct, debugging ability — these require effort to assess.

    ⚠️ The signal problem: If you can pass a data engineering certification with two weeks of practice exams and no production experience — and you can — then having the certification tells an interviewer almost nothing about whether you can do the job. It tells them you can study for a test. That’s useful. But it’s not the same thing.

    The engineers most dismissive of certifications are often the most experienced. The engineers who lean most heavily on cert lists are often the ones who haven’t done enough production work to know what the gap actually looks like.


    THE STALE CONTENT PROBLEM

    Data engineering moves fast. The tooling landscape in 2024 looks materially different from 2021. dbt Core has changed substantially. Apache Iceberg has gone from niche to mainstream. Lakehouse architecture has shifted from concept to default.

    Certification exams do not move at this speed. Exam content is updated infrequently — sometimes annually, sometimes less. You can hold an AWS Data Engineer cert that emphasises EMR and Glue in patterns most teams have replaced with more modern tooling. You can hold a Databricks cert that doesn’t reflect how Unity Catalog has fundamentally changed governance.

    The cert is not wrong. It’s just dated. And dated knowledge in data engineering isn’t neutral — it can actively mislead you about how things should be built.

    I wrote about a related version of this in “Why I Stopped Using Snowflake Tasks for Orchestration” — official documentation and certification content often lags behind what practitioners have already learned through trial and error in production.


    WHAT YOU ACTUALLY LEARN WHEN YOU STUDY FOR A CERT

    Here’s the part I want to be fair about. Studying for a data engineering certification isn’t worthless. It’s just worth something different from what most people think.

    When you study for the Google Professional Data Engineer exam, you learn the GCP data ecosystem — BigQuery, Dataflow, Pub/Sub, Cloud Composer, Dataproc — in a structured way. You develop a vocabulary. You understand how services relate to each other.

    What it doesn’t give you is judgment. Judgment about when to use Dataflow versus Dataproc. When BigQuery’s cost model makes it the wrong tool despite its performance. When a simple Cloud Function is a better answer than a fully orchestrated pipeline.

    The honest framing: a certification is a vocabulary test with a structured curriculum. If you’ve never worked on a platform and need to get up to speed quickly, studying for the cert is efficient. If you already have production experience, the cert adds limited signal beyond what’s already on your resume.


    THE PRACTICE EXAM LOOPHOLE NOBODY WANTS TO DISCUSS

    Most data engineering certifications can be passed with aggressive practice exam grinding and minimal practical experience. Platforms like Udemy , Whizlabs and ExamTopics sell practice exam bundles close enough to real questions that a disciplined studier can reverse-engineer most of the exam in two to three weeks.

    I’ve seen candidates with zero Snowflake production experience pass the SnowPro Core exam in a week of evening study. I’ve seen engineers memorise their way through the AWS Data Engineer Associate without writing a single Glue job. The credential is indistinguishable from someone who earned it through genuine depth.

    The vendors know this. They update exam content periodically to counter braindump culture, but it’s an arms race they’re perpetually losing.


    WHAT ACTUALLY SIGNALS ENGINEERING COMPETENCE

    If I’m hiring a data engineer, here’s what I actually want to see.

    Tell me about a pipeline that broke in production. Not a hypothetical. What broke, how you found out, what the root cause was, how you fixed it, what you changed to prevent recurrence. This conversation reveals more engineering judgment than any certification.

    Show me something you built. A GitHub repo .A dbt project. A pipeline architecture diagram with a written explanation. The work I’ve been documenting — from the problem with dbt incremental models to Snowflake zero-copy cloning gotchas — is far more useful signal than any certification I hold.

    Tell me about a technical decision you disagreed with. Engineering judgment includes knowing when to push back, when to compromise, how to argue for a position with evidence. No cert tests this.

    Walk me through how you’d approach this problem. Give them a real scenario — a data quality issue, a cost spike, a schema migration in a live system. Watch how they think, not just what they know.

    The gap between what certifications measure and what engineering competence looks like is large enough that I’d rather see zero certifications with a detailed post-mortem of a real incident than four certs with nothing to show for the work.


    WHEN CERTIFICATIONS ARE ACTUALLY WORTH PURSUING

    You’re breaking into the field. If you’re transitioning into data engineering, certifications serve a genuine purpose. They give you structured curriculum and a credential that signals seriousness to employers who don’t yet have anything else to evaluate you on.

    Your employer requires it. Many enterprise organisations and consulting firms have vendor partnership requirements mandating certified staff levels. In that case, the cert has real organisational value regardless of signal quality.

    You’re learning a new platform systematically. Using cert study as structured onboarding to a new tool is legitimate. The curriculum forces breadth coverage self-directed learning often misses. Just know that completing the cert doesn’t mean you know how to use the platform well.

    You’re in a market where it’s table stakes. In some geographies and sectors, certain certs are required to get an interview. Clear the gate, then demonstrate real depth in the room.

    The certification isn’t the problem. The mythology around it is. The idea that passing the exam means you can build reliable data systems — that’s the fiction that causes real damage.


    WHAT THE INDUSTRY SHOULD DO INSTEAD

    Portfolio-based evaluation. A documented data engineering project — architecture decisions, tradeoffs, failures encountered — tells a hiring team far more than an exam score. GitHub already supports this.

    Incident post-mortems as credentials. A well-written post-mortem demonstrates debugging methodology, systems thinking, and the ability to learn from failure. No certification tests these.

    Practical assessments over multiple choice. The Databricks Data Engineer Professional is harder than most — it has a coding component requiring actual proficiency. More exams should work this way.

    Open curriculum from neutral sources. The Data Engineering Handbook and open-source community resources are doing more for actual engineering capability than most vendor certification programmes.


    FREQUENTLY ASKED QUESTIONS

    Are data engineering certifications worth it in 2024?
    It depends on where you are in your career. For someone entering the field, certs provide structured curriculum and a credential that signals seriousness. For experienced engineers, your production track record carries far more weight with strong technical hiring teams. Certs are worth what they cost if you understand what they are: a vocabulary test, not a competence test.

    Which data engineering certification is the most respected?
    Among practitioners, the Databricks Data Engineer Professional is generally seen as harder and more meaningful because it includes a practical component. Google Professional Data Engineer has strong enterprise name recognition. AWS Certified Data Engineer Associate is widely recognised in cloud-native teams. But respected by whom matters — strong engineering teams care less about cert logos than about demonstrated ability.

    Can you become a data engineer without certifications?
    Absolutely. Many strong data engineers have no certifications at all. A track record of real work — systems built, incidents resolved, architectural decisions owned — is equally or more compelling to technical hiring teams worth impressing.

    How long does it take to pass data engineering certification exams?
    Most candidates report 2–6 weeks of focused study. With aggressive practice exam preparation, some pass in under two weeks — which is part of what makes the credentials less meaningful than they appear.

    Do data engineering certifications expire?
    Yes. AWS certifications expire after three years, Google Cloud after two, Databricks varies by level. Recertification tends to be easier than initial certification and often doesn’t reflect how dramatically the tooling has evolved.

    What should a data engineering portfolio include instead of certifications?
    End-to-end pipeline projects with documented architecture decisions. Written post-mortems of production incidents. Data quality testing approaches. dbt projects with meaningful transformation logic. Cost analyses or performance optimisations from real environments. Anything that shows how you think, not just what tools you’ve touched.


  • The Problem with Zero-Copy Cloning in Snowflake That Nobody Talks About

    The Problem with Zero-Copy Cloning in Snowflake That Nobody Talks About

    Every time I demo Snowflake to someone new, zero-copy cloning gets the biggest reaction. You type one line. You get an instant copy of a table — or an entire database — with no data duplication, no storage cost at the moment of creation. It feels like magic.

    And it is genuinely impressive engineering. I’m not here to tell you it’s a bad feature. It’s one of my favourite things about Snowflake and I use it constantly.

    But I’ve watched teams get badly surprised by it. A dev environment clone that started silently inflating the storage bill. A cloned database used for UAT that bypassed data masking policies on PII columns. A Time Travel query on a clone that returned nothing because the source table’s retention window had already expired.

    None of these are edge cases. They’re predictable consequences of how zero-copy cloning actually works — consequences that the marketing language around “instant, free copies” tends to obscure. Let me get into it.

    TL;DR

    • Zero-copy cloning is one of Snowflake’s best features — and one of the most misunderstood ones in production
    • Clones share micropartitions with the source — any modification to either side starts writing new storage, and that cost adds up fast in ways that aren’t visible upfront
    • Clones don’t inherit resource monitors, row-level security policies, or dynamic data masking by default — this is a compliance and governance trap waiting to happen
    • Time Travel on clones behaves differently from what most people expect, especially when the source table has already moved past its retention window
    • Clone sprawl is real — it’s invisible in the UI, expensive to audit, and teams rarely have a cleanup strategy until the bill arrives
    • This article covers what zero-copy cloning actually does under the hood, where it silently fails you, and how to use it without it becoming a liability

    HOW ZERO-COPY CLONING ACTUALLY WORKS

    When you clone a table in Snowflake, you’re not copying data. You’re creating a new metadata pointer that references the same underlying micropartitions as the source object.

    -- Instant. No data movement. No storage cost at this moment.
    CREATE TABLE orders_clone CLONE orders;
    
    -- Works for schemas too
    CREATE SCHEMA dev_schema CLONE prod_schema;
    
    -- And entire databases
    CREATE DATABASE dev_db CLONE prod_db;

    At the moment of creation, the clone costs you nothing in storage. Both the original and the clone point to the same micropartitions on disk. The moment either side changes, Snowflake uses copy-on-write. The modified micropartition gets written fresh for whichever side made the change.

    Think of it like a fork in a Git repo. At fork time, both repos share the same commit history. The moment either side commits, they diverge. The more divergence, the more independent storage you accumulate. Zero-copy cloning works exactly like this — except the “commits” are DML operations and the cost is real money.


    PROBLEM 1 — STORAGE COSTS THAT CREEP UP INVISIBLY

    A team clones production to create a development environment. The dev team runs experiments, updates records, backfills some columns. Six weeks later, storage is up 40%. Every modified micropartition in the dev database is now independent storage. Production kept its micropartitions too. You’re paying for both.

    The cost also compounds with Time Travel. If production has 90-day retention and you clone it for dev, that clone also starts with 90-day retention. DML operations in dev accumulate 90 days of write history.

    -- Clone with reduced Time Travel for non-production environments
    CREATE DATABASE dev_db CLONE prod_db;
    
    -- Immediately reduce Time Travel on the clone
    ALTER DATABASE dev_db SET DATA_RETENTION_TIME_IN_DAYS = 1;
    
    -- Or set it at schema level for finer control
    ALTER SCHEMA dev_db.analytics SET DATA_RETENTION_TIME_IN_DAYS = 0;

    Audit clone storage footprint regularly:

    -- Find clones and their storage footprint
    SELECT
        table_catalog,
        table_schema,
        table_name,
        clone_group_id,
        bytes / (1024 * 1024 * 1024)        AS size_gb,
        row_count,
        created                              AS clone_created_at
    FROM snowflake.account_usage.tables
    WHERE clone_group_id IS NOT NULL
      AND deleted IS NULL
    ORDER BY bytes DESC;

    This same class of invisible cost creep comes up a lot with Snowflake features that look free until you read the bill — similar to what I covered in “Why I Stopped Using Snowflake Tasks for Orchestration“


    PROBLEM 2 — GOVERNANCE POLICIES DON’T FOLLOW THE CLONE

    Dynamic data masking policies are not automatically inherited by clones. The clone is a new object with no masking policies applied.

    -- On prod table — email is masked for non-PII roles
    SELECT email FROM prod_orders LIMIT 5;
    -- Result: ***@***.com (masked)
    
    -- On the clone without explicit policy assignment
    SELECT email FROM orders_clone LIMIT 5;
    -- Result: [email protected] (unmasked raw PII)

    Same problem with row access policies. A user restricted to one region in production can see all regions on the clone.

    The fix — make policy application part of your clone process:

    -- Step 1: Clone the table
    CREATE TABLE dev_db.analytics.orders CLONE prod_db.analytics.orders;
    
    -- Step 2: Re-apply masking policies immediately
    ALTER TABLE dev_db.analytics.orders
        MODIFY COLUMN email
        SET MASKING POLICY prod_db.security.email_mask;
    
    ALTER TABLE dev_db.analytics.orders
        MODIFY COLUMN phone_number
        SET MASKING POLICY prod_db.security.phone_mask;
    
    -- Step 3: Re-apply row access policy
    ALTER TABLE dev_db.analytics.orders
        ADD ROW ACCESS POLICY prod_db.security.region_access_policy
        ON (region_code);

    Better: wrap it in a stored procedure that enforces policy application as part of the clone operation:

    CREATE OR REPLACE PROCEDURE create_governed_clone(
        source_table      VARCHAR,
        target_table      VARCHAR,
        masking_policies  ARRAY
    )
    RETURNS STRING
    LANGUAGE JAVASCRIPT
    AS
    $$
        var clone_stmt = snowflake.execute({
            sqlText: `CREATE TABLE ${TARGET_TABLE} CLONE ${SOURCE_TABLE}`
        });
    
        for (var i = 0; i < MASKING_POLICIES.length; i++) {
            var policy = MASKING_POLICIES[i];
            snowflake.execute({
                sqlText: `ALTER TABLE ${TARGET_TABLE}
                          MODIFY COLUMN ${policy.column}
                          SET MASKING POLICY ${policy.policy_name}`
            });
        }
    
        return 'Clone created with governance policies applied: ' + TARGET_TABLE;
    $$;

    A clone that exists without its governance policies re-applied is a compliance gap, not a convenience feature. If you’re running dbt on top of Snowflake, the same mindset applies — see “The Problem with dbt Tests Nobody Talks About“:


    PROBLEM 3 — TIME TRAVEL ON CLONES ISN’T WHAT YOU THINK

    A clone’s Time Travel history starts from its creation date. You cannot go back to a point before the clone was created on the clone object.

    -- Source table: exists since 2024-01-01, 90-day retention
    -- Clone created: 2024-03-01
    
    -- This works — within the clone's own history
    SELECT * FROM orders_clone
    AT (TIMESTAMP => '2024-03-15 10:00:00'::TIMESTAMP_TZ);
    
    -- This FAILS — before the clone existed
    SELECT * FROM orders_clone
    AT (TIMESTAMP => '2024-02-01 10:00:00'::TIMESTAMP_TZ);
    -- Error: Statement time travel is not available for this object
    
    -- For pre-clone history, query the SOURCE table
    SELECT * FROM orders
    AT (TIMESTAMP => '2024-02-01 10:00:00'::TIMESTAMP_TZ);

    Also watch: if you clone a table that’s near the end of its retention window, any history that expires on the source is gone. The clone can’t access expired source history.


    PROBLEM 4 — CLONE SPRAWL AND THE INVISIBLE COST PROBLEM

    Zero-copy cloning is so easy that people create clones for everything — UAT, load testing, feature branches, one-off investigations that were supposed to be deleted on Friday. Three months later, nobody knows what exists or how diverged it’s become.

    Full clone audit query:

    SELECT
        t.table_catalog                                   AS database_name,
        t.table_schema                                    AS schema_name,
        t.table_name,
        t.clone_group_id,
        t.row_count,
        ROUND(t.bytes / POW(1024, 3), 3)                  AS size_gb,
        t.created                                         AS created_at,
        t.last_altered                                    AS last_modified_at,
        DATEDIFF('day', t.created, CURRENT_TIMESTAMP())   AS age_days,
        CASE
            WHEN DATEDIFF('day', t.last_altered, CURRENT_TIMESTAMP()) > 30
            THEN 'STALE — review for deletion'
            ELSE 'Active'
        END AS staleness_flag
    FROM snowflake.account_usage.tables t
    WHERE t.clone_group_id IS NOT NULL
      AND t.deleted IS NULL
    ORDER BY t.bytes DESC;

    Tag every clone at creation with expiry metadata:

    CREATE DATABASE uat_db CLONE prod_db
        COMMENT = '{"purpose": "UAT for v2.4 release", "owner": "[email protected]", "expires": "2024-04-30", "ticket": "JIRA-1234"}';
    
    -- Query clones past their expiry date
    SELECT
        table_catalog,
        table_schema,
        table_name,
        TRY_PARSE_JSON(comment):expires::DATE AS expiry_date,
        TRY_PARSE_JSON(comment):owner::STRING AS owner
    FROM snowflake.account_usage.tables
    WHERE clone_group_id IS NOT NULL
      AND deleted IS NULL
      AND TRY_PARSE_JSON(comment):expires::DATE < CURRENT_DATE();

    PROBLEM 5 — CLONING STREAMS AND TASKS DOESN’T WORK HOW YOU EXPECT

    Streams are not cloned when you clone a table or schema. The clone contains the data but has no streams attached.

    -- Prod table has a stream attached
    SHOW STREAMS ON TABLE prod_db.analytics.orders;
    -- Returns: orders_cdc_stream
    
    -- Clone the table
    CREATE TABLE dev_db.analytics.orders CLONE prod_db.analytics.orders;
    
    -- Check streams on clone
    SHOW STREAMS ON TABLE dev_db.analytics.orders;
    -- Returns: (empty)

    If you need CDC streams on cloned tables, create them explicitly after cloning:

    CREATE OR REPLACE STREAM dev_db.analytics.orders_cdc_stream
        ON TABLE dev_db.analytics.orders
        APPEND_ONLY = FALSE
        SHOW_INITIAL_ROWS = FALSE;
    
    CREATE OR REPLACE TASK dev_db.analytics.process_orders_changes
        WAREHOUSE = dev_wh
        SCHEDULE = '5 minute'
        WHEN SYSTEM$STREAM_HAS_DATA('dev_db.analytics.orders_cdc_stream')
    AS
        CALL dev_db.analytics.process_orders_sp();

    Tasks are cloned but start in a SUSPENDED state — they don’t auto-resume, which is correct behaviour (you don’t want dev tasks firing against prod targets), but it surprises teams expecting a live pipeline copy. If your pipeline relies on dbt incremental models consuming from those streams, the failure compounds further — see “The Problem with Incremental Models in dbt Nobody Talks About“


    WHEN ZERO-COPY CLONING IS THE RIGHT TOOL

    Before risky migrations — clone first, get an instant rollback point:

    -- Before a risky migration
    CREATE TABLE orders_pre_migration CLONE orders;
    
    -- Run your migration
    ALTER TABLE orders ADD COLUMN new_column VARCHAR;
    UPDATE orders SET new_column = derive_value(existing_column);
    
    -- If something went wrong:
    -- DROP TABLE orders;
    -- ALTER TABLE orders_pre_migration RENAME TO orders;

    Instant dev environments, UAT cycles, zero-downtime data fixes — all excellent use cases. The feature is great. Using it without understanding the lifecycle is where teams get into trouble. If you want to go further on cost reduction for dev workloads, pairing clone strategy with DuckDB is worth exploring — “How to Query Snowflake in DuckDB and Cut Your Bill While Doing It“


    FREQUENTLY ASKED QUESTIONS

    Q: Does zero-copy cloning in Snowflake really cost nothing?
    A: At creation: yes. The cost begins the moment either side is modified via copy-on-write. In active dev environments that are modified frequently, storage costs can grow significantly over weeks. Time Travel retention on the clone compounds this further.

    Q: Do data masking policies transfer when you clone a table?
    A: No. Masking policies are not inherited by clones. Sensitive columns are exposed in plaintext on the clone unless you explicitly re-apply policies after creation. Treat clone creation and policy application as a single atomic operation.

    Q: Can I use Time Travel on a clone to go back before it was created?
    A: No. A clone’s Time Travel history starts at its creation date. For history before the clone was created, query the source table directly.

    Q: Are Snowflake Streams copied when you clone a table?
    A: No. Streams are not part of the clone operation. Create them explicitly on the clone if your pipeline depends on CDC. Tasks are cloned but start suspended.

    Q: How do I audit all clones in my Snowflake account?
    A: Query snowflake.account_usage.tables filtering on clone_group_id IS NOT NULL. Tag clones at creation with JSON metadata in the COMMENT field — owner, expiry, purpose — to make audits actionable.

    Q: What’s the best practice for cloning production for dev?
    A: Clone, then immediately: reduce Time Travel retention to 0 or 1 day, re-apply all masking and row access policies, set a resource monitor on dev warehouses, and tag the clone with an expiry date in the COMMENT field.


    Related blogs

    → Snowflake official docs — cloning objects
    → Snowflake dynamic data masking docs
    → Snowflake Time Travel docs
    → Snowflake resource monitors docs
    → Snowflake Streams 

  • How to Query Snowflake in DuckDB (And Cut Your Bill While Doing It)

    How to Query Snowflake in DuckDB (And Cut Your Bill While Doing It)

    TL;DR

    • Snowflake’s 60-second minimum billing means a 4-second query gets charged for a full minute — you’re paying for 55 seconds of nothing
    • You can query Snowflake data in DuckDB via two routes: Iceberg tables on S3 (no warehouse needed) or ADBC using Apache Arrow (up to 38x faster than ODBC)
    • Once data is local in DuckDB, every subsequent query is free — no cloud credits consumed
    • A hybrid triage approach (short queries → DuckDB/MotherDuck, heavy ETL → Snowflake) cuts BI compute costs by 70–90% in practice
    • Dev and CI/CD workloads moved to local DuckDB eliminate an entire category of cloud spend entirely

    I’ve been building on Snowflake long enough to know the ritual. Warehouse wakes up. Query runs in three seconds. Warehouse idles. You get billed for sixty seconds anyway. Multiply that by every analyst, every BI dashboard refresh, every dbt run in your dev environment — and suddenly you’re staring at a bill that feels completely disconnected from the actual work that happened.

    For a long time I assumed this was just the price of doing business on a best-in-class cloud warehouse. What I didn’t realise — until I started taking DuckDB seriously — is that a meaningful chunk of that bill doesn’t have to exist at all.

    This article covers three concrete methods to get Snowflake data into DuckDB, the cost math behind why you’d want to, and how to decide what actually belongs on which engine.


    THE REAL PROBLEM: YOU’RE PAYING FOR COMPUTE YOU DIDN’T USE

    Snowflake bills compute per second — but only after a 60-second minimum each time a warehouse resumes from suspension. A query that takes five seconds gets billed for a full minute. You paid for 55 seconds of nothing.

    It gets worse at scale. When a BI dashboard fires 20 queries on load, each taking three seconds, that single page view triggers 1,200 seconds of billed compute time. The actual work? One minute.

    And then warehouse sizing compounds it further. Each size increase in Snowflake doubles credit consumption. Teams defaulting to Medium or Large for everything are paying a 4x to 8x cost premium for workloads that could run perfectly well on X-Small.

    I’ve seen this exact pattern on almost every Snowflake environment I’ve worked in. Oversized warehouse, auto-suspend set to ten minutes, no resource monitors, nobody looking at query history.


    QUICK WINS INSIDE SNOWFLAKE FIRST

    Before touching the architecture, fix the obvious things. These alone can cut spend by 20–40%.

    Set AUTO_SUSPEND to exactly 60 seconds. Not lower — setting it below 60 is counterproductive because a query arriving in that first minute triggers another 60-second minimum. Not higher — every idle second past 60 is wasted money.

    Default to X-Small warehouses. Only scale up when a specific workload has a documented SLA that requires it.

    Add resource monitors:

    CREATE OR REPLACE RESOURCE MONITOR monthly_etl_monitor
    WITH CREDIT_QUOTA = 5000
    TRIGGERS ON 75 PERCENT DO NOTIFY
            ON 100 PERCENT DO SUSPEND;
    
    ALTER WAREHOUSE etl_heavy_wh 
    SET RESOURCE_MONITOR = monthly_etl_monitor;

    METHOD 1 — QUERYING SNOWFLAKE ICEBERG TABLES DIRECTLY IN DUCKDB

    If your organisation has moved to Iceberg tables with underlying data stored in S3, you can read those tables directly in DuckDB — no Snowflake warehouse running, no credits consumed.

    Install the extensions:

    INSTALL httpfs;
    LOAD httpfs;
    INSTALL iceberg;
    LOAD iceberg;

    Configure AWS credentials:

    CREATE SECRET (
        TYPE S3,
        PROVIDER CREDENTIAL_CHAIN
    );

    Find the current metadata file for your Snowflake-managed Iceberg table:

    SELECT PARSE_JSON(
      SYSTEM$GET_ICEBERG_TABLE_INFORMATION('YOUR_DB.YOUR_SCHEMA.YOUR_TABLE')
    )['metadataLocation']::varchar;

    Query it in DuckDB:

    SELECT
        customer_id,
        COUNT(*)
    FROM iceberg_scan('s3://your-bucket/path/to/metadata/00001-xxxx.metadata.json')
    GROUP BY 1;

    Materialise once for fast repeated queries:

    CREATE TABLE payments AS 
    SELECT * FROM iceberg_scan('s3://your-bucket/.../metadata.json');

    After this: same aggregation runs in 1.5s instead of 54s.

    Real benchmark: a SELECT * on a 110-million row table finished in 29 seconds in DuckDB on an M1 MacBook. Same query on an X-Small Snowflake warehouse took 72 seconds.

    The honest limitation: DuckDB’s Iceberg support is still maturing. You need direct S3 access and have to point DuckDB at a specific metadata file rather than a catalog. This will improve over time, but it works today.


    METHOD 2 — QUERYING NATIVE SNOWFLAKE TABLES VIA ADBC

    Not on Iceberg yet? ADBC (Arrow Database Connectivity) is the right tool here.

    Apache Arrow is a columnar memory format. When you connect Snowflake to DuckDB via ADBC, data stays columnar the entire way. Traditional ODBC forces Snowflake to convert columnar → row for transfer, then DuckDB converts row → columnar for processing. DuckDB’s benchmarks show ADBC is up to 38x faster than ODBC.

    Install:

    pip install adbc_driver_snowflake pyarrow duckdb cryptography

    Connect to Snowflake and pull data as an Arrow table:

    import adbc_driver_snowflake.dbapi
    import duckdb
    import os
    from read_private_key import read_private_key
    
    SNOWFLAKE_CONFIG = {
        'adbc.snowflake.sql.account': os.getenv('SNOWFLAKE_ACCOUNT'),
        'adbc.snowflake.sql.warehouse': os.getenv('SNOWFLAKE_WAREHOUSE'),
        'adbc.snowflake.sql.role': os.getenv('SNOWFLAKE_ROLE'),
        'adbc.snowflake.sql.database': os.getenv('SNOWFLAKE_DATABASE'),
        'username': os.getenv('SNOWFLAKE_USER'),
        'adbc.snowflake.sql.client_option.jwt_private_key_pkcs8_value': pem_key,
        'adbc.snowflake.sql.auth_type': 'auth_jwt'
    }
    
    snowflake_conn = adbc_driver_snowflake.dbapi.connect(
        db_kwargs={**SNOWFLAKE_CONFIG}
    )
    
    snowflake_cursor = snowflake_conn.cursor()
    snowflake_cursor.execute("SELECT * FROM SANDBOX_DB.MY_SCHEMA.RAW_ORDERS")
    
    # Fetch as Arrow table — stays columnar, no serialisation overhead
    arrow_table = snowflake_cursor.fetch_arrow_table()
    
    # Persist locally in DuckDB
    duckdb_conn = duckdb.connect('demo.db')
    duckdb_conn.execute("""
        CREATE TABLE IF NOT EXISTS raw_orders AS 
        SELECT * FROM arrow_table
    """)

    One heads-up: figuring out the connection parameters using a private key is not straightforward — the docs aren’t great on this point. The private key needs to be re-encoded into PEM format before passing it to the ADBC driver:

    from cryptography.hazmat.primitives import serialization
    
    def read_private_key(private_key_path: str, private_key_passphrase: str = None) -&gt; str:
        with open(private_key_path, 'rb') as key_file:
            private_key = serialization.load_pem_private_key(
                key_file.read(),
                password=private_key_passphrase.encode() if private_key_passphrase else None
            )
            pem_key = private_key.private_bytes(
                encoding=serialization.Encoding.PEM,
                format=serialization.PrivateFormat.PKCS8,
                encryption_algorithm=serialization.NoEncryption()
            )
            return pem_key.decode('utf-8')

    Once that’s sorted, the workflow is clean: pull data from Snowflake via ADBC once, materialise it locally in DuckDB, query it as many times as you want — zero Snowflake credits consumed after the initial pull.


    METHOD 3 — THE HYBRID ARCHITECTURE: ROUTE WORKLOADS BY TYPE

    The two methods above are great for development and ad-hoc analysis. For production BI workloads, the cleanest solution I’ve seen is a hybrid architecture where you triage queries by workload type.


    The insight that unlocked this for me was using Snowflake’s query_history to actually categorise what’s running:

    WITH query_stats AS (
        SELECT
            warehouse_name,
            user_name,
            query_id,
            execution_time / 1000 AS execution_seconds
        FROM snowflake.account_usage.query_history
        WHERE
            start_time &gt;= DATEADD('day', -30, CURRENT_TIMESTAMP())
            AND warehouse_name IS NOT NULL
            AND execution_status = 'SUCCESS'
    )
    SELECT
        warehouse_name,
        user_name,
        COUNT(query_id) AS query_count,
        MEDIAN(execution_seconds) AS median_execution_seconds,
        CASE
            WHEN query_count &gt; 1000 AND median_execution_seconds &lt; 30 
                THEN 'Interactive BI / High Frequency'
            WHEN query_count &lt;= 1000 AND median_execution_seconds &lt; 60 
                THEN 'Ad-Hoc Exploration'
            WHEN median_execution_seconds &gt;= 300 
                THEN 'Batch ETL / Heavy Analytics'
            ELSE 'General Purpose'
        END AS workload_category
    FROM query_stats
    GROUP BY warehouse_name, user_name
    ORDER BY query_count DESC;

    Use MEDIAN not AVG — outlier queries skew the average and give a misleading picture of typical duration.

    Routing logic:

    • Short and bursty BI (sub-30s, high frequency) → move to usage-based engine. Real math: $528/month on Snowflake X-Small running continuously vs $5.87/month on per-second billing for the same workload.
    • Dev and CI/CD → local DuckDB, zero cloud credits
    • Heavy batch ETL, multi-TB → keep on Snowflake, 60s minimum is irrelevant for hour-long jobs

    This is the same principle I apply when thinking about orchestration — use the right tool for the job, not the most powerful tool for everything. I wrote about a similar decision process in why I stopped using Snowflake Tasks for orchestration — the short version is that mature orchestration tools give you far more control over exactly this kind of workload routing.


    WHEN TO STAY ON SNOWFLAKE

    Multi-terabyte batch processing — predictable provisioned compute matters more than idle cost savings when a job runs for hours.

    Enterprise governance — complex data masking, RBAC at scale, data residency requirements. Snowflake’s security surface is mature. DuckDB isn’t designed for this.

    Already-efficient workloads — if a warehouse runs at high utilisation for 8 hours straight, there’s no idle tax to eliminate. Don’t fix what isn’t broken.


    WHAT REAL COST SAVINGS LOOK LIKE

    • One SaaS company: 70%+ reduction in warehousing costs after moving to DuckDB-based solution
    • Okta: $60,000/month Snowflake spend for threat detection reduced substantially using parallel DuckDB instances
    • A data engineering team: 79% immediate reduction in Snowflake BI spend using DuckDB as a caching layer, 7x faster query times

    None of these required abandoning Snowflake. They required deciding which workloads actually needed it.


    FREQUENTLY ASKED QUESTIONS

    Can you query Snowflake data in DuckDB without a Snowflake warehouse running?
    Yes — two ways. Iceberg tables via the iceberg extension (no warehouse), or native tables via ADBC. Both require a brief initial connection, but once data is materialised locally, all subsequent queries are free.

    What is ADBC and why is it faster than ODBC?
    ADBC keeps data in columnar format throughout. ODBC forces columnar → row → columnar conversion. DuckDB benchmarks show ADBC up to 38x faster for transfers.

    How much can I realistically save?
    For short, high-frequency dashboard queries: 70–90% is consistent across documented cases. The 60-second minimum means a 4-second query costs 15x what it should.

    Is DuckDB production-ready?
    For single-node analytical workloads under a few terabytes: yes. Multi-user concurrency at scale and enterprise governance: not yet.

    Do I need Iceberg?
    No. ADBC works with native Snowflake tables. The main friction is private key encoding, which the docs don’t explain well.

    Will this work with dbt?
    Yes. dbt-duckdb lets you run your full dbt project locally against DuckDB. Pull source data once from Snowflake, develop and test for free, deploy to Snowflake in production only.This eliminates cloud compute costs for the entire development loop. I’ve written about dbt native projects and pipeline patterns if you want more context on how this fits into a Snowflake-first stack.


    Related blogs :

    → DuckDB official docs and installation
    → DuckDB ADBC benchmarks
    → Apache Arrow project
    → Snowflake query_history view docs
    → dbt-duckdb adapter on GitHub
    → Greybeam ADBC connection code on GitHub

  • Why I Stopped Using Snowflake Tasks for Orchestration

    Why I Stopped Using Snowflake Tasks for Orchestration


    I want to be clear about something before I say anything critical: Snowflake Tasks are genuinely good. I used them for months. I recommended them to people. I wrote internal documentation about how to set them up.

    And then, slowly, quietly, I stopped reaching for them — and started reaching for Airflow instead.

    This isn’t a hit piece on Snowflake Tasks. It’s an honest look at where they work beautifully, where they start to crack, and the specific moment I realised I was fighting the tool instead of using it. If you’re in that same place right now — Tasks running fine on paper, increasingly painful in practice — this is for you.


    Why I Started With Tasks in the First Place

    The pitch for Snowflake Tasks is genuinely compelling: schedule and orchestrate your data pipelines without leaving Snowflake. No extra infrastructure. No Airflow server to maintain. No Docker containers. No YAML config files. Just SQL.

    For a solo data engineer or a small team running straightforward ELT pipelines that live entirely inside Snowflake, this is actually a great deal. You write a Task, you chain a few of them together, you set a cron schedule on the root, and the whole thing runs on serverless compute that Snowflake manages for you. Clean. Simple. Zero ops overhead.

    I built my first Task tree on a customer dimension pipeline — about 6 tasks chained together to handle raw landing, staging, SCD2 merge, and a downstream mart refresh. It worked perfectly. I was genuinely impressed.

    So I built more of them. And that’s where things started to get interesting.


    The First Sign Something Was Off

    The thing about Snowflake Tasks is that they look fine at small scale. Five tasks. Eight tasks. Even fifteen tasks chained together works reasonably well.

    The cracks start showing when your pipelines grow, when requirements get more complex, and when something goes wrong at 7am and you need to figure out what happened and why.

    My first real frustration was observability. When a Task fails, Snowflake logs it — but finding that log, understanding the full execution context, and connecting it to what came before and after requires digging through TASK_HISTORY in ACCOUNT_USAGE or calling INFORMATION_SCHEMA.TASK_HISTORY(). There’s no single screen that shows you, visually, what ran, what passed, what failed, and what the downstream impact was.

    Compare that to opening the Airflow UI, clicking into a DAG run, and seeing every task coloured green or red with full logs one click away. The difference in time-to-diagnosis is not small. I once spent 40 minutes reconstructing a failed task tree execution from TASK_HISTORY queries that would have taken me 3 minutes in Airflow.

    -- How you debug a failed Snowflake Task
    SELECT
        name,
        state,
        scheduled_time,
        completed_time,
        error_code,
        error_message
    FROM TABLE(INFORMATION_SCHEMA.TASK_HISTORY(
        SCHEDULED_TIME_RANGE_START => DATEADD('hour', -6, CURRENT_TIMESTAMP()),
        RESULT_LIMIT => 100
    ))
    WHERE name ILIKE '%customer_dim%'
    ORDER BY scheduled_time DESC;

    That query works. But it’s not a dashboard. It’s archaeology.


    The Retry Problem

    This one hurt me in production.

    Snowflake Tasks have basic retry configuration — you can set SUSPEND_TASK_AFTER_NUM_FAILURES to pause a task after repeated failures, which is useful. But what you can’t do natively is retry a specific failed task in the middle of a tree and resume from that point forward.

    If Task 6 in a 10-task chain fails, you fix the problem, and you want to re-run from Task 6 onwards — you’re doing it manually. You can resume the root task, but it re-runs everything from the beginning on the next scheduled tick. Or you run Task 6’s SQL manually, then manually kick off Task 7, Task 8… you see where this is going.

    In Airflow, you right-click the failed task node, click “Clear”, and it re-runs that task and everything downstream. That’s it. One click. No manual intervention, no risk of accidentally re-running something upstream that already completed correctly and shouldn’t run twice.

    For pipelines with expensive upstream tasks — large MERGE operations, heavy aggregations — re-running from the beginning when only a downstream step failed is both wasteful and risky. Wasteful because you’re burning compute credits on work already done. Risky because some operations are not safely idempotent and running them twice produces wrong results.


    The Conditional Logic Wall

    Here’s the limitation that finally pushed me to switch.

    My pipelines started needing branching logic. Specifically: run the full pipeline on weekdays, run a lighter version on weekends. Or: if the row count from the previous step is zero, skip the downstream merge and send an alert instead of running an empty MERGE that silently succeeds.

    In Airflow, this is a BranchPythonOperator. Three lines of Python. Clean, explicit, version-controlled.

    In Snowflake Tasks, this requires workarounds. You can use a stored procedure with SYSTEM$TASK_DEPENDENTS_ENABLE logic, or try to simulate branching with conditional stored procedures that check a flag and decide whether to execute. It works — technically — but it’s brittle, hard to read, and the logic is buried inside a stored procedure rather than visible in the orchestration layer where it belongs.

    Snowflake Tasks can only execute SQL statements and stored procedures. For more complex logic in Python, Java, or other languages, external schedulers are required. Flexera

    When your pipeline logic is entirely SQL, Tasks are fine. The moment you need to make an orchestration decision based on runtime data — not just “did this succeed or fail” but “what did this return, and what should I do about it” — you’re working against the grain.


    The Scale Limit Nobody Mentions

    There is a hard limit of 1,000 Tasks per data pipeline. For very large implementations this is an issue and you need to split out the data pipeline into multiple separate data pipelines as a workaround. Sonra

    Most teams won’t hit 1,000 tasks. But if you’re building a platform for multiple teams — separate pipelines per business domain, each with their own task trees — you will eventually bump into governance and management complexity that a 1,000-task-per-pipeline limit doesn’t help with.

    More practically: managing dozens of separate task trees, each owned by a different role, with different schedules, different failure behaviours, and no unified view across all of them — is hard. There’s no Snowflake-native equivalent of Airflow’s DAG list view where you can see all pipelines, their last run status, and their next scheduled run in one place.


    What I Use Instead — And Why

    I switched to Airflow with the SQLExecuteQueryOperator for Snowflake, and I’ve written about this setup in depth in my post How I Wired Snowflake’s Native dbt Projects to Airflow. The short version of why it works better for me:

    Airflow owns orchestration. Snowflake owns execution. That’s the right division of responsibility. Airflow is purpose-built for DAG management, dependency handling, retries, branching, alerting, and observability. Snowflake is purpose-built for data processing at scale. Letting each tool do what it’s best at — instead of asking Snowflake Tasks to be a general-purpose orchestrator — is the cleaner architecture.

    Here’s the pattern I use for a typical pipeline:

    from airflow import DAG
    from airflow.providers.snowflake.operators.snowflake import SQLExecuteQueryOperator
    from airflow.operators.python import BranchPythonOperator
    from datetime import datetime, timedelta
    
    with DAG(
        dag_id='customer_dimension_pipeline',
        schedule_interval='0 6 * * *',
        start_date=datetime(2024, 1, 1),
        catchup=False,
    ) as dag:
    
        load_raw = SQLExecuteQueryOperator(
            task_id='load_raw_customers',
            conn_id='snowflake_analytics',
            sql="CALL raw.sp_load_customers();",
        )
    
        validate_raw = SQLExecuteQueryOperator(
            task_id='validate_raw_row_count',
            conn_id='snowflake_analytics',
            sql="""
                SELECT CASE
                    WHEN COUNT(*) = 0 THEN 1/0  -- Forces task failure if no rows
                    ELSE COUNT(*)
                END FROM raw.customers_staging
                WHERE load_date = CURRENT_DATE();
            """,
        )
    
        run_scd2_merge = SQLExecuteQueryOperator(
            task_id='run_scd2_merge',
            conn_id='snowflake_analytics',
            sql="CALL transforms.sp_customer_scd2_merge();",
        )
    
        refresh_mart = SQLExecuteQueryOperator(
            task_id='refresh_customer_mart',
            conn_id='snowflake_analytics',
            sql="CALL marts.sp_refresh_customer_summary();",
        )
    
        load_raw >> validate_raw >> run_scd2_merge >> refresh_mart

    If validate_raw fails because there are zero rows, the pipeline stops. run_scd2_merge never runs. I get an Airflow alert. I can clear and rerun just validate_raw and everything downstream once the issue is fixed — without touching load_raw again.

    That conditional validation step alone — stopping a pipeline when upstream data is missing — was nearly impossible to implement cleanly with Tasks. With Airflow it’s a forced division by zero in the validation SQL. Ugly, but effective. There are cleaner ways with ShortCircuitOperator too.


    To Be Fair: When Snowflake Tasks Are Still the Right Choice

    I don’t want this to read as “never use Tasks.” That’s not what I’m saying.

    Tasks are still my first choice for:

    Micro-refresh patterns. A Task + Stream combination for near-real-time SCD2 updates — triggering only when the stream has data — is elegant and genuinely hard to replicate cleanly in Airflow. I covered exactly this pattern in my post Snowflake Streams and Tasks for SCD2 — How I Actually Use Them.

    Simple scheduled SQL. A single SQL statement that needs to run every 30 minutes with no dependencies? Task all the way. Zero ops overhead for maximum simplicity.

    Snowflake-only pipelines with no external context. If your pipeline never needs to know anything about the outside world — no API calls, no file system checks, no cross-system dependencies — Tasks keep everything in one place.

    Small teams with no existing orchestration infrastructure. If you’re a team of two data engineers and setting up Airflow feels like overkill, Tasks get you 80% of the way there with 10% of the setup cost.

    The honest decision framework:

    ScenarioUse
    Simple scheduled SQL, no branchingSnowflake Tasks
    Stream-triggered incremental loadsSnowflake Tasks + Streams
    Multi-step pipeline with retry requirementsAirflow
    Conditional branching based on row countsAirflow
    Pipelines touching external systemsAirflow
    Cross-team pipelines needing unified observabilityAirflow
    Large task trees (50+ steps)Airflow

    The Honest Summary

    I didn’t stop using Snowflake Tasks because they’re bad. I stopped using them as my primary orchestration layer because that’s not what they’re optimised for — and I was asking them to do a job they weren’t built to do well.

    Snowflake Tasks handle most data orchestration patterns, but the real question is who actually plays well with Snowflake as more than just a place to send SQL. Monte Carlo Tasks are Snowflake’s answer to simple scheduling. Airflow is the answer to complex orchestration. Knowing which problem you actually have is the whole game.

    If your pipelines are growing, your failure debugging is getting slower, and you’ve found yourself writing stored procedures to simulate branching logic — that’s the sign. That’s the moment I had. The switch to Airflow was a weekend of setup and a week of migration, and I haven’t looked back.

  • 2026 Guide: Snowflake Cortex Code Cost Control

    2026 Guide: Snowflake Cortex Code Cost Control

    When I first started using Cortex Code, cost was the last thing on my mind. It’s right there in the Snowsight UI, it feels like a built-in feature, and nothing in the interface tells you that tokens are being consumed behind the scenes.

    Then I went digging through the official docs properly — not just the feature overview, but the cost controls section — and found something I hadn’t seen covered anywhere: Snowflake has a native, dedicated cost control system for Cortex Code that most people don’t know exists. Two specific parameters. Per-user. Per-surface. Configurable by any ACCOUNTADMIN in seconds.

    This article is specifically about that. If you’re running Cortex Code in your Snowflake account — especially with a team — you need to set these up.


    How Cortex Code Billing Works

    If you’re new to Cortex Code, I’d recommend reading my earlier post What Is Snowflake Cortex Code and How I Taught Myself to Use It first — it covers what the tool actually does and how to access it in Snowsight. This article picks up at the cost control layer.

    Cortex Code runs on two surfaces, each with its own billing model:

    Cortex Code in Snowsight — the browser-based UI. Token-based billing tied to your existing Snowflake account. Snowflake will notify users before charges formally begin, but AI service costs are already accumulating underneath.

    Cortex Code CLI — the command-line agent. Already billing via token consumption — pay-as-you-go for existing Snowflake accounts, or a subscription model for individual sign-ups.

    The key thing to understand: unlike virtual warehouses where you can set a Resource Monitor with a credit quota, Cortex Code has its own separate cost control parameters that standard Resource Monitors don’t cover. You need to set these explicitly.

    For the full official billing breakdown, see Snowflake’s Cortex Code billing documentation.


    The Two Parameters You Need to Know

    Snowflake provides exactly two parameters for Cortex Code cost control. Both are set by ACCOUNTADMIN only, operate on a rolling 24-hour window per user, and are completely independent of each other.

    ParameterSurface ControlledDefault
    CORTEX_CODE_CLI_DAILY_EST_CREDIT_LIMIT_PER_USERCortex Code CLI-1 (unlimited)
    CORTEX_CODE_SNOWSIGHT_DAILY_EST_CREDIT_LIMIT_PER_USERCortex Code in Snowsight-1 (unlimited)

    And here’s what the values actually mean:

    ValueBehaviour
    -1 (default)No limit — unlimited access
    0Access blocked entirely for that user
    Any positive numberAccess blocked once estimated credits exceed that number in the past 24 hours

    Full parameter reference: Cost controls for Cortex Code — Snowflake Docs


    Setting Account-Level Limits

    This is the first thing I’d do — set a sensible default for all users at the account level. If someone goes over it, they’re blocked until the rolling window resets. No silent runaway spend.

    -- Cap all users at 20 credits per day for CLI
    ALTER ACCOUNT SET CORTEX_CODE_CLI_DAILY_EST_CREDIT_LIMIT_PER_USER = 20;
    
    -- Cap all users at 20 credits per day in Snowsight
    ALTER ACCOUNT SET CORTEX_CODE_SNOWSIGHT_DAILY_EST_CREDIT_LIMIT_PER_USER = 20;

    To remove an account-level limit and restore unlimited access:

    ALTER ACCOUNT UNSET CORTEX_CODE_CLI_DAILY_EST_CREDIT_LIMIT_PER_USER;
    ALTER ACCOUNT UNSET CORTEX_CODE_SNOWSIGHT_DAILY_EST_CREDIT_LIMIT_PER_USER;

    For the full ALTER ACCOUNT syntax reference, see ALTER ACCOUNT — Snowflake Docs.


    Setting Per-User Limits (Override)

    User-level settings override the account-level setting for that specific user. Everyone else keeps the account default.

    -- Power user gets a higher CLI limit than the account default
    ALTER USER power_user SET CORTEX_CODE_CLI_DAILY_EST_CREDIT_LIMIT_PER_USER = 50;
    
    -- Junior analyst gets a tighter Snowsight limit
    ALTER USER junior_analyst SET CORTEX_CODE_SNOWSIGHT_DAILY_EST_CREDIT_LIMIT_PER_USER = 5;
    
    -- Block a service account or contractor from Snowsight entirely
    ALTER USER contractor_account SET CORTEX_CODE_SNOWSIGHT_DAILY_EST_CREDIT_LIMIT_PER_USER = 0;

    Setting the value to 0 blocks access completely for that user on that surface. Useful for service accounts that should never be touching Cortex Code interactively.

    To remove a user-level override and fall back to the account default:

    ALTER USER junior_analyst UNSET CORTEX_CODE_SNOWSIGHT_DAILY_EST_CREDIT_LIMIT_PER_USER;

    See ALTER USER — Snowflake Docs for the full syntax.


    A Practical Setup for a Real Team

    Here’s how I’d configure this for a typical data engineering team — let’s say you have senior engineers, analysts, and a few service accounts:

    -- Step 1: Set sensible defaults for the whole account
    ALTER ACCOUNT SET CORTEX_CODE_CLI_DAILY_EST_CREDIT_LIMIT_PER_USER = 20;
    ALTER ACCOUNT SET CORTEX_CODE_SNOWSIGHT_DAILY_EST_CREDIT_LIMIT_PER_USER = 20;
    
    -- Step 2: Give senior engineers more headroom on CLI
    ALTER USER senior_eng_1 SET CORTEX_CODE_CLI_DAILY_EST_CREDIT_LIMIT_PER_USER = 50;
    ALTER USER senior_eng_2 SET CORTEX_CODE_CLI_DAILY_EST_CREDIT_LIMIT_PER_USER = 50;
    
    -- Step 3: Block service accounts from both surfaces
    ALTER USER airflow_svc SET CORTEX_CODE_CLI_DAILY_EST_CREDIT_LIMIT_PER_USER = 0;
    ALTER USER airflow_svc SET CORTEX_CODE_SNOWSIGHT_DAILY_EST_CREDIT_LIMIT_PER_USER = 0;
    
    ALTER USER dbt_svc SET CORTEX_CODE_CLI_DAILY_EST_CREDIT_LIMIT_PER_USER = 0;
    ALTER USER dbt_svc SET CORTEX_CODE_SNOWSIGHT_DAILY_EST_CREDIT_LIMIT_PER_USER = 0;

    If you’re using dbt projects natively inside Snowflake, you’ll already have dedicated service accounts set up — I covered that in detail in How I Wired Snowflake’s Native dbt Projects to Airflow. Those same service accounts should be blocked from Cortex Code with a 0 limit.

    Service accounts should never be using Cortex Code. Setting them to 0 explicitly means even if someone accidentally grants CORTEX_AGENT_USER to a service role, the credit limit acts as a safety net.


    Auditing Who Has Custom Limits

    Snowflake provides a script in the official docs to list all users with per-user overrides. This is exactly what you need for a quarterly governance review:

    -- Audit all users with a custom CLI credit limit override
    EXECUTE IMMEDIATE $$
    DECLARE
      current_user STRING;
      rs_users RESULTSET;
      res      RESULTSET;
    BEGIN
      CREATE OR REPLACE TEMPORARY TABLE _param_overrides (user_name STRING, param_value STRING);
    
      SHOW USERS;
      rs_users := (SELECT "name" FROM TABLE(RESULT_SCAN(LAST_QUERY_ID())));
    
      FOR record IN rs_users DO
        current_user := record."name";
    
        EXECUTE IMMEDIATE
          'SHOW PARAMETERS LIKE ''CORTEX_CODE_CLI_DAILY_EST_CREDIT_LIMIT_PER_USER'' IN USER "' || :current_user || '"';
    
        INSERT INTO _param_overrides (user_name, param_value)
          SELECT :current_user, "value"
          FROM TABLE(RESULT_SCAN(LAST_QUERY_ID()))
          WHERE "level" = 'USER';
      END FOR;
    
      res := (SELECT * FROM _param_overrides);
      RETURN TABLE(res);
    END;
    $$;

    To audit Snowsight overrides instead, swap CORTEX_CODE_CLI_DAILY_EST_CREDIT_LIMIT_PER_USER for CORTEX_CODE_SNOWSIGHT_DAILY_EST_CREDIT_LIMIT_PER_USER in the SHOW PARAMETERS LIKE clause. Run both regularly. This script is taken directly from the official Snowflake cost controls page — I haven’t modified it.


    What Happens When a User Hits Their Limit

    When a user’s rolling 24-hour usage exceeds the configured threshold, that surface returns an error telling them the daily credit limit has been reached. They can’t use it again until enough time passes for usage to drop below the limit. The other surface is unaffected — hitting the CLI limit doesn’t lock them out of Snowsight.

    Admins can adjust or remove the limit at any time to restore access immediately, without waiting for the window to roll.

    This is worth communicating to your team before you set limits — nobody likes being surprised by a sudden block mid-workflow. Set the limits, tell people what they are, and tell them who to contact if they need a temporary increase.


    One More Layer: Monitor Actual Usage Too

    The credit limits control access — they stop runaway spend. But you also want visibility into what’s being consumed before limits are hit. Pair your limits with a monitoring query:

    sql

    SELECT
        DATE_TRUNC('day', start_time)   AS usage_day,
        function_name,
        model_name,
        SUM(tokens_used)                AS total_tokens,
        SUM(credits_used)               AS total_credits
    FROM snowflake.account_usage.cortex_functions_usage_history
    WHERE start_time >= CURRENT_DATE - 30
    GROUP BY 1, 2, 3
    ORDER BY usage_day DESC, total_credits DESC;

    This tells you which models and functions are driving costs across your account. The credit limits stop the bleeding. This query tells you where the bleeding is coming from.

    For a deeper look at how Snowflake bills for AI functions generally — token rates, model pricing differences, and what “output tokens” means for your bill — see my post Snowflake Cortex Pricing — What Every Data Engineer Should Know and the official Snowflake Service Consumption Table which is the source of truth for current credit rates.


    The Bottom Line

    The right Cortex Code cost control setup in 2026 takes about 10 minutes and covers you completely:

    1. Set account-level daily limits for both CLI and Snowsight — start with 20 credits as a reasonable default
    2. Override upward for power users who legitimately need more headroom
    3. Override to 0 for all service accounts
    4. Run the audit script quarterly to catch drift as people join or leave the team
    5. Monitor cortex_functions_usage_history for spend patterns alongside your limits

    The alternative is the default: -1 for every user, unlimited spend, and a bill that arrives before anyone realized the meter was running.

  • Snowflake Interview Questions — Expert Level

    Snowflake Interview Questions — Expert Level

    After all of this, the real tell at the senior level isn’t whether you know all these answers. It’s whether you can connect them.

    The best signal a senior candidate gives is when they answer one question and naturally reference another. “That’s the same clustering depth issue I mentioned earlier — the root cause is the same even though the symptoms look different.” That kind of connected thinking is what separates someone who’s read about Snowflake from someone who’s been debugging production issues at midnight.

    These aren’t different topics. They’re the same platform, looked at from different angles. That’s what the senior interview is actually testing.

    Good luck — and if any of these scenarios show up in your interview and you crack it, come back and let me know.

    I’ve been on both sides of the Snowflake interview table now. I’ve been the nervous one trying to remember what CLUSTER_BY actually does under pressure, and I’ve been the one reviewing candidates who clearly memorized a blog post instead of actually understanding the platform.

    There’s a massive gap between what the generic “Top 50 Snowflake Interview Questions” articles prepare you for and what a senior data engineering interview actually feels like. This article is about that gap.

    Nobody at the senior level is going to ask you “what is a virtual warehouse.” What they’re going to do is describe a production scenario — something that went wrong, or a design problem they’re genuinely wrestling with — and see how you think through it. They want to know if you’ve actually been inside the query profile. If you’ve ever had a Snowpipe backlog and had to figure out why. If you understand what “spilling to remote storage” means for your credit bill.

    This is the article I wish existed when I was preparing. I’m writing it from everything I’ve learned building real things on Snowflake — not from documentation tabs.


    How to Read This Article

    I’ve organized questions into six categories that match how real interviews are structured:

    1. Architecture & Internals
    2. Performance Tuning & Query Optimization
    3. Cost Management & Credits
    4. Data Ingestion & Pipelines
    5. Security, Governance & RBAC
    6. Real-World Scenario Questions (the hardest ones)

    Each question comes with what a strong answer actually sounds like — not bullet points you memorize, but the kind of connected thinking that signals you’ve done this for real.


    Category One: Architecture & Internals

    These questions separate people who’ve read the docs from people who understand the engine.


    Q1. Walk me through what actually happens when a query hits Snowflake — from the moment I press Run to the moment results appear.

    This is the architecture question that trips up even experienced people because they know the three layers but can’t connect them into a real execution story.

    A strong answer: When you submit a query, it first hits the Cloud Services layer — this is Snowflake’s brain. It authenticates you, parses the SQL, checks metadata in the global metadata store (which knows exactly which micro-partitions hold your data without touching a warehouse), and builds a query execution plan. This whole step runs without your virtual warehouse even waking up.

    If the results of this exact query are already in the result cache — and the underlying data hasn’t changed — Snowflake returns those results instantly from Cloud Services. Zero warehouse credits consumed. This is why identical repeated queries on unchanged data are free.

    If there’s no cached result, the query gets dispatched to your virtual warehouse — the compute layer. The warehouse spins up worker nodes (if suspended, there’s a small cold-start cost), and each node fetches the relevant micro-partitions from cloud storage — S3, Azure Blob, or GCS. This data gets loaded into the warehouse’s local SSD cache (the local disk cache). The workers process it in parallel using MPP (Massively Parallel Processing), and results flow back to the Cloud Services layer, which caches them in the result cache, then returns them to you.

    The key insight: storage is completely decoupled from compute. Multiple warehouses can read the same data simultaneously without contention. This is the fundamental reason Snowflake handles concurrent workloads better than traditional MPP databases.


    Q2. Explain micro-partitioning. How does it actually affect query performance and what can go wrong?

    Snowflake automatically divides table data into micro-partitions — immutable, compressed columnar files, typically between 50 and 500 MB of uncompressed data. Each micro-partition stores metadata: min and max values for each column, number of distinct values, null counts.

    When a query runs with a WHERE clause, Snowflake uses this metadata to prune micro-partitions — skipping any partition where the filter condition can’t possibly match. If your query is WHERE order_date = '2024-01-15', Snowflake checks the min/max metadata on order_date for each micro-partition and skips any partition where 2024-01-15 is outside that range. It never reads those partitions.

    The failure mode: if data is poorly clustered — meaning order_date values are scattered randomly across hundreds of micro-partitions — pruning becomes ineffective. Snowflake has to read most of the table to find your rows. This is when you see large “Bytes scanned” numbers in the query profile despite a tight filter.

    Check clustering health with:

    SELECT SYSTEM$CLUSTERING_INFORMATION(
        'orders',
        '(order_date)'
    );

    Look at average_depth. A value close to 1.0 means excellent pruning. Higher values mean data is scattered and clustering is poor.

    What can make clustering degrade over time: heavy DML operations (lots of individual INSERTs or UPDATEs) scatter data across new micro-partitions without regard to clustering order. For tables that receive continuous small inserts, consider a Clustering Key with automatic reclustering enabled.


    Q3. What is the difference between Automatic Clustering and manually running CLUSTER BY?

    When you define a clustering key on a table, Snowflake can maintain it in two ways.

    Manually triggering ALTER TABLE orders CLUSTER BY (order_date) runs a one-time reclustering operation. It reorganizes existing micro-partitions to improve clustering depth on the specified column. It costs credits and is a point-in-time fix.

    Automatic Clustering is a continuous background service. Once enabled, Snowflake monitors clustering depth and automatically triggers reclustering operations when it detects degradation — typically after significant DML activity. You pay for the compute, but you don’t manage it.

    When should you use automatic clustering vs. not? Automatic clustering makes sense for large tables (generally multi-TB+) that receive continuous writes and are frequently queried with filters on a high-cardinality column. For smaller tables, or tables that are mostly read-only after initial load, the cost of continuous clustering may not justify the query performance improvement. Always benchmark before enabling it.

    A practical note: clustering keys should have high cardinality (lots of distinct values) but not too high — date columns, category columns with dozens of values, and region columns are good candidates. Using a UUID or primary key as a clustering key is usually a mistake — too many distinct values means micro-partitions can’t contain meaningful ranges.


    Q4. Describe the three types of caching in Snowflake and the situations where each one helps you — or fails you.

    This is a question where depth matters more than breadth.

    Result Cache (Cloud Services layer): Stores the complete output of recent queries for up to 24 hours. If you run the same query and the underlying data hasn’t changed, you get the result back instantly with zero warehouse compute. The “data hasn’t changed” check is metadata-based — if any DML touched the relevant tables, the result cache is invalidated.

    Where it fails: if your query uses non-deterministic functions like CURRENT_TIMESTAMP(), RANDOM(), or CURRENT_USER() — or if a WHERE clause references CURRENT_DATE() — the result cache is bypassed entirely because the result could differ on each execution. This is a common gotcha: wrapping a query in a scheduled task that adds WHERE load_date = CURRENT_DATE() means the result cache never kicks in.

    Local Disk Cache (Warehouse layer): When worker nodes fetch micro-partitions from cloud storage, they cache them on local SSD. Subsequent queries that touch the same micro-partitions within the same warehouse session can read from local disk rather than going back to object storage — significantly faster. This cache is per-warehouse and is lost when the warehouse suspends.

    Where it fails: if you suspend your warehouse and resume it, the local cache is cold. For dashboards or BI tools that run queries on a fixed schedule against a specific warehouse, keeping that warehouse running between query bursts (or using auto-resume with a short suspend timeout) preserves the local cache and makes repeated runs faster.

    Metadata Cache (Cloud Services layer): Snowflake maintains metadata about table structure, row counts, min/max values, and partition information entirely in memory in the Cloud Services layer. Many COUNT(*), MIN(), and MAX() queries on simple tables can be answered from metadata alone — without engaging a warehouse at all.

    Where it helps the most: SELECT COUNT(*) FROM large_table that would cost credits on any other platform is often free on Snowflake.


    Category 2: Performance Tuning & Query Optimization


    Q5. You inherit a query that takes 45 minutes. Walk me through your exact diagnostic process.

    A senior candidate’s answer is a structured diagnostic process, not a list of tips.

    Step 1: Open the Query Profile. In Snowsight, find the query in query history and click into the profile. The profile shows you a DAG of query operators — where time was spent, which operator consumed the most credits, and critically: whether there was spillage.

    Step 2: Check for spillage first. Look for “Bytes spilled to local storage” or “Bytes spilled to remote storage” in the query profile. Spillage means the warehouse ran out of memory and had to write intermediate results to disk. Remote spillage is particularly expensive — it means even local SSD wasn’t enough. This is your first escalation trigger. If you see remote spillage, the warehouse is too small for this query. Upsizing the warehouse (SMALL → MEDIUM → LARGE) doubles memory at each step and can eliminate spillage entirely. Upsizing isn’t always the right long-term answer, but it tells you quickly whether memory is the constraint.

    Step 3: Check partition pruning. Look at “Partitions scanned” vs “Partitions total.” If the ratio is high (e.g., 2.4M / 2.5M partitions scanned), the query is doing a near-full table scan. This means either no clustering key, wrong clustering key, or the filter column isn’t selective. Add a filter on a well-clustered column, or reconsider the clustering key.

    Step 4: Check for cartesian joins or bad join order. In the profile DAG, look at the row counts flowing between join operators. If an intermediate join is exploding rows to billions before a later filter reduces them, the query logic needs rewriting — push filters earlier or restructure CTEs.

    Step 5: Check for repeated subqueries. A correlated subquery inside a CASE statement or a nested SELECT that re-executes per row is a classic SQL performance killer. Rewrite using window functions or pre-aggregated CTEs.

    Step 6: Check the warehouse size vs. the data volume. Snowflake’s query optimizer makes different decisions at different warehouse sizes. An XL warehouse processes 16x the data per unit time as an XS. For one-off heavy queries, temporarily bumping to a larger warehouse is often cheaper (faster completion = fewer total credits) than running a smaller warehouse for much longer.


    Q6. Scenario: A dbt model that was running in 8 minutes is now taking 35 minutes. Nothing in the SQL changed. What do you investigate?

    This is the scenario that makes senior candidates shine because it forces them to think about things outside the SQL.

    The first question I’d ask: did anything change in the data, even if the SQL didn’t change? If row volume doubled (new data source onboarded, historical backfill ran), the same query legitimately takes longer. Check row counts in the source tables against last month.

    Second: did the warehouse change? If the warehouse was downsized or its MAX_CLUSTER_COUNT was reduced, the same query runs with less compute. Check warehouse history.

    Third: clustering degradation. If the table receives continuous inserts and has a clustering key, clustering depth degrades over time as new micro-partitions are added in insertion order rather than cluster order. Run SYSTEM$CLUSTERING_INFORMATION and compare to a baseline. If average_depth climbed significantly, reclustering is overdue.

    Fourth: check if a new index of activity appeared in QUERY_HISTORY around the same time the slowdown started. Sometimes a new workload hitting the same warehouse — a new analyst running heavy queries during your pipeline window — causes credit contention that slows your job. The fix is workload isolation: move the pipeline to a dedicated warehouse.

    Fifth: run the query on a larger warehouse and compare. If it’s proportionally faster, it’s a compute-bound problem. If it’s not faster, it’s a data or SQL problem.


    Q7. What is a multi-cluster warehouse, and when would you specifically choose NOT to use one?

    A multi-cluster warehouse allows Snowflake to spin up additional warehouse clusters automatically when queuing occurs — concurrency scaling for a single warehouse. Instead of queries waiting in line because the single-cluster warehouse is at max concurrency, a second (or third, up to your max setting) cluster spins up to serve additional queries.

    When to use it: high-concurrency workloads where many users or processes are querying simultaneously. BI tools with dozens of dashboards refreshing at the same time. ETL pipelines that run many parallel tasks. Anything where queries are frequently queuing.

    When NOT to use it: for a single-user or low-concurrency workload, multi-cluster adds cost with no benefit — you’re paying for cluster management overhead. Also: multi-cluster doesn’t help a single slow query. One 45-minute query does not benefit from a second cluster spinning up because the bottleneck is compute for that one query, not concurrency. For that scenario, you want a larger single-cluster warehouse, not more clusters.

    The billing nuance: each active cluster in a multi-cluster warehouse is billed separately. A MEDIUM warehouse running 3 clusters simultaneously costs 3x a single MEDIUM warehouse. This is appropriate and expected for high-concurrency workloads — but if multi-cluster is accidentally enabled on a warehouse that doesn’t need it, you’ll see credits spike unnecessarily.


    Category 3: Cost Management & Credits


    Q8. Scenario: Your Snowflake credits spiked 3x last Tuesday vs. the previous Tuesday. How do you diagnose this?

    This is a real operational scenario and interviewers love it because it tests whether you know ACCOUNT_USAGE views.

    -- Step 1: Identify which warehouses drove the spike
    SELECT
        warehouse_name,
        DATE(start_time)            AS usage_date,
        SUM(credits_used)           AS total_credits
    FROM snowflake.account_usage.warehouse_metering_history
    WHERE start_time >= DATEADD('day', -14, CURRENT_TIMESTAMP())
    GROUP BY warehouse_name, DATE(start_time)
    ORDER BY usage_date, total_credits DESC;

    This immediately tells you which warehouse(s) were responsible.

    -- Step 2: Drill into queries on that warehouse during the spike window
    SELECT
        query_id,
        query_text,
        user_name,
        warehouse_name,
        execution_time / 1000       AS execution_seconds,
        bytes_spilled_to_remote_storage,
        bytes_scanned,
        credits_used_cloud_services
    FROM snowflake.account_usage.query_history
    WHERE warehouse_name = 'TRANSFORM_WH'
      AND DATE(start_time) = '2024-01-16'
    ORDER BY execution_time DESC
    LIMIT 20;

    Look for: unusually long execution times, massive bytes_scanned values, or high bytes_spilled_to_remote_storage — all signals of expensive queries. Also check if the number of queries is simply higher (new pipeline deployed, more users added).

    -- Step 3: Check if a new pipeline or user caused the spike
    SELECT
        user_name,
        COUNT(*) AS query_count,
        SUM(execution_time) / 1000 AS total_execution_seconds
    FROM snowflake.account_usage.query_history
    WHERE DATE(start_time) = '2024-01-16'
    GROUP BY user_name
    ORDER BY total_execution_seconds DESC;

    Once you identify the culprit, add a Resource Monitor to prevent recurrence:

    CREATE OR REPLACE RESOURCE MONITOR transform_wh_monitor
      WITH CREDIT_QUOTA = 500
      FREQUENCY = MONTHLY
      START_TIMESTAMP = IMMEDIATELY
      TRIGGERS
        ON 80 PERCENT DO NOTIFY
        ON 100 PERCENT DO SUSPEND_IMMEDIATE;
    
    ALTER WAREHOUSE transform_wh SET RESOURCE_MONITOR = transform_wh_monitor;

    Q9. What’s the practical difference between Transient and Permanent tables, and when would misusing them burn you?

    Permanent tables have full Time Travel (up to 90 days) and a 7-day Fail-safe period after Time Travel expires. Both features consume storage — Snowflake keeps the historical micro-partitions for the retention window.

    Transient tables have Time Travel capped at 1 day, and zero Fail-safe. They cost less storage because there’s no extended historical data being maintained.

    When transient tables make sense: staging tables and intermediate transformation tables that you rebuild completely each run. If a table is truncated and reloaded daily, paying for 90 days of Time Travel on that data is pure waste.

    The burn scenario: someone creates all their core business tables as transient to save costs. Then a developer runs DELETE FROM customer_dim WHERE segment = 'Enterprise' with a bug in the WHERE clause — and instead of filtering to test accounts, it deletes 40,000 live customer records. With a permanent table, you’d recover with a Time Travel query in 5 minutes. With a transient table, if this happened 2 days ago, that data is gone. Fail-safe doesn’t exist for transient tables. Recovery now means a full source-system re-extract and reload.

    The right pattern: transient for staging and scratch tables. Permanent for anything that is a source of truth.


    Category #4: Data Ingestion & Pipelines


    Q10. You set up Snowpipe to ingest data from an S3 bucket. After two weeks it stops loading new files. Files are landing in S3 but not appearing in Snowflake. Walk me through your debugging process.

    Snowpipe is event-driven — it relies on S3 event notifications (or Snowflake’s AUTO_INGEST) to trigger. If notifications stop firing, files pile up silently.

    Step 1: Check the pipe status.

    SELECT SYSTEM$PIPE_STATUS('my_schema.my_pipe');

    Look at pendingFileCount, lastIngestedTimestamp, and executionState. If executionState is PAUSED or STOPPED, that’s your immediate answer.

    Step 2: Check for load errors.

    SELECT *
    FROM TABLE(INFORMATION_SCHEMA.COPY_HISTORY(
        TABLE_NAME => 'target_table',
        START_TIME => DATEADD('day', -3, CURRENT_TIMESTAMP())
    ))
    WHERE STATUS != 'Loaded'
    ORDER BY LAST_LOAD_TIME DESC;

    If files attempted to load and failed — schema mismatch, malformed JSON, wrong delimiter — Snowpipe silently skips them after the error threshold. Files that fail repeatedly get quarantined and stop triggering new notifications.

    Step 3: Check SQS queue (for S3 auto-ingest).

    For S3-triggered Snowpipe, Snowflake creates an SQS queue that receives S3 event notifications. If that SQS queue fills up (it has limits) or the S3 event notification configuration was accidentally removed, events stop flowing.

    Step 4: Check file format.

    If someone modified the source system’s file format — added a column, changed the delimiter from comma to pipe, switched from gzip to uncompressed — and the Snowpipe file format definition wasn’t updated, every new file fails silently.

    Prevention: set up a monitoring query as a scheduled Snowflake Task that alerts when lastIngestedTimestamp hasn’t updated in more than X hours. Don’t wait for a stakeholder to notice missing data.


    Q11. What is the difference between Streams and Tasks in Snowflake, and give me a real scenario where you’d use them together?

    A Stream is a change data capture object. It sits on top of a table and tracks INSERT, UPDATE, and DELETE operations that occurred since the stream was last consumed. It doesn’t store data — it stores metadata about what changed and where those changed rows are. Querying a stream returns the rows that changed and a METADATA$ACTION column telling you whether they were INSERTed, UPDATEd (shows as DELETE + INSERT), or DELETEd.

    A Task is a scheduled SQL executor. It runs a SQL statement on a schedule (cron-style) or as part of a task tree (dependency-based). Tasks can be configured to run only if a stream has data — this is the key integration.

    Real scenario — SCD Type 2 on a customer dimension:

    -- Stream on the raw customer table to capture changes
    CREATE OR REPLACE STREAM customer_changes_stream
      ON TABLE raw.customers;
    
    -- Task that runs every 15 minutes but only if stream has data
    CREATE OR REPLACE TASK apply_scd2_changes
      WAREHOUSE = transform_wh
      SCHEDULE = '15 MINUTE'
      WHEN SYSTEM$STREAM_HAS_DATA('customer_changes_stream')
    AS
      MERGE INTO dim_customers AS target
      USING (
          SELECT
              customer_id,
              customer_name,
              email,
              segment,
              METADATA$ACTION,
              METADATA$ISUPDATE
          FROM customer_changes_stream
          WHERE METADATA$ACTION = 'INSERT'
      ) AS changes
      ON target.customer_id = changes.customer_id
         AND target.is_current = TRUE
      WHEN MATCHED AND changes.METADATA$ISUPDATE = TRUE THEN
          UPDATE SET target.end_date = CURRENT_DATE(),
                     target.is_current = FALSE
      WHEN NOT MATCHED THEN
          INSERT (customer_id, customer_name, email, segment,
                  start_date, end_date, is_current)
          VALUES (changes.customer_id, changes.customer_name,
                  changes.email, changes.segment,
                  CURRENT_DATE(), NULL, TRUE);
    
    ALTER TASK apply_scd2_changes RESUME;

    The WHEN SYSTEM$STREAM_HAS_DATA() condition is critical — it means the task doesn’t consume warehouse credits on runs where nothing changed. The task wakes up, checks the condition, finds the stream empty, and goes back to sleep. Only on runs where actual customer data changed does it spin up the warehouse and execute the MERGE.


    Category Five: Security, Governance & RBAC


    Q12. Your security team asks you to ensure that the analytics team can see a salary column in aggregate reports but cannot see individual employee salaries. How do you implement this in Snowflake?

    This is a column-level security scenario. Snowflake has two tools here: Dynamic Data Masking and Secure Views.

    The cleaner approach for this scenario is Dynamic Data Masking:

    -- Create a masking policy
    CREATE OR REPLACE MASKING POLICY mask_salary
      AS (val NUMBER) RETURNS NUMBER ->
        CASE
            WHEN CURRENT_ROLE() IN ('HR_ADMIN', 'SYSADMIN') THEN val
            ELSE NULL   -- Or return 0, or a rounded bucket
        END;
    
    -- Apply it to the salary column
    ALTER TABLE employees
      MODIFY COLUMN salary
      SET MASKING POLICY mask_salary;

    Now anyone in HR_ADMIN sees actual salaries. Everyone else sees NULL in that column. The table structure is the same — no separate view to maintain, no duplication of logic across multiple views. The policy travels with the column.

    For the aggregate-only requirement (they can see AVG(salary) but not individual values), a Secure View is the right tool:

    CREATE OR REPLACE SECURE VIEW employee_salary_summary AS
    SELECT
        department,
        job_grade,
        COUNT(*) AS headcount,
        ROUND(AVG(salary), -3) AS avg_salary_rounded,  -- rounds to nearest 1000
        MIN(salary) AS min_salary,
        MAX(salary) AS max_salary
    FROM employees
    GROUP BY department, job_grade
    HAVING COUNT(*) >= 5;  -- don't expose groups with fewer than 5 employees

    The HAVING COUNT(*) >= 5 clause is deliberate — without it, a determined analyst could infer individual salaries by filtering to groups of one. This is basic statistical disclosure control, and senior candidates who mention it unprompted stand out.

    The SECURE keyword on the view hides the view definition from users who don’t own it — they can’t inspect the SQL to reverse-engineer what’s being hidden.

    Q13. Walk me through how you’d structure Snowflake RBAC for a mid-sized company — say 3 teams: Data Engineering, Analytics, and Finance BI.

    This is an architecture question. A weak answer lists roles. A strong answer describes a hierarchy.

    The Snowflake recommendation is a functional role hierarchy:

    ACCOUNTADMIN (Snowflake admin only — almost never used day-to-day)
        └── SYSADMIN
                ├── DE_ADMIN (Data Engineering admin role)
                │       ├── DE_DEVELOPER (read/write on raw + staging)
                │       └── DE_READER (read-only on staging)
                ├── ANALYTICS_ADMIN
                │       ├── ANALYTICS_DEVELOPER (read/write on marts)
                │       └── ANALYTICS_READER (read-only on marts)
                └── FINANCE_BI_ADMIN
                        └── FINANCE_BI_READER (read-only on finance mart)

    Each human user gets a user role (e.g., user_bug) which is granted the appropriate functional role. Users never log in as SYSADMIN or ACCOUNTADMIN. Those are break-glass roles used only for admin operations, and any use of them should generate an alert via ACCOUNT_USAGE.LOGIN_HISTORY.

    Object ownership matters: every object should be owned by a functional role (like DE_ADMIN), not by a personal user account. If the person who created the table leaves, their user is disabled — and if they owned the objects, those objects become inaccessible until ownership is transferred. This is a real operational problem that catches teams who didn’t plan RBAC carefully.

    Service accounts (Airflow, dbt, Fivetran) get dedicated roles scoped to exactly what they need. An Airflow service account does not need SYSADMIN. It needs USAGE on specific schemas, CREATE TABLE in specific schemas, and EXECUTE TASK if it manages Tasks. Nothing more.


    Category six: Real-World Scenario Questions

    These are the ones that show up in final-round interviews and engineering manager panels.


    Q14. A finance stakeholder tells you that a revenue number in their Snowflake dashboard changed retroactively — last Tuesday it showed $4.2M, today it shows $3.8M for the same date. How do you investigate and what do you tell the stakeholder?

    The first thing I’d do is use Time Travel to find out what the data actually looked like at the time they reported $4.2M:

    -- See what the table looked like when they saw 4.2M
    SELECT SUM(order_amount) AS total_revenue
    FROM orders
      AT (TIMESTAMP => '2024-01-16 09:00:00')  -- time they saw the report
    WHERE DATE(order_date) = '2024-01-09';
    
    -- Compare to current value
    SELECT SUM(order_amount) AS total_revenue
    FROM orders
    WHERE DATE(order_date) = '2024-01-09';

    If there’s a difference, the data changed between those two timestamps. Now find out what changed:

    -- Use a stream-style comparison or Time Travel diff
    SELECT order_id, order_amount
    FROM orders
      AT (TIMESTAMP => '2024-01-16 09:00:00')
    
    MINUS
    
    SELECT order_id, order_amount
    FROM orders;

    This gives you the rows that existed in the old version but were removed or changed. Cross-reference these order IDs with the source system — were they returns? Cancelled orders? A pipeline bug that incorrectly deleted records?

    What to tell the stakeholder: be transparent and specific. “The revenue figure changed because X orders were retroactively cancelled/corrected in the source system between Tuesday and today. Here are the specific order IDs. The current figure is correct.” Don’t just tell them the number changed — give them the reason and the evidence. Stakeholders who trust your data trust your investigative process.

    Q15. You’re designing a Snowflake data platform for a company that has data analysts, data scientists, and BI developers all sharing the same account. Each group runs different query patterns. How do you structure warehouses?

    Workload isolation is the core principle. Different query patterns compete for the same resources if they share a warehouse — and they do so invisibly, making debugging harder and degrading everyone’s experience.

    My recommended structure:

    Analysts warehouse (ANALYTICS_WH — MEDIUM, multi-cluster max 3): Handles ad-hoc analyst queries. These are typically medium-length, moderately complex, need fast turnaround. Multi-cluster handles concurrency spikes when everyone runs queries at once. Auto-suspend after 5 minutes.

    BI/Reporting warehouse (BI_WH — SMALL, auto-suspend 1 minute): Serves Tableau/Power BI dashboard queries. These are typically fast, repetitive, and benefit heavily from result cache. Small warehouse is fine because result cache handles most of the work. Short auto-suspend because dashboards tend to query in bursts.

    Data Science warehouse (DS_WH — LARGE or XL, auto-suspend 10 minutes): Data scientists run long-running exploratory queries, training feature pipelines, and large aggregations. These need raw compute power. Larger warehouse, longer suspend time to preserve local cache between iterative runs.

    Pipeline/ELT warehouse (TRANSFORM_WH — MEDIUM or LARGE, no auto-suspend during pipeline window): Dedicated to dbt, Airflow, and scheduled tasks. Isolated from user-facing workloads so a heavy pipeline run doesn’t slow down analyst queries. Configure Resource Monitor here to prevent runaway pipeline credits.

    The key point for interviewers: I explicitly call out that the BI warehouse is small because of result cache behavior. Most candidates say “make BI bigger for performance.” The correct answer is: BI workloads are naturally cache-friendly, so a small warm warehouse outperforms a large cold one for dashboard queries.


    Q16. Describe a time Snowflake’s architecture surprised you — either positively or negatively.

    I’ll give you a real one from my experience.

    I inherited a data pipeline where someone had set up a Snowflake Task tree — about 12 tasks chained together to run a nightly data quality suite. Each task ran a SQL check and wrote results to a DQ log table. The whole tree was supposed to complete in about 25 minutes.

    It was taking 3 hours.

    When I investigated, I found that the task tree was configured with a 1-minute schedule on the root task but the individual tasks had their dependencies set up as AFTER relationships. What nobody had realized: each task in the chain was waiting for its predecessor to complete AND for the next 1-minute schedule tick before executing. With 12 tasks, each waiting 30-45 seconds plus a 1-minute tick, the chain took enormously longer than expected.

    The fix was switching the root task to a scheduled trigger and ensuring the child tasks had AFTER relationships correctly configured as true dependencies (not schedule-based). The tree went from 3 hours to 22 minutes.

    What I learned: Snowflake’s Task scheduling documentation is clear, but the interaction between schedule-based and dependency-based triggers is easy to misconfigure in ways that aren’t immediately obvious. Test task trees with short intervals first, watch the execution history, and validate that the AFTER relationships are actually working as dependencies rather than just ordering constraints.


    Q17. How would you implement row-level security so that regional sales managers can only see data for their region?

    Row-level security in Snowflake is implemented with Row Access Policies.

    -- Create a mapping table: which role sees which regions
    CREATE OR REPLACE TABLE region_access_map (
        role_name  VARCHAR,
        region     VARCHAR
    );
    
    INSERT INTO region_access_map VALUES
        ('SALES_APAC', 'APAC'),
        ('SALES_EMEA', 'EMEA'),
        ('SALES_AMER', 'AMER'),
        ('SALES_ADMIN', 'APAC'),   -- Admin sees everything
        ('SALES_ADMIN', 'EMEA'),
        ('SALES_ADMIN', 'AMER');
    
    -- Create the row access policy
    CREATE OR REPLACE ROW ACCESS POLICY region_row_policy
      AS (row_region VARCHAR) RETURNS BOOLEAN ->
        CURRENT_ROLE() = 'SYSADMIN'
        OR EXISTS (
            SELECT 1 FROM region_access_map
            WHERE role_name = CURRENT_ROLE()
              AND region = row_region
        );
    
    -- Apply to the sales table
    ALTER TABLE sales_transactions
      ADD ROW ACCESS POLICY region_row_policy
      ON (region);

    Now when SALES_APAC queries sales_transactions, Snowflake invisibly adds a filter to every query on that table — they only see APAC rows. They can’t bypass it. They can’t see the policy definition if it’s secured.

    Important senior-level detail: Row Access Policies apply at query time, not at storage time. The data is stored together. The filter happens when Snowflake executes the query. This means you can’t accidentally expose data through joins to other tables — the policy applies to the base table regardless of how it’s joined.

    One gotcha: if a user’s role changes mid-session, the new policy takes effect on the next query. There’s no session-level snapshot of the policy evaluation.

    What Interviewers Are Really Looking For

    After all of this, the real tell at the senior level isn’t whether you know all these answers. It’s whether you can connect them.

    The best signal a senior candidate gives is when they answer one question and naturally reference another. “That’s the same clustering depth issue I mentioned earlier — the root cause is the same even though the symptoms look different.” That kind of connected thinking is what separates someone who’s read about Snowflake from someone who’s been debugging production issues at midnight.

    These aren’t different topics. They’re the same platform, looked at from different angles. That’s what the senior interview is actually testing.

    Good luck — and if any of these scenarios show up in your interview and you crack it, come back and let me know.

  • Orchestrating Snowflake dbt Projects with Airflow — End-to-End Pipeline Guide

    Orchestrating Snowflake dbt Projects with Airflow — End-to-End Pipeline Guide

    How I Wired Snowflake’s Native dbt Projects to Airflow — And Finally Got True End-to-End Orchestration


    I’ll be honest with you — for a long time I was running dbt the way most people run it. dbt Core installed on a server, profiles.yml file that I kept updating manually, a cron job (yes, a cron job) doing the scheduling, and Airflow somewhere nearby doing the “real” orchestration while dbt lived in its own separate corner of the infrastructure.

    It worked. It was fine. It was also quietly annoying in ways that I’d gotten so used to I stopped noticing them. Managing the dbt server separately. Keeping the Snowflake credentials synced in two places. Debugging failures by jumping between the Airflow UI, SSH logs on the dbt server, and Snowsight — all at once.

    Then Snowflake went GA with dbt Projects in November 2025, and I spent a weekend rebuilding the whole thing. This article is what I learned.

    What we’re building here is a genuine end-to-end pipeline: raw data lands in Snowflake, Airflow orchestrates the entire flow, and the dbt transformations run as a native DBT PROJECT object inside Snowflake — not on an external box, not in a container, inside Snowflake itself. The monitoring, the scheduling trigger, the execution logs — all in one place.

    Let’s build it from the ground up.


    First — What Exactly Is a dbt Project on Snowflake?

    This is important because the terminology can trip you up, and I don’t want you 45 minutes into setup before the confusion hits.

    dbt Projects on Snowflake let you use familiar Snowflake features to create, edit, test, run, and manage dbt Core projects. You can use Workspaces in Snowsight to work with dbt project files and directories and deploy a dbt project as a schema-level DBT PROJECT object.

    The key word there is object. Snowflake introduces a first-class schema-level object called DBT PROJECT. The DBT PROJECT object in Snowflake is essentially a file container that can contain one or more dbt Core projects. Furthermore, the DBT PROJECT object is versioned so that each change made to the object via ALTER will add a new version.

    This means your dbt project — the models, the sources YAML, the dbt_project.yml — lives inside Snowflake as a versioned, native object. Not on a VM. Not in an S3 bucket somewhere. In Snowflake itself.

    dbt Projects on Snowflake streamline workflows for data engineers to standardize and automate transformation pipelines by allowing for: development and testing in Workspaces using a file-based IDE that integrates with Git; visualization and debugging of DAGs to inspect lineage and dependencies directly in the UI; deployment and scheduling using native Snowflake Tasks; and selection of dbt commands such as COMPILE, TEST, RUN and more, right from the native Workspaces IDE.

    So yes — you can schedule and run it purely with Snowflake Tasks and never touch Airflow. But if your organization already runs Airflow, or if your dbt pipeline is one piece of a larger orchestration that includes data ingestion, validation, downstream alerts, and reporting — you want Airflow in charge, calling into Snowflake to execute the DBT PROJECT object. That hybrid approach is exactly what this article covers.


    The Architecture We’re Building

    Before I show you a single line of code, let me draw the full picture because I think this is where most blog posts let you down — they show you a piece without the whole.

    [Source System / S3 / API]
            ↓
    [Airflow DAG starts]
            ↓
      Task 1: Load raw data → Snowflake staging table (via COPY INTO or S3 stage)
            ↓
      Task 2: Run data quality checks on raw data (SQLExecuteQueryOperator)
            ↓
      Task 3: EXECUTE DBT PROJECT → runs dbt build on your native Snowflake dbt project
            ↓
      Task 4: Post-run row count validation (SQLExecuteQueryOperator)
            ↓
      Task 5: Trigger downstream alert / Slack notification / refresh BI layer
            ↓
    [Pipeline complete]

    Airflow owns the orchestration. Snowflake owns the execution of the dbt transformations. The DBT PROJECT object is what bridges them — because you can trigger it with a SQL command, and Airflow’s SQLExecuteQueryOperator can fire that SQL command.

    That SQL command, by the way, is beautifully simple:

    EXECUTE DBT PROJECT my_database.my_schema.my_dbt_project
      ARGS = 'dbt build'
      VERSION = 'LAST';

    EXECUTE DBT PROJECT executes the specified dbt project object or the dbt project in a Snowflake workspace using the dbt command and command-line options specified. Snowflake Documentation

    One SQL statement. That’s all Airflow needs to fire. Let me now show you the full setup to make that work.


    Step 1: Snowflake Setup — Roles, Warehouse, and Permissions

    I always start here because bad permissions cause the most confusing failures, and they surface late in the process when you’re tired and frustrated.

    USE ROLE ACCOUNTADMIN;
    
    -- Create a dedicated role for dbt execution
    CREATE OR REPLACE ROLE dbt_executor_role;
    GRANT ROLE dbt_executor_role TO ROLE SYSADMIN;
    
    -- Create the service user Airflow will use
    CREATE OR REPLACE USER airflow_svc_user
      PASSWORD = 'YourStrongPassword123!'
      DEFAULT_ROLE = dbt_executor_role
      DEFAULT_WAREHOUSE = dbt_transform_wh
      COMMENT = 'Airflow service user for dbt orchestration';
    
    GRANT ROLE dbt_executor_role TO USER airflow_svc_user;
    
    -- Create a dedicated warehouse for dbt runs
    USE ROLE SYSADMIN;
    CREATE OR REPLACE WAREHOUSE dbt_transform_wh
      WITH WAREHOUSE_SIZE = 'SMALL'
      AUTO_SUSPEND = 120
      AUTO_RESUME = TRUE
      INITIALLY_SUSPENDED = TRUE;
    
    GRANT ALL ON WAREHOUSE dbt_transform_wh TO ROLE dbt_executor_role;
    
    -- Grant database and schema privileges
    GRANT USAGE ON DATABASE analytics_db TO ROLE dbt_executor_role;
    GRANT USAGE, CREATE TABLE, CREATE VIEW ON SCHEMA analytics_db.staging TO ROLE dbt_executor_role;
    GRANT USAGE, CREATE TABLE, CREATE VIEW ON SCHEMA analytics_db.marts TO ROLE dbt_executor_role;
    
    -- Grant the ability to execute dbt project objects
    GRANT EXECUTE DBT PROJECT ON SCHEMA analytics_db.transforms TO ROLE dbt_executor_role;

    I made a mistake my first time through — I granted object-level access but forgot the schema-level EXECUTE DBT PROJECT privilege, which is separate. The error message wasn’t obvious. Save yourself that 20-minute debugging session.


    Step 2: Deploy Your dbt Project as a Native Snowflake Object

    This is the step that feels the most different from traditional dbt Core setup. You’re not installing dbt on a server. You’re registering your project inside Snowflake.

    Option A: Via Snowsight Workspaces (recommended for first time)

    Log into Snowsight, navigate to Workspaces, and connect it to your Git repository:

    -- First, create an API integration for GitHub
    CREATE OR REPLACE API INTEGRATION github_integration
      API_PROVIDER = git_https_api
      API_ALLOWED_PREFIXES = ('https://github.com/yourorg/')
      ENABLED = TRUE;
    
    -- Create the Git repository object in Snowflake
    CREATE OR REPLACE GIT REPOSITORY dbt_project_repo
      API_INTEGRATION = github_integration
      GIT_CREDENTIALS = my_github_secret
      ORIGIN = 'https://github.com/yourorg/your-dbt-project.git';

    Option B: Deploy via SQL (great for CI/CD)

    -- Create the DBT PROJECT object from your connected Git repo
    CREATE OR REPLACE DBT PROJECT analytics_db.transforms.sales_dbt_project
      FROM GIT REPOSITORY dbt_project_repo
      REF = 'main'
      TARGET_PATH = 'models/'
      WAREHOUSE = dbt_transform_wh;

    Install dbt dependencies:

    Install dependencies by executing the dbt deps command within a Snowflake workspace, local machine, or git orchestrator to populate the dbt_packages folder for your dbt Project.

    -- Run this once after creating the project, or include in CI/CD
    EXECUTE DBT PROJECT analytics_db.transforms.sales_dbt_project
      ARGS = 'dbt deps'
      VERSION = 'LAST';

    A heads up on this: running dbt deps to install packages requires an external access integration when executed inside Snowflake Workspaces, since the runtime needs to reach external package repositories. Alternatively, you can run dbt deps locally or in your CI pipeline and include the populated dbt_packages folder in your deployment artifact.

    I found it cleaner to run dbt deps in my GitHub Actions pipeline and commit the dbt_packages folder, rather than configuring external access integrations for every environment. Your call — both approaches work.

    Verify it deployed correctly:

    -- Check your dbt project versions
    SHOW DBT PROJECTS IN SCHEMA analytics_db.transforms;
    
    -- Test execute manually before wiring Airflow
    EXECUTE DBT PROJECT analytics_db.transforms.sales_dbt_project
      ARGS = 'dbt compile'
      VERSION = 'LAST';

    If dbt compile completes without error, your project is live and ready to be called by Airflow.


    Step 3: Set Up a Real dbt Project Structure

    Let me show you what the actual project looks like. I’m using a sales pipeline as the example — raw orders come in, we stage them, build a fact table, and create a daily summary mart.

    dbt_project.yml:

    name: 'sales_pipeline'
    version: '1.0.0'
    config-version: 2
    
    profile: 'snowflake_prod'
    
    model-paths: ["models"]
    test-paths: ["tests"]
    seed-paths: ["seeds"]
    
    models:
      sales_pipeline:
        staging:
          +schema: staging
          +materialized: view
        marts:
          +schema: marts
          +materialized: table

    models/staging/stg_orders.sql:

    -- Staging model: clean and type-cast raw orders
    WITH raw AS (
        SELECT * FROM {{ source('raw', 'orders_raw') }}
    ),
    
    cleaned AS (
        SELECT
            order_id::VARCHAR           AS order_id,
            customer_id::VARCHAR        AS customer_id,
            order_date::DATE            AS order_date,
            UPPER(TRIM(status))         AS order_status,
            amount::DECIMAL(18, 2)      AS order_amount,
            region::VARCHAR             AS region,
            CURRENT_TIMESTAMP()         AS _loaded_at
        FROM raw
        WHERE order_id IS NOT NULL
          AND order_date >= '2023-01-01'
    )
    
    SELECT * FROM cleaned

    models/marts/fct_daily_orders.sql:

    -- Fact table: daily order summary by region
    WITH staged AS (
        SELECT * FROM {{ ref('stg_orders') }}
    )
    
    SELECT
        order_date,
        region,
        order_status,
        COUNT(DISTINCT order_id)                    AS total_orders,
        COUNT(DISTINCT customer_id)                 AS unique_customers,
        SUM(order_amount)                           AS total_revenue,
        AVG(order_amount)                           AS avg_order_value,
        SUM(CASE WHEN order_status = 'RETURNED' 
                 THEN order_amount ELSE 0 END)      AS returned_amount,
        CURRENT_TIMESTAMP()                         AS _refreshed_at
    FROM staged
    GROUP BY order_date, region, order_status
    ORDER BY order_date DESC, region

    models/staging/sources.yml:

    version: 2

    sources:

    • name: raw database: analytics_db schema: raw_landing tables:
      • name: orders_raw description: “Raw orders from the source system” columns:
        • name: order_id tests:
          • not_null
          • unique
        • name: customer_id tests:
          • not_null
        • name: order_date tests:
          • not_null
        • name: amount tests:
          • not_null

    models/marts/schema.yml:

    version: 2
    
    models:
      - name: fct_daily_orders
        description: "Daily order summary by region and status"
        columns:
          - name: order_date
            tests:
              - not_null
          - name: total_orders
            tests:
              - not_null
          - name: total_revenue
            tests:
              - not_null

    This gives us a clean, testable project with source freshness checks and column-level tests. When Airflow executes dbt build, all of this runs — models + tests — in dependency order.


    Step 4: Wire It All Together in Airflow

    Now the fun part. I’m going to show you a complete Airflow DAG that:

    1. Validates raw data arrived in Snowflake
    2. Fires the native dbt project execution
    3. Validates row counts on the output marts
    4. Sends a Slack notification on success or failure

    First, install the Snowflake provider if you haven’t:

    pip install apache-airflow-providers-snowflake

    Set up your Snowflake connection in the Airflow UI (Admin → Connections):

    Connection ID : snowflake_analytics
    Connection Type : Snowflake
    Account  : yourorg.us-east-1
    Login    : airflow_svc_user
    Password : YourStrongPassword123!
    Schema   : transforms
    Database : analytics_db
    Warehouse: dbt_transform_wh
    Role     : dbt_executor_role

    Now the DAG:

    dags/sales_pipeline_dag.py:

    from airflow import DAG
    from airflow.providers.snowflake.operators.snowflake import SQLExecuteQueryOperator
    from airflow.operators.python import PythonOperator, BranchPythonOperator
    from airflow.operators.empty import EmptyOperator
    from airflow.utils.dates import days_ago
    from datetime import datetime, timedelta
    import logging
    
    # ── Default args ────────────────────────────────────────────────
    default_args = {
        'owner': 'data-engineering',
        'depends_on_past': False,
        'retries': 1,
        'retry_delay': timedelta(minutes=5),
        'email_on_failure': True,
        'email': ['[email protected]'],
    }
    
    SNOWFLAKE_CONN = 'snowflake_analytics'
    
    # ── SQL snippets ─────────────────────────────────────────────────
    RAW_DATA_CHECK_SQL = """
    SELECT COUNT(*) AS raw_row_count
    FROM analytics_db.raw_landing.orders_raw
    WHERE order_date = CURRENT_DATE() - 1;
    """
    
    EXECUTE_DBT_SQL = """
    EXECUTE DBT PROJECT analytics_db.transforms.sales_dbt_project
      ARGS = 'dbt build --select staging.stg_orders+ --vars "{\\"run_date\\": \\"{{ ds }}\\"}"'
      VERSION = 'LAST';
    """
    
    MART_VALIDATION_SQL = """
    SELECT
        COUNT(*) AS mart_row_count,
        MAX(order_date) AS latest_date,
        SUM(total_revenue) AS total_revenue
    FROM analytics_db.marts.fct_daily_orders
    WHERE order_date = CURRENT_DATE() - 1;
    """
    
    ROW_COUNT_GUARD_SQL = """
    SELECT
        CASE
            WHEN COUNT(*) = 0
            THEN 'FAIL: No rows found in mart for yesterday'
            ELSE 'PASS: ' || COUNT(*) || ' rows present'
        END AS validation_result
    FROM analytics_db.marts.fct_daily_orders
    WHERE order_date = CURRENT_DATE() - 1;
    """
    
    # ── DAG definition ───────────────────────────────────────────────
    with DAG(
        dag_id='sales_pipeline_end_to_end',
        default_args=default_args,
        description='End-to-end sales pipeline: raw → dbt native project → marts',
        schedule_interval='0 6 * * *',     # 6 AM UTC daily
        start_date=days_ago(1),
        catchup=False,
        tags=['snowflake', 'dbt', 'sales'],
    ) as dag:
    
        # Task 1: Check raw data arrived
        check_raw_data = SQLExecuteQueryOperator(
            task_id='check_raw_data_arrived',
            conn_id=SNOWFLAKE_CONN,
            sql=RAW_DATA_CHECK_SQL,
            handler=lambda cursor: logging.info(
                f"Raw row count: {cursor.fetchone()[0]}"
            ),
        )
    
        # Task 2: Execute the native dbt project on Snowflake
        run_dbt_project = SQLExecuteQueryOperator(
            task_id='execute_dbt_project_snowflake',
            conn_id=SNOWFLAKE_CONN,
            sql=EXECUTE_DBT_SQL,
            # Give dbt build enough time for large projects
            execution_timeout=timedelta(hours=2),
        )
    
        # Task 3: Post-run mart validation
        validate_mart_output = SQLExecuteQueryOperator(
            task_id='validate_mart_output',
            conn_id=SNOWFLAKE_CONN,
            sql=ROW_COUNT_GUARD_SQL,
            handler=lambda cursor: logging.info(
                f"Validation result: {cursor.fetchone()[0]}"
            ),
        )
    
        # Task 4: Run broader stats query (logged for observability)
        log_mart_stats = SQLExecuteQueryOperator(
            task_id='log_mart_statistics',
            conn_id=SNOWFLAKE_CONN,
            sql=MART_VALIDATION_SQL,
        )
    
        # Task 5: Success marker
        pipeline_complete = EmptyOperator(task_id='pipeline_complete')
    
        # ── Dependencies ─────────────────────────────────────────────
        (
            check_raw_data
            >> run_dbt_project
            >> validate_mart_output
            >> log_mart_stats
            >> pipeline_complete
        )

    Step 5: Running Specific dbt Selectors from Airflow

    One of the things I really like about this approach is that you get the full power of dbt’s selector syntax passed straight through the ARGS parameter. You don’t have to run the entire project every time.

    Run only staging models:

    EXECUTE_STAGING_ONLY = """
    EXECUTE DBT PROJECT analytics_db.transforms.sales_dbt_project
      ARGS = 'dbt run --select staging.*'
      VERSION = 'LAST';
    """

    Run a specific model and all its downstream dependencies:

    EXECUTE_ORDERS_DOWNSTREAM = """
    EXECUTE DBT PROJECT analytics_db.transforms.sales_dbt_project
      ARGS = 'dbt build --select stg_orders+'
      VERSION = 'LAST';
    """

    Run tests only, separate from the model run:

    RUN_DBT_TESTS = """
    EXECUTE DBT PROJECT analytics_db.transforms.sales_dbt_project
      ARGS = 'dbt test --select staging.*'
      VERSION = 'LAST';
    """

    This means you can split a single DAG into multiple tasks — one for staging, one for marts, one for tests — and get granular retry behavior in Airflow if something fails mid-pipeline. Instead of rerunning everything, Airflow retries only the failed task.

    Here’s that pattern as a DAG:

    run_staging = SQLExecuteQueryOperator(
        task_id='run_dbt_staging',
        conn_id=SNOWFLAKE_CONN,
        sql="""
            EXECUTE DBT PROJECT analytics_db.transforms.sales_dbt_project
              ARGS = 'dbt run --select staging.*'
              VERSION = 'LAST';
        """,
    )
    
    test_staging = SQLExecuteQueryOperator(
        task_id='test_dbt_staging',
        conn_id=SNOWFLAKE_CONN,
        sql="""
            EXECUTE DBT PROJECT analytics_db.transforms.sales_dbt_project
              ARGS = 'dbt test --select staging.*'
              VERSION = 'LAST';
        """,
    )
    
    run_marts = SQLExecuteQueryOperator(
        task_id='run_dbt_marts',
        conn_id=SNOWFLAKE_CONN,
        sql="""
            EXECUTE DBT PROJECT analytics_db.transforms.sales_dbt_project
              ARGS = 'dbt run --select marts.*'
              VERSION = 'LAST';
        """,
    )
    
    run_staging >> test_staging >> run_marts

    This is how I actually run it in practice. If staging tests fail, marts never execute. If marts fail, I retry marts without re-running staging. Clean dependency management with minimal code.


    Step 6: Handling New Versions of Your dbt Project

    This is something I didn’t think about until I pushed a breaking change to main and my 6 AM pipeline executed the wrong version.

    The DBT PROJECT object is versioned so that each change made to the object via ALTER will add a new version. The versions are named according to the pattern VERSION$<num>.

    In practice, your CI/CD pipeline (GitHub Actions, etc.) should update the DBT PROJECT object after any merge to main:

    # .github/workflows/deploy_dbt.yml
    name: Deploy dbt Project to Snowflake
    
    on:
      push:
        branches: [main]
    
    jobs:
      deploy:
        runs-on: ubuntu-latest
        steps:
          - uses: actions/checkout@v3
    
          - name: Install Snowflake CLI
            run: pip install snowflake-cli-labs
    
          - name: Deploy new dbt project version
            env:
              SNOWFLAKE_ACCOUNT: ${{ secrets.SNOWFLAKE_ACCOUNT }}
              SNOWFLAKE_USER: ${{ secrets.SNOWFLAKE_USER }}
              SNOWFLAKE_PASSWORD: ${{ secrets.SNOWFLAKE_PASSWORD }}
            run: |
              snow dbt deploy \
                --project-name analytics_db.transforms.sales_dbt_project \
                --from-git \
                --ref main

    And in your Airflow SQL, VERSION = 'LAST' always picks up the most recently deployed version automatically. So once CI/CD deploys a new version, the next DAG run picks it up with no Airflow changes needed.


    Step 7: Monitoring — What to Watch and Where

    Before this setup, I was watching three screens at once when something went wrong. Now it’s mostly one.

    In Snowsight:

    -- Check recent dbt project execution history
    SELECT
        query_id,
        query_text,
        execution_status,
        start_time,
        end_time,
        DATEDIFF('second', start_time, end_time) AS duration_seconds,
        error_message
    FROM TABLE(
        INFORMATION_SCHEMA.QUERY_HISTORY(
            END_TIME_RANGE_START => DATEADD('day', -1, CURRENT_TIMESTAMP()),
            RESULT_LIMIT => 50
        )
    )
    WHERE query_text ILIKE '%EXECUTE DBT PROJECT%'
    ORDER BY start_time DESC;

    Row count drift detection (add this as an Airflow task):

    -- Compare today's mart row count to yesterday's
    -- Flag if it drops more than 20%
    WITH today AS (
        SELECT COUNT(*) AS cnt
        FROM analytics_db.marts.fct_daily_orders
        WHERE order_date = CURRENT_DATE() - 1
    ),
    yesterday AS (
        SELECT COUNT(*) AS cnt
        FROM analytics_db.marts.fct_daily_orders
        WHERE order_date = CURRENT_DATE() - 2
    )
    SELECT
        today.cnt                                               AS today_rows,
        yesterday.cnt                                           AS yesterday_rows,
        ROUND((today.cnt - yesterday.cnt) / NULLIF(yesterday.cnt, 0) * 100, 2) AS pct_change,
        CASE
            WHEN today.cnt < yesterday.cnt * 0.80
            THEN 'ALERT: Row count dropped over 20%'
            ELSE 'OK'
        END AS status
    FROM today, yesterday;

    I added this query as a SQLExecuteQueryOperator task right after the mart validation step. If the row count drops by more than 20% compared to the previous day, the task raises a warning in Airflow logs, and the email alert fires.

    Not every data quality problem shows up as a dbt test failure. Sometimes the data just quietly shrinks because an upstream feed stopped delivering. This catches that.


    What This Setup Actually Changed for Me

    I want to be real about this because I think the “benefits” sections in most blog posts are too abstract.

    Before: My pipeline had six moving parts. Airflow DAG on one server. dbt installed on a separate instance. profiles.yml with credentials that needed updating every time we rotated passwords. Separate monitoring in CloudWatch for the dbt server. Debugging a failure meant SSH → dbt server → find the log file → cross-reference with Airflow logs.

    After: The pipeline has three moving parts — Airflow, Snowflake, and GitHub. The dbt credentials are managed by Airflow’s Snowflake connection, which I was already maintaining. Debugging a failure means clicking into the Airflow task logs (which capture the SQL response from Snowflake) and if I need more detail, running the QUERY_HISTORY query above in Snowsight.

    Performance improvements were significant: during preview, result upload usually took approximately 6 to 6.5 minutes. Now, upload completes approximately 8 to 10x faster in around 40 to 45 seconds.

    The startup time improvement alone was worth it for me. My morning pipeline used to take 28-32 minutes. It now consistently runs in 18-22 minutes. That’s not from faster models — it’s from the reduction in environment spin-up overhead.


    A Few Gotchas I Hit Along the Way

    1. The EXECUTE DBT PROJECT command is synchronous by default. Airflow will wait for it to complete before marking the task done. For large projects this is fine — you want that behavior. Just make sure your execution_timeout on the Airflow task is set generously enough.

    2. Cross-project references don’t work the way you might expect. Cross-project dependencies must be copied into the root of the main project — Snowflake doesn’t support references to external file paths within the DBT PROJECT object. If you have multiple dbt projects, plan your consolidation before deploying.

    3. The VERSION = 'LAST' behavior. This always runs the most recently deployed version. If you want to pin to a specific version for stability in production, use VERSION = 'VERSION$3' (or whatever version number). I run LAST in dev and a pinned version in prod, deployed via CI/CD.

    4. Warehouse auto-resume and the first task. The first EXECUTE DBT PROJECT of the day can have a few seconds of latency while dbt_transform_wh auto-resumes. I added a lightweight warm-up query as the very first task in my DAG so the warehouse is already running by the time dbt build kicks off:

    warm_up_warehouse = SQLExecuteQueryOperator(
        task_id='warm_up_warehouse',
        conn_id=SNOWFLAKE_CONN,
        sql="SELECT CURRENT_TIMESTAMP();",
    )
    
    warm_up_warehouse >> check_raw_data >> run_dbt_project >> ...

    Costs almost nothing. Saves 5-10 seconds of variability at the start of every run.


    Why I Think This Is the Right Direction

    I started exploring this because nobody told me to. My team’s existing setup worked. A reasonable person would have left it alone.

    But the more I looked at this setup, the more I kept thinking about the overhead we carry when tools don’t talk to each other natively. Every boundary between systems is a place where credentials leak, latency is added, and debugging gets harder. The native dbt project in Snowflake closes one of those boundaries. Airflow still owns orchestration — which is where it belongs — but the transformation execution lives where the data lives.

    For the growing number of organizations that have standardized on Snowflake, the native integration offers something genuinely compelling: one fewer system to run, one fewer vendor to manage, and one fewer boundary between your data and the logic that transforms it.

    That sentence landed for me when I read it. That’s exactly what this is.

    If you’ve been running dbt Core on a server and Airflow alongside it and you’ve been tolerating that overhead long enough that you’ve stopped noticing it — try this weekend rebuild. You might be surprised how much lighter the pipeline feels on the other side.

    And if you do try it and hit something weird, drop it in the comments. I’m still learning this myself.