Category: Snowflake

Dive deep into the Snowflake Data Cloud. Guides on building a modern cloud data warehouse, data sharing, performance optimization, and leveraging advanced features like Snowpipe and Streams.

  • How to Query Snowflake in DuckDB (And Cut Your Bill While Doing It)

    How to Query Snowflake in DuckDB (And Cut Your Bill While Doing It)

    TL;DR

    • Snowflake’s 60-second minimum billing means a 4-second query gets charged for a full minute — you’re paying for 55 seconds of nothing
    • You can query Snowflake data in DuckDB via two routes: Iceberg tables on S3 (no warehouse needed) or ADBC using Apache Arrow (up to 38x faster than ODBC)
    • Once data is local in DuckDB, every subsequent query is free — no cloud credits consumed
    • A hybrid triage approach (short queries → DuckDB/MotherDuck, heavy ETL → Snowflake) cuts BI compute costs by 70–90% in practice
    • Dev and CI/CD workloads moved to local DuckDB eliminate an entire category of cloud spend entirely

    I’ve been building on Snowflake long enough to know the ritual. Warehouse wakes up. Query runs in three seconds. Warehouse idles. You get billed for sixty seconds anyway. Multiply that by every analyst, every BI dashboard refresh, every dbt run in your dev environment — and suddenly you’re staring at a bill that feels completely disconnected from the actual work that happened.

    For a long time I assumed this was just the price of doing business on a best-in-class cloud warehouse. What I didn’t realise — until I started taking DuckDB seriously — is that a meaningful chunk of that bill doesn’t have to exist at all.

    This article covers three concrete methods to get Snowflake data into DuckDB, the cost math behind why you’d want to, and how to decide what actually belongs on which engine.


    THE REAL PROBLEM: YOU’RE PAYING FOR COMPUTE YOU DIDN’T USE

    Snowflake bills compute per second — but only after a 60-second minimum each time a warehouse resumes from suspension. A query that takes five seconds gets billed for a full minute. You paid for 55 seconds of nothing.

    It gets worse at scale. When a BI dashboard fires 20 queries on load, each taking three seconds, that single page view triggers 1,200 seconds of billed compute time. The actual work? One minute.

    And then warehouse sizing compounds it further. Each size increase in Snowflake doubles credit consumption. Teams defaulting to Medium or Large for everything are paying a 4x to 8x cost premium for workloads that could run perfectly well on X-Small.

    I’ve seen this exact pattern on almost every Snowflake environment I’ve worked in. Oversized warehouse, auto-suspend set to ten minutes, no resource monitors, nobody looking at query history.


    QUICK WINS INSIDE SNOWFLAKE FIRST

    Before touching the architecture, fix the obvious things. These alone can cut spend by 20–40%.

    Set AUTO_SUSPEND to exactly 60 seconds. Not lower — setting it below 60 is counterproductive because a query arriving in that first minute triggers another 60-second minimum. Not higher — every idle second past 60 is wasted money.

    Default to X-Small warehouses. Only scale up when a specific workload has a documented SLA that requires it.

    Add resource monitors:

    CREATE OR REPLACE RESOURCE MONITOR monthly_etl_monitor
    WITH CREDIT_QUOTA = 5000
    TRIGGERS ON 75 PERCENT DO NOTIFY
            ON 100 PERCENT DO SUSPEND;
    
    ALTER WAREHOUSE etl_heavy_wh 
    SET RESOURCE_MONITOR = monthly_etl_monitor;

    METHOD 1 — QUERYING SNOWFLAKE ICEBERG TABLES DIRECTLY IN DUCKDB

    If your organisation has moved to Iceberg tables with underlying data stored in S3, you can read those tables directly in DuckDB — no Snowflake warehouse running, no credits consumed.

    Install the extensions:

    INSTALL httpfs;
    LOAD httpfs;
    INSTALL iceberg;
    LOAD iceberg;

    Configure AWS credentials:

    CREATE SECRET (
        TYPE S3,
        PROVIDER CREDENTIAL_CHAIN
    );

    Find the current metadata file for your Snowflake-managed Iceberg table:

    SELECT PARSE_JSON(
      SYSTEM$GET_ICEBERG_TABLE_INFORMATION('YOUR_DB.YOUR_SCHEMA.YOUR_TABLE')
    )['metadataLocation']::varchar;

    Query it in DuckDB:

    SELECT
        customer_id,
        COUNT(*)
    FROM iceberg_scan('s3://your-bucket/path/to/metadata/00001-xxxx.metadata.json')
    GROUP BY 1;

    Materialise once for fast repeated queries:

    CREATE TABLE payments AS 
    SELECT * FROM iceberg_scan('s3://your-bucket/.../metadata.json');

    After this: same aggregation runs in 1.5s instead of 54s.

    Real benchmark: a SELECT * on a 110-million row table finished in 29 seconds in DuckDB on an M1 MacBook. Same query on an X-Small Snowflake warehouse took 72 seconds.

    The honest limitation: DuckDB’s Iceberg support is still maturing. You need direct S3 access and have to point DuckDB at a specific metadata file rather than a catalog. This will improve over time, but it works today.


    METHOD 2 — QUERYING NATIVE SNOWFLAKE TABLES VIA ADBC

    Not on Iceberg yet? ADBC (Arrow Database Connectivity) is the right tool here.

    Apache Arrow is a columnar memory format. When you connect Snowflake to DuckDB via ADBC, data stays columnar the entire way. Traditional ODBC forces Snowflake to convert columnar → row for transfer, then DuckDB converts row → columnar for processing. DuckDB’s benchmarks show ADBC is up to 38x faster than ODBC.

    Install:

    pip install adbc_driver_snowflake pyarrow duckdb cryptography

    Connect to Snowflake and pull data as an Arrow table:

    import adbc_driver_snowflake.dbapi
    import duckdb
    import os
    from read_private_key import read_private_key
    
    SNOWFLAKE_CONFIG = {
        'adbc.snowflake.sql.account': os.getenv('SNOWFLAKE_ACCOUNT'),
        'adbc.snowflake.sql.warehouse': os.getenv('SNOWFLAKE_WAREHOUSE'),
        'adbc.snowflake.sql.role': os.getenv('SNOWFLAKE_ROLE'),
        'adbc.snowflake.sql.database': os.getenv('SNOWFLAKE_DATABASE'),
        'username': os.getenv('SNOWFLAKE_USER'),
        'adbc.snowflake.sql.client_option.jwt_private_key_pkcs8_value': pem_key,
        'adbc.snowflake.sql.auth_type': 'auth_jwt'
    }
    
    snowflake_conn = adbc_driver_snowflake.dbapi.connect(
        db_kwargs={**SNOWFLAKE_CONFIG}
    )
    
    snowflake_cursor = snowflake_conn.cursor()
    snowflake_cursor.execute("SELECT * FROM SANDBOX_DB.MY_SCHEMA.RAW_ORDERS")
    
    # Fetch as Arrow table — stays columnar, no serialisation overhead
    arrow_table = snowflake_cursor.fetch_arrow_table()
    
    # Persist locally in DuckDB
    duckdb_conn = duckdb.connect('demo.db')
    duckdb_conn.execute("""
        CREATE TABLE IF NOT EXISTS raw_orders AS 
        SELECT * FROM arrow_table
    """)

    One heads-up: figuring out the connection parameters using a private key is not straightforward — the docs aren’t great on this point. The private key needs to be re-encoded into PEM format before passing it to the ADBC driver:

    from cryptography.hazmat.primitives import serialization
    
    def read_private_key(private_key_path: str, private_key_passphrase: str = None) -> str:
        with open(private_key_path, 'rb') as key_file:
            private_key = serialization.load_pem_private_key(
                key_file.read(),
                password=private_key_passphrase.encode() if private_key_passphrase else None
            )
            pem_key = private_key.private_bytes(
                encoding=serialization.Encoding.PEM,
                format=serialization.PrivateFormat.PKCS8,
                encryption_algorithm=serialization.NoEncryption()
            )
            return pem_key.decode('utf-8')

    Once that’s sorted, the workflow is clean: pull data from Snowflake via ADBC once, materialise it locally in DuckDB, query it as many times as you want — zero Snowflake credits consumed after the initial pull.


    METHOD 3 — THE HYBRID ARCHITECTURE: ROUTE WORKLOADS BY TYPE

    The two methods above are great for development and ad-hoc analysis. For production BI workloads, the cleanest solution I’ve seen is a hybrid architecture where you triage queries by workload type.


    The insight that unlocked this for me was using Snowflake’s query_history to actually categorise what’s running:

    WITH query_stats AS (
        SELECT
            warehouse_name,
            user_name,
            query_id,
            execution_time / 1000 AS execution_seconds
        FROM snowflake.account_usage.query_history
        WHERE
            start_time >= DATEADD('day', -30, CURRENT_TIMESTAMP())
            AND warehouse_name IS NOT NULL
            AND execution_status = 'SUCCESS'
    )
    SELECT
        warehouse_name,
        user_name,
        COUNT(query_id) AS query_count,
        MEDIAN(execution_seconds) AS median_execution_seconds,
        CASE
            WHEN query_count > 1000 AND median_execution_seconds < 30 
                THEN 'Interactive BI / High Frequency'
            WHEN query_count <= 1000 AND median_execution_seconds < 60 
                THEN 'Ad-Hoc Exploration'
            WHEN median_execution_seconds >= 300 
                THEN 'Batch ETL / Heavy Analytics'
            ELSE 'General Purpose'
        END AS workload_category
    FROM query_stats
    GROUP BY warehouse_name, user_name
    ORDER BY query_count DESC;

    Use MEDIAN not AVG — outlier queries skew the average and give a misleading picture of typical duration.

    Routing logic:

    • Short and bursty BI (sub-30s, high frequency) → move to usage-based engine. Real math: $528/month on Snowflake X-Small running continuously vs $5.87/month on per-second billing for the same workload.
    • Dev and CI/CD → local DuckDB, zero cloud credits
    • Heavy batch ETL, multi-TB → keep on Snowflake, 60s minimum is irrelevant for hour-long jobs

    This is the same principle I apply when thinking about orchestration — use the right tool for the job, not the most powerful tool for everything. I wrote about a similar decision process in why I stopped using Snowflake Tasks for orchestration — the short version is that mature orchestration tools give you far more control over exactly this kind of workload routing.


    WHEN TO STAY ON SNOWFLAKE

    Multi-terabyte batch processing — predictable provisioned compute matters more than idle cost savings when a job runs for hours.

    Enterprise governance — complex data masking, RBAC at scale, data residency requirements. Snowflake’s security surface is mature. DuckDB isn’t designed for this.

    Already-efficient workloads — if a warehouse runs at high utilisation for 8 hours straight, there’s no idle tax to eliminate. Don’t fix what isn’t broken.


    WHAT REAL COST SAVINGS LOOK LIKE

    • One SaaS company: 70%+ reduction in warehousing costs after moving to DuckDB-based solution
    • Okta: $60,000/month Snowflake spend for threat detection reduced substantially using parallel DuckDB instances
    • A data engineering team: 79% immediate reduction in Snowflake BI spend using DuckDB as a caching layer, 7x faster query times

    None of these required abandoning Snowflake. They required deciding which workloads actually needed it.


    FREQUENTLY ASKED QUESTIONS

    Can you query Snowflake data in DuckDB without a Snowflake warehouse running?
    Yes — two ways. Iceberg tables via the iceberg extension (no warehouse), or native tables via ADBC. Both require a brief initial connection, but once data is materialised locally, all subsequent queries are free.

    What is ADBC and why is it faster than ODBC?
    ADBC keeps data in columnar format throughout. ODBC forces columnar → row → columnar conversion. DuckDB benchmarks show ADBC up to 38x faster for transfers.

    How much can I realistically save?
    For short, high-frequency dashboard queries: 70–90% is consistent across documented cases. The 60-second minimum means a 4-second query costs 15x what it should.

    Is DuckDB production-ready?
    For single-node analytical workloads under a few terabytes: yes. Multi-user concurrency at scale and enterprise governance: not yet.

    Do I need Iceberg?
    No. ADBC works with native Snowflake tables. The main friction is private key encoding, which the docs don’t explain well.

    Will this work with dbt?
    Yes. dbt-duckdb lets you run your full dbt project locally against DuckDB. Pull source data once from Snowflake, develop and test for free, deploy to Snowflake in production only.This eliminates cloud compute costs for the entire development loop. I’ve written about dbt native projects and pipeline patterns if you want more context on how this fits into a Snowflake-first stack.


    Related blogs :

    → DuckDB official docs and installation
    → DuckDB ADBC benchmarks
    → Apache Arrow project
    → Snowflake query_history view docs
    → dbt-duckdb adapter on GitHub
    → Greybeam ADBC connection code on GitHub

  • Why I Stopped Using Snowflake Tasks for Orchestration

    Why I Stopped Using Snowflake Tasks for Orchestration


    I want to be clear about something before I say anything critical: Snowflake Tasks are genuinely good. I used them for months. I recommended them to people. I wrote internal documentation about how to set them up.

    And then, slowly, quietly, I stopped reaching for them — and started reaching for Airflow instead.

    This isn’t a hit piece on Snowflake Tasks. It’s an honest look at where they work beautifully, where they start to crack, and the specific moment I realised I was fighting the tool instead of using it. If you’re in that same place right now — Tasks running fine on paper, increasingly painful in practice — this is for you.


    Why I Started With Tasks in the First Place

    The pitch for Snowflake Tasks is genuinely compelling: schedule and orchestrate your data pipelines without leaving Snowflake. No extra infrastructure. No Airflow server to maintain. No Docker containers. No YAML config files. Just SQL.

    For a solo data engineer or a small team running straightforward ELT pipelines that live entirely inside Snowflake, this is actually a great deal. You write a Task, you chain a few of them together, you set a cron schedule on the root, and the whole thing runs on serverless compute that Snowflake manages for you. Clean. Simple. Zero ops overhead.

    I built my first Task tree on a customer dimension pipeline — about 6 tasks chained together to handle raw landing, staging, SCD2 merge, and a downstream mart refresh. It worked perfectly. I was genuinely impressed.

    So I built more of them. And that’s where things started to get interesting.


    The First Sign Something Was Off

    The thing about Snowflake Tasks is that they look fine at small scale. Five tasks. Eight tasks. Even fifteen tasks chained together works reasonably well.

    The cracks start showing when your pipelines grow, when requirements get more complex, and when something goes wrong at 7am and you need to figure out what happened and why.

    My first real frustration was observability. When a Task fails, Snowflake logs it — but finding that log, understanding the full execution context, and connecting it to what came before and after requires digging through TASK_HISTORY in ACCOUNT_USAGE or calling INFORMATION_SCHEMA.TASK_HISTORY(). There’s no single screen that shows you, visually, what ran, what passed, what failed, and what the downstream impact was.

    Compare that to opening the Airflow UI, clicking into a DAG run, and seeing every task coloured green or red with full logs one click away. The difference in time-to-diagnosis is not small. I once spent 40 minutes reconstructing a failed task tree execution from TASK_HISTORY queries that would have taken me 3 minutes in Airflow.

    -- How you debug a failed Snowflake Task
    SELECT
        name,
        state,
        scheduled_time,
        completed_time,
        error_code,
        error_message
    FROM TABLE(INFORMATION_SCHEMA.TASK_HISTORY(
        SCHEDULED_TIME_RANGE_START => DATEADD('hour', -6, CURRENT_TIMESTAMP()),
        RESULT_LIMIT => 100
    ))
    WHERE name ILIKE '%customer_dim%'
    ORDER BY scheduled_time DESC;

    That query works. But it’s not a dashboard. It’s archaeology.


    The Retry Problem

    This one hurt me in production.

    Snowflake Tasks have basic retry configuration — you can set SUSPEND_TASK_AFTER_NUM_FAILURES to pause a task after repeated failures, which is useful. But what you can’t do natively is retry a specific failed task in the middle of a tree and resume from that point forward.

    If Task 6 in a 10-task chain fails, you fix the problem, and you want to re-run from Task 6 onwards — you’re doing it manually. You can resume the root task, but it re-runs everything from the beginning on the next scheduled tick. Or you run Task 6’s SQL manually, then manually kick off Task 7, Task 8… you see where this is going.

    In Airflow, you right-click the failed task node, click “Clear”, and it re-runs that task and everything downstream. That’s it. One click. No manual intervention, no risk of accidentally re-running something upstream that already completed correctly and shouldn’t run twice.

    For pipelines with expensive upstream tasks — large MERGE operations, heavy aggregations — re-running from the beginning when only a downstream step failed is both wasteful and risky. Wasteful because you’re burning compute credits on work already done. Risky because some operations are not safely idempotent and running them twice produces wrong results.


    The Conditional Logic Wall

    Here’s the limitation that finally pushed me to switch.

    My pipelines started needing branching logic. Specifically: run the full pipeline on weekdays, run a lighter version on weekends. Or: if the row count from the previous step is zero, skip the downstream merge and send an alert instead of running an empty MERGE that silently succeeds.

    In Airflow, this is a BranchPythonOperator. Three lines of Python. Clean, explicit, version-controlled.

    In Snowflake Tasks, this requires workarounds. You can use a stored procedure with SYSTEM$TASK_DEPENDENTS_ENABLE logic, or try to simulate branching with conditional stored procedures that check a flag and decide whether to execute. It works — technically — but it’s brittle, hard to read, and the logic is buried inside a stored procedure rather than visible in the orchestration layer where it belongs.

    Snowflake Tasks can only execute SQL statements and stored procedures. For more complex logic in Python, Java, or other languages, external schedulers are required. Flexera

    When your pipeline logic is entirely SQL, Tasks are fine. The moment you need to make an orchestration decision based on runtime data — not just “did this succeed or fail” but “what did this return, and what should I do about it” — you’re working against the grain.


    The Scale Limit Nobody Mentions

    There is a hard limit of 1,000 Tasks per data pipeline. For very large implementations this is an issue and you need to split out the data pipeline into multiple separate data pipelines as a workaround. Sonra

    Most teams won’t hit 1,000 tasks. But if you’re building a platform for multiple teams — separate pipelines per business domain, each with their own task trees — you will eventually bump into governance and management complexity that a 1,000-task-per-pipeline limit doesn’t help with.

    More practically: managing dozens of separate task trees, each owned by a different role, with different schedules, different failure behaviours, and no unified view across all of them — is hard. There’s no Snowflake-native equivalent of Airflow’s DAG list view where you can see all pipelines, their last run status, and their next scheduled run in one place.


    What I Use Instead — And Why

    I switched to Airflow with the SQLExecuteQueryOperator for Snowflake, and I’ve written about this setup in depth in my post How I Wired Snowflake’s Native dbt Projects to Airflow. The short version of why it works better for me:

    Airflow owns orchestration. Snowflake owns execution. That’s the right division of responsibility. Airflow is purpose-built for DAG management, dependency handling, retries, branching, alerting, and observability. Snowflake is purpose-built for data processing at scale. Letting each tool do what it’s best at — instead of asking Snowflake Tasks to be a general-purpose orchestrator — is the cleaner architecture.

    Here’s the pattern I use for a typical pipeline:

    from airflow import DAG
    from airflow.providers.snowflake.operators.snowflake import SQLExecuteQueryOperator
    from airflow.operators.python import BranchPythonOperator
    from datetime import datetime, timedelta
    
    with DAG(
        dag_id='customer_dimension_pipeline',
        schedule_interval='0 6 * * *',
        start_date=datetime(2024, 1, 1),
        catchup=False,
    ) as dag:
    
        load_raw = SQLExecuteQueryOperator(
            task_id='load_raw_customers',
            conn_id='snowflake_analytics',
            sql="CALL raw.sp_load_customers();",
        )
    
        validate_raw = SQLExecuteQueryOperator(
            task_id='validate_raw_row_count',
            conn_id='snowflake_analytics',
            sql="""
                SELECT CASE
                    WHEN COUNT(*) = 0 THEN 1/0  -- Forces task failure if no rows
                    ELSE COUNT(*)
                END FROM raw.customers_staging
                WHERE load_date = CURRENT_DATE();
            """,
        )
    
        run_scd2_merge = SQLExecuteQueryOperator(
            task_id='run_scd2_merge',
            conn_id='snowflake_analytics',
            sql="CALL transforms.sp_customer_scd2_merge();",
        )
    
        refresh_mart = SQLExecuteQueryOperator(
            task_id='refresh_customer_mart',
            conn_id='snowflake_analytics',
            sql="CALL marts.sp_refresh_customer_summary();",
        )
    
        load_raw >> validate_raw >> run_scd2_merge >> refresh_mart

    If validate_raw fails because there are zero rows, the pipeline stops. run_scd2_merge never runs. I get an Airflow alert. I can clear and rerun just validate_raw and everything downstream once the issue is fixed — without touching load_raw again.

    That conditional validation step alone — stopping a pipeline when upstream data is missing — was nearly impossible to implement cleanly with Tasks. With Airflow it’s a forced division by zero in the validation SQL. Ugly, but effective. There are cleaner ways with ShortCircuitOperator too.


    To Be Fair: When Snowflake Tasks Are Still the Right Choice

    I don’t want this to read as “never use Tasks.” That’s not what I’m saying.

    Tasks are still my first choice for:

    Micro-refresh patterns. A Task + Stream combination for near-real-time SCD2 updates — triggering only when the stream has data — is elegant and genuinely hard to replicate cleanly in Airflow. I covered exactly this pattern in my post Snowflake Streams and Tasks for SCD2 — How I Actually Use Them.

    Simple scheduled SQL. A single SQL statement that needs to run every 30 minutes with no dependencies? Task all the way. Zero ops overhead for maximum simplicity.

    Snowflake-only pipelines with no external context. If your pipeline never needs to know anything about the outside world — no API calls, no file system checks, no cross-system dependencies — Tasks keep everything in one place.

    Small teams with no existing orchestration infrastructure. If you’re a team of two data engineers and setting up Airflow feels like overkill, Tasks get you 80% of the way there with 10% of the setup cost.

    The honest decision framework:

    ScenarioUse
    Simple scheduled SQL, no branchingSnowflake Tasks
    Stream-triggered incremental loadsSnowflake Tasks + Streams
    Multi-step pipeline with retry requirementsAirflow
    Conditional branching based on row countsAirflow
    Pipelines touching external systemsAirflow
    Cross-team pipelines needing unified observabilityAirflow
    Large task trees (50+ steps)Airflow

    The Honest Summary

    I didn’t stop using Snowflake Tasks because they’re bad. I stopped using them as my primary orchestration layer because that’s not what they’re optimised for — and I was asking them to do a job they weren’t built to do well.

    Snowflake Tasks handle most data orchestration patterns, but the real question is who actually plays well with Snowflake as more than just a place to send SQL. Monte Carlo Tasks are Snowflake’s answer to simple scheduling. Airflow is the answer to complex orchestration. Knowing which problem you actually have is the whole game.

    If your pipelines are growing, your failure debugging is getting slower, and you’ve found yourself writing stored procedures to simulate branching logic — that’s the sign. That’s the moment I had. The switch to Airflow was a weekend of setup and a week of migration, and I haven’t looked back.

  • 2026 Guide: Snowflake Cortex Code Cost Control

    2026 Guide: Snowflake Cortex Code Cost Control

    When I first started using Cortex Code, cost was the last thing on my mind. It’s right there in the Snowsight UI, it feels like a built-in feature, and nothing in the interface tells you that tokens are being consumed behind the scenes.

    Then I went digging through the official docs properly — not just the feature overview, but the cost controls section — and found something I hadn’t seen covered anywhere: Snowflake has a native, dedicated cost control system for Cortex Code that most people don’t know exists. Two specific parameters. Per-user. Per-surface. Configurable by any ACCOUNTADMIN in seconds.

    This article is specifically about that. If you’re running Cortex Code in your Snowflake account — especially with a team — you need to set these up.


    How Cortex Code Billing Works

    If you’re new to Cortex Code, I’d recommend reading my earlier post What Is Snowflake Cortex Code and How I Taught Myself to Use It first — it covers what the tool actually does and how to access it in Snowsight. This article picks up at the cost control layer.

    Cortex Code runs on two surfaces, each with its own billing model:

    Cortex Code in Snowsight — the browser-based UI. Token-based billing tied to your existing Snowflake account. Snowflake will notify users before charges formally begin, but AI service costs are already accumulating underneath.

    Cortex Code CLI — the command-line agent. Already billing via token consumption — pay-as-you-go for existing Snowflake accounts, or a subscription model for individual sign-ups.

    The key thing to understand: unlike virtual warehouses where you can set a Resource Monitor with a credit quota, Cortex Code has its own separate cost control parameters that standard Resource Monitors don’t cover. You need to set these explicitly.

    For the full official billing breakdown, see Snowflake’s Cortex Code billing documentation.


    The Two Parameters You Need to Know

    Snowflake provides exactly two parameters for Cortex Code cost control. Both are set by ACCOUNTADMIN only, operate on a rolling 24-hour window per user, and are completely independent of each other.

    ParameterSurface ControlledDefault
    CORTEX_CODE_CLI_DAILY_EST_CREDIT_LIMIT_PER_USERCortex Code CLI-1 (unlimited)
    CORTEX_CODE_SNOWSIGHT_DAILY_EST_CREDIT_LIMIT_PER_USERCortex Code in Snowsight-1 (unlimited)

    And here’s what the values actually mean:

    ValueBehaviour
    -1 (default)No limit — unlimited access
    0Access blocked entirely for that user
    Any positive numberAccess blocked once estimated credits exceed that number in the past 24 hours

    Full parameter reference: Cost controls for Cortex Code — Snowflake Docs


    Setting Account-Level Limits

    This is the first thing I’d do — set a sensible default for all users at the account level. If someone goes over it, they’re blocked until the rolling window resets. No silent runaway spend.

    -- Cap all users at 20 credits per day for CLI
    ALTER ACCOUNT SET CORTEX_CODE_CLI_DAILY_EST_CREDIT_LIMIT_PER_USER = 20;
    
    -- Cap all users at 20 credits per day in Snowsight
    ALTER ACCOUNT SET CORTEX_CODE_SNOWSIGHT_DAILY_EST_CREDIT_LIMIT_PER_USER = 20;

    To remove an account-level limit and restore unlimited access:

    ALTER ACCOUNT UNSET CORTEX_CODE_CLI_DAILY_EST_CREDIT_LIMIT_PER_USER;
    ALTER ACCOUNT UNSET CORTEX_CODE_SNOWSIGHT_DAILY_EST_CREDIT_LIMIT_PER_USER;

    For the full ALTER ACCOUNT syntax reference, see ALTER ACCOUNT — Snowflake Docs.


    Setting Per-User Limits (Override)

    User-level settings override the account-level setting for that specific user. Everyone else keeps the account default.

    -- Power user gets a higher CLI limit than the account default
    ALTER USER power_user SET CORTEX_CODE_CLI_DAILY_EST_CREDIT_LIMIT_PER_USER = 50;
    
    -- Junior analyst gets a tighter Snowsight limit
    ALTER USER junior_analyst SET CORTEX_CODE_SNOWSIGHT_DAILY_EST_CREDIT_LIMIT_PER_USER = 5;
    
    -- Block a service account or contractor from Snowsight entirely
    ALTER USER contractor_account SET CORTEX_CODE_SNOWSIGHT_DAILY_EST_CREDIT_LIMIT_PER_USER = 0;

    Setting the value to 0 blocks access completely for that user on that surface. Useful for service accounts that should never be touching Cortex Code interactively.

    To remove a user-level override and fall back to the account default:

    ALTER USER junior_analyst UNSET CORTEX_CODE_SNOWSIGHT_DAILY_EST_CREDIT_LIMIT_PER_USER;

    See ALTER USER — Snowflake Docs for the full syntax.


    A Practical Setup for a Real Team

    Here’s how I’d configure this for a typical data engineering team — let’s say you have senior engineers, analysts, and a few service accounts:

    -- Step 1: Set sensible defaults for the whole account
    ALTER ACCOUNT SET CORTEX_CODE_CLI_DAILY_EST_CREDIT_LIMIT_PER_USER = 20;
    ALTER ACCOUNT SET CORTEX_CODE_SNOWSIGHT_DAILY_EST_CREDIT_LIMIT_PER_USER = 20;
    
    -- Step 2: Give senior engineers more headroom on CLI
    ALTER USER senior_eng_1 SET CORTEX_CODE_CLI_DAILY_EST_CREDIT_LIMIT_PER_USER = 50;
    ALTER USER senior_eng_2 SET CORTEX_CODE_CLI_DAILY_EST_CREDIT_LIMIT_PER_USER = 50;
    
    -- Step 3: Block service accounts from both surfaces
    ALTER USER airflow_svc SET CORTEX_CODE_CLI_DAILY_EST_CREDIT_LIMIT_PER_USER = 0;
    ALTER USER airflow_svc SET CORTEX_CODE_SNOWSIGHT_DAILY_EST_CREDIT_LIMIT_PER_USER = 0;
    
    ALTER USER dbt_svc SET CORTEX_CODE_CLI_DAILY_EST_CREDIT_LIMIT_PER_USER = 0;
    ALTER USER dbt_svc SET CORTEX_CODE_SNOWSIGHT_DAILY_EST_CREDIT_LIMIT_PER_USER = 0;

    If you’re using dbt projects natively inside Snowflake, you’ll already have dedicated service accounts set up — I covered that in detail in How I Wired Snowflake’s Native dbt Projects to Airflow. Those same service accounts should be blocked from Cortex Code with a 0 limit.

    Service accounts should never be using Cortex Code. Setting them to 0 explicitly means even if someone accidentally grants CORTEX_AGENT_USER to a service role, the credit limit acts as a safety net.


    Auditing Who Has Custom Limits

    Snowflake provides a script in the official docs to list all users with per-user overrides. This is exactly what you need for a quarterly governance review:

    -- Audit all users with a custom CLI credit limit override
    EXECUTE IMMEDIATE $$
    DECLARE
      current_user STRING;
      rs_users RESULTSET;
      res      RESULTSET;
    BEGIN
      CREATE OR REPLACE TEMPORARY TABLE _param_overrides (user_name STRING, param_value STRING);
    
      SHOW USERS;
      rs_users := (SELECT "name" FROM TABLE(RESULT_SCAN(LAST_QUERY_ID())));
    
      FOR record IN rs_users DO
        current_user := record."name";
    
        EXECUTE IMMEDIATE
          'SHOW PARAMETERS LIKE ''CORTEX_CODE_CLI_DAILY_EST_CREDIT_LIMIT_PER_USER'' IN USER "' || :current_user || '"';
    
        INSERT INTO _param_overrides (user_name, param_value)
          SELECT :current_user, "value"
          FROM TABLE(RESULT_SCAN(LAST_QUERY_ID()))
          WHERE "level" = 'USER';
      END FOR;
    
      res := (SELECT * FROM _param_overrides);
      RETURN TABLE(res);
    END;
    $$;

    To audit Snowsight overrides instead, swap CORTEX_CODE_CLI_DAILY_EST_CREDIT_LIMIT_PER_USER for CORTEX_CODE_SNOWSIGHT_DAILY_EST_CREDIT_LIMIT_PER_USER in the SHOW PARAMETERS LIKE clause. Run both regularly. This script is taken directly from the official Snowflake cost controls page — I haven’t modified it.


    What Happens When a User Hits Their Limit

    When a user’s rolling 24-hour usage exceeds the configured threshold, that surface returns an error telling them the daily credit limit has been reached. They can’t use it again until enough time passes for usage to drop below the limit. The other surface is unaffected — hitting the CLI limit doesn’t lock them out of Snowsight.

    Admins can adjust or remove the limit at any time to restore access immediately, without waiting for the window to roll.

    This is worth communicating to your team before you set limits — nobody likes being surprised by a sudden block mid-workflow. Set the limits, tell people what they are, and tell them who to contact if they need a temporary increase.


    One More Layer: Monitor Actual Usage Too

    The credit limits control access — they stop runaway spend. But you also want visibility into what’s being consumed before limits are hit. Pair your limits with a monitoring query:

    sql

    SELECT
        DATE_TRUNC('day', start_time)   AS usage_day,
        function_name,
        model_name,
        SUM(tokens_used)                AS total_tokens,
        SUM(credits_used)               AS total_credits
    FROM snowflake.account_usage.cortex_functions_usage_history
    WHERE start_time >= CURRENT_DATE - 30
    GROUP BY 1, 2, 3
    ORDER BY usage_day DESC, total_credits DESC;

    This tells you which models and functions are driving costs across your account. The credit limits stop the bleeding. This query tells you where the bleeding is coming from.

    For a deeper look at how Snowflake bills for AI functions generally — token rates, model pricing differences, and what “output tokens” means for your bill — see my post Snowflake Cortex Pricing — What Every Data Engineer Should Know and the official Snowflake Service Consumption Table which is the source of truth for current credit rates.


    The Bottom Line

    The right Cortex Code cost control setup in 2026 takes about 10 minutes and covers you completely:

    1. Set account-level daily limits for both CLI and Snowsight — start with 20 credits as a reasonable default
    2. Override upward for power users who legitimately need more headroom
    3. Override to 0 for all service accounts
    4. Run the audit script quarterly to catch drift as people join or leave the team
    5. Monitor cortex_functions_usage_history for spend patterns alongside your limits

    The alternative is the default: -1 for every user, unlimited spend, and a bill that arrives before anyone realized the meter was running.

  • Snowflake Interview Questions — Expert Level

    Snowflake Interview Questions — Expert Level

    After all of this, the real tell at the senior level isn’t whether you know all these answers. It’s whether you can connect them.

    The best signal a senior candidate gives is when they answer one question and naturally reference another. “That’s the same clustering depth issue I mentioned earlier — the root cause is the same even though the symptoms look different.” That kind of connected thinking is what separates someone who’s read about Snowflake from someone who’s been debugging production issues at midnight.

    These aren’t different topics. They’re the same platform, looked at from different angles. That’s what the senior interview is actually testing.

    Good luck — and if any of these scenarios show up in your interview and you crack it, come back and let me know.

    I’ve been on both sides of the Snowflake interview table now. I’ve been the nervous one trying to remember what CLUSTER_BY actually does under pressure, and I’ve been the one reviewing candidates who clearly memorized a blog post instead of actually understanding the platform.

    There’s a massive gap between what the generic “Top 50 Snowflake Interview Questions” articles prepare you for and what a senior data engineering interview actually feels like. This article is about that gap.

    Nobody at the senior level is going to ask you “what is a virtual warehouse.” What they’re going to do is describe a production scenario — something that went wrong, or a design problem they’re genuinely wrestling with — and see how you think through it. They want to know if you’ve actually been inside the query profile. If you’ve ever had a Snowpipe backlog and had to figure out why. If you understand what “spilling to remote storage” means for your credit bill.

    This is the article I wish existed when I was preparing. I’m writing it from everything I’ve learned building real things on Snowflake — not from documentation tabs.


    How to Read This Article

    I’ve organized questions into six categories that match how real interviews are structured:

    1. Architecture & Internals
    2. Performance Tuning & Query Optimization
    3. Cost Management & Credits
    4. Data Ingestion & Pipelines
    5. Security, Governance & RBAC
    6. Real-World Scenario Questions (the hardest ones)

    Each question comes with what a strong answer actually sounds like — not bullet points you memorize, but the kind of connected thinking that signals you’ve done this for real.


    Category One: Architecture & Internals

    These questions separate people who’ve read the docs from people who understand the engine.


    Q1. Walk me through what actually happens when a query hits Snowflake — from the moment I press Run to the moment results appear.

    This is the architecture question that trips up even experienced people because they know the three layers but can’t connect them into a real execution story.

    A strong answer: When you submit a query, it first hits the Cloud Services layer — this is Snowflake’s brain. It authenticates you, parses the SQL, checks metadata in the global metadata store (which knows exactly which micro-partitions hold your data without touching a warehouse), and builds a query execution plan. This whole step runs without your virtual warehouse even waking up.

    If the results of this exact query are already in the result cache — and the underlying data hasn’t changed — Snowflake returns those results instantly from Cloud Services. Zero warehouse credits consumed. This is why identical repeated queries on unchanged data are free.

    If there’s no cached result, the query gets dispatched to your virtual warehouse — the compute layer. The warehouse spins up worker nodes (if suspended, there’s a small cold-start cost), and each node fetches the relevant micro-partitions from cloud storage — S3, Azure Blob, or GCS. This data gets loaded into the warehouse’s local SSD cache (the local disk cache). The workers process it in parallel using MPP (Massively Parallel Processing), and results flow back to the Cloud Services layer, which caches them in the result cache, then returns them to you.

    The key insight: storage is completely decoupled from compute. Multiple warehouses can read the same data simultaneously without contention. This is the fundamental reason Snowflake handles concurrent workloads better than traditional MPP databases.


    Q2. Explain micro-partitioning. How does it actually affect query performance and what can go wrong?

    Snowflake automatically divides table data into micro-partitions — immutable, compressed columnar files, typically between 50 and 500 MB of uncompressed data. Each micro-partition stores metadata: min and max values for each column, number of distinct values, null counts.

    When a query runs with a WHERE clause, Snowflake uses this metadata to prune micro-partitions — skipping any partition where the filter condition can’t possibly match. If your query is WHERE order_date = '2024-01-15', Snowflake checks the min/max metadata on order_date for each micro-partition and skips any partition where 2024-01-15 is outside that range. It never reads those partitions.

    The failure mode: if data is poorly clustered — meaning order_date values are scattered randomly across hundreds of micro-partitions — pruning becomes ineffective. Snowflake has to read most of the table to find your rows. This is when you see large “Bytes scanned” numbers in the query profile despite a tight filter.

    Check clustering health with:

    SELECT SYSTEM$CLUSTERING_INFORMATION(
        'orders',
        '(order_date)'
    );

    Look at average_depth. A value close to 1.0 means excellent pruning. Higher values mean data is scattered and clustering is poor.

    What can make clustering degrade over time: heavy DML operations (lots of individual INSERTs or UPDATEs) scatter data across new micro-partitions without regard to clustering order. For tables that receive continuous small inserts, consider a Clustering Key with automatic reclustering enabled.


    Q3. What is the difference between Automatic Clustering and manually running CLUSTER BY?

    When you define a clustering key on a table, Snowflake can maintain it in two ways.

    Manually triggering ALTER TABLE orders CLUSTER BY (order_date) runs a one-time reclustering operation. It reorganizes existing micro-partitions to improve clustering depth on the specified column. It costs credits and is a point-in-time fix.

    Automatic Clustering is a continuous background service. Once enabled, Snowflake monitors clustering depth and automatically triggers reclustering operations when it detects degradation — typically after significant DML activity. You pay for the compute, but you don’t manage it.

    When should you use automatic clustering vs. not? Automatic clustering makes sense for large tables (generally multi-TB+) that receive continuous writes and are frequently queried with filters on a high-cardinality column. For smaller tables, or tables that are mostly read-only after initial load, the cost of continuous clustering may not justify the query performance improvement. Always benchmark before enabling it.

    A practical note: clustering keys should have high cardinality (lots of distinct values) but not too high — date columns, category columns with dozens of values, and region columns are good candidates. Using a UUID or primary key as a clustering key is usually a mistake — too many distinct values means micro-partitions can’t contain meaningful ranges.


    Q4. Describe the three types of caching in Snowflake and the situations where each one helps you — or fails you.

    This is a question where depth matters more than breadth.

    Result Cache (Cloud Services layer): Stores the complete output of recent queries for up to 24 hours. If you run the same query and the underlying data hasn’t changed, you get the result back instantly with zero warehouse compute. The “data hasn’t changed” check is metadata-based — if any DML touched the relevant tables, the result cache is invalidated.

    Where it fails: if your query uses non-deterministic functions like CURRENT_TIMESTAMP(), RANDOM(), or CURRENT_USER() — or if a WHERE clause references CURRENT_DATE() — the result cache is bypassed entirely because the result could differ on each execution. This is a common gotcha: wrapping a query in a scheduled task that adds WHERE load_date = CURRENT_DATE() means the result cache never kicks in.

    Local Disk Cache (Warehouse layer): When worker nodes fetch micro-partitions from cloud storage, they cache them on local SSD. Subsequent queries that touch the same micro-partitions within the same warehouse session can read from local disk rather than going back to object storage — significantly faster. This cache is per-warehouse and is lost when the warehouse suspends.

    Where it fails: if you suspend your warehouse and resume it, the local cache is cold. For dashboards or BI tools that run queries on a fixed schedule against a specific warehouse, keeping that warehouse running between query bursts (or using auto-resume with a short suspend timeout) preserves the local cache and makes repeated runs faster.

    Metadata Cache (Cloud Services layer): Snowflake maintains metadata about table structure, row counts, min/max values, and partition information entirely in memory in the Cloud Services layer. Many COUNT(*), MIN(), and MAX() queries on simple tables can be answered from metadata alone — without engaging a warehouse at all.

    Where it helps the most: SELECT COUNT(*) FROM large_table that would cost credits on any other platform is often free on Snowflake.


    Category 2: Performance Tuning & Query Optimization


    Q5. You inherit a query that takes 45 minutes. Walk me through your exact diagnostic process.

    A senior candidate’s answer is a structured diagnostic process, not a list of tips.

    Step 1: Open the Query Profile. In Snowsight, find the query in query history and click into the profile. The profile shows you a DAG of query operators — where time was spent, which operator consumed the most credits, and critically: whether there was spillage.

    Step 2: Check for spillage first. Look for “Bytes spilled to local storage” or “Bytes spilled to remote storage” in the query profile. Spillage means the warehouse ran out of memory and had to write intermediate results to disk. Remote spillage is particularly expensive — it means even local SSD wasn’t enough. This is your first escalation trigger. If you see remote spillage, the warehouse is too small for this query. Upsizing the warehouse (SMALL → MEDIUM → LARGE) doubles memory at each step and can eliminate spillage entirely. Upsizing isn’t always the right long-term answer, but it tells you quickly whether memory is the constraint.

    Step 3: Check partition pruning. Look at “Partitions scanned” vs “Partitions total.” If the ratio is high (e.g., 2.4M / 2.5M partitions scanned), the query is doing a near-full table scan. This means either no clustering key, wrong clustering key, or the filter column isn’t selective. Add a filter on a well-clustered column, or reconsider the clustering key.

    Step 4: Check for cartesian joins or bad join order. In the profile DAG, look at the row counts flowing between join operators. If an intermediate join is exploding rows to billions before a later filter reduces them, the query logic needs rewriting — push filters earlier or restructure CTEs.

    Step 5: Check for repeated subqueries. A correlated subquery inside a CASE statement or a nested SELECT that re-executes per row is a classic SQL performance killer. Rewrite using window functions or pre-aggregated CTEs.

    Step 6: Check the warehouse size vs. the data volume. Snowflake’s query optimizer makes different decisions at different warehouse sizes. An XL warehouse processes 16x the data per unit time as an XS. For one-off heavy queries, temporarily bumping to a larger warehouse is often cheaper (faster completion = fewer total credits) than running a smaller warehouse for much longer.


    Q6. Scenario: A dbt model that was running in 8 minutes is now taking 35 minutes. Nothing in the SQL changed. What do you investigate?

    This is the scenario that makes senior candidates shine because it forces them to think about things outside the SQL.

    The first question I’d ask: did anything change in the data, even if the SQL didn’t change? If row volume doubled (new data source onboarded, historical backfill ran), the same query legitimately takes longer. Check row counts in the source tables against last month.

    Second: did the warehouse change? If the warehouse was downsized or its MAX_CLUSTER_COUNT was reduced, the same query runs with less compute. Check warehouse history.

    Third: clustering degradation. If the table receives continuous inserts and has a clustering key, clustering depth degrades over time as new micro-partitions are added in insertion order rather than cluster order. Run SYSTEM$CLUSTERING_INFORMATION and compare to a baseline. If average_depth climbed significantly, reclustering is overdue.

    Fourth: check if a new index of activity appeared in QUERY_HISTORY around the same time the slowdown started. Sometimes a new workload hitting the same warehouse — a new analyst running heavy queries during your pipeline window — causes credit contention that slows your job. The fix is workload isolation: move the pipeline to a dedicated warehouse.

    Fifth: run the query on a larger warehouse and compare. If it’s proportionally faster, it’s a compute-bound problem. If it’s not faster, it’s a data or SQL problem.


    Q7. What is a multi-cluster warehouse, and when would you specifically choose NOT to use one?

    A multi-cluster warehouse allows Snowflake to spin up additional warehouse clusters automatically when queuing occurs — concurrency scaling for a single warehouse. Instead of queries waiting in line because the single-cluster warehouse is at max concurrency, a second (or third, up to your max setting) cluster spins up to serve additional queries.

    When to use it: high-concurrency workloads where many users or processes are querying simultaneously. BI tools with dozens of dashboards refreshing at the same time. ETL pipelines that run many parallel tasks. Anything where queries are frequently queuing.

    When NOT to use it: for a single-user or low-concurrency workload, multi-cluster adds cost with no benefit — you’re paying for cluster management overhead. Also: multi-cluster doesn’t help a single slow query. One 45-minute query does not benefit from a second cluster spinning up because the bottleneck is compute for that one query, not concurrency. For that scenario, you want a larger single-cluster warehouse, not more clusters.

    The billing nuance: each active cluster in a multi-cluster warehouse is billed separately. A MEDIUM warehouse running 3 clusters simultaneously costs 3x a single MEDIUM warehouse. This is appropriate and expected for high-concurrency workloads — but if multi-cluster is accidentally enabled on a warehouse that doesn’t need it, you’ll see credits spike unnecessarily.


    Category 3: Cost Management & Credits


    Q8. Scenario: Your Snowflake credits spiked 3x last Tuesday vs. the previous Tuesday. How do you diagnose this?

    This is a real operational scenario and interviewers love it because it tests whether you know ACCOUNT_USAGE views.

    -- Step 1: Identify which warehouses drove the spike
    SELECT
        warehouse_name,
        DATE(start_time)            AS usage_date,
        SUM(credits_used)           AS total_credits
    FROM snowflake.account_usage.warehouse_metering_history
    WHERE start_time >= DATEADD('day', -14, CURRENT_TIMESTAMP())
    GROUP BY warehouse_name, DATE(start_time)
    ORDER BY usage_date, total_credits DESC;

    This immediately tells you which warehouse(s) were responsible.

    -- Step 2: Drill into queries on that warehouse during the spike window
    SELECT
        query_id,
        query_text,
        user_name,
        warehouse_name,
        execution_time / 1000       AS execution_seconds,
        bytes_spilled_to_remote_storage,
        bytes_scanned,
        credits_used_cloud_services
    FROM snowflake.account_usage.query_history
    WHERE warehouse_name = 'TRANSFORM_WH'
      AND DATE(start_time) = '2024-01-16'
    ORDER BY execution_time DESC
    LIMIT 20;

    Look for: unusually long execution times, massive bytes_scanned values, or high bytes_spilled_to_remote_storage — all signals of expensive queries. Also check if the number of queries is simply higher (new pipeline deployed, more users added).

    -- Step 3: Check if a new pipeline or user caused the spike
    SELECT
        user_name,
        COUNT(*) AS query_count,
        SUM(execution_time) / 1000 AS total_execution_seconds
    FROM snowflake.account_usage.query_history
    WHERE DATE(start_time) = '2024-01-16'
    GROUP BY user_name
    ORDER BY total_execution_seconds DESC;

    Once you identify the culprit, add a Resource Monitor to prevent recurrence:

    CREATE OR REPLACE RESOURCE MONITOR transform_wh_monitor
      WITH CREDIT_QUOTA = 500
      FREQUENCY = MONTHLY
      START_TIMESTAMP = IMMEDIATELY
      TRIGGERS
        ON 80 PERCENT DO NOTIFY
        ON 100 PERCENT DO SUSPEND_IMMEDIATE;
    
    ALTER WAREHOUSE transform_wh SET RESOURCE_MONITOR = transform_wh_monitor;

    Q9. What’s the practical difference between Transient and Permanent tables, and when would misusing them burn you?

    Permanent tables have full Time Travel (up to 90 days) and a 7-day Fail-safe period after Time Travel expires. Both features consume storage — Snowflake keeps the historical micro-partitions for the retention window.

    Transient tables have Time Travel capped at 1 day, and zero Fail-safe. They cost less storage because there’s no extended historical data being maintained.

    When transient tables make sense: staging tables and intermediate transformation tables that you rebuild completely each run. If a table is truncated and reloaded daily, paying for 90 days of Time Travel on that data is pure waste.

    The burn scenario: someone creates all their core business tables as transient to save costs. Then a developer runs DELETE FROM customer_dim WHERE segment = 'Enterprise' with a bug in the WHERE clause — and instead of filtering to test accounts, it deletes 40,000 live customer records. With a permanent table, you’d recover with a Time Travel query in 5 minutes. With a transient table, if this happened 2 days ago, that data is gone. Fail-safe doesn’t exist for transient tables. Recovery now means a full source-system re-extract and reload.

    The right pattern: transient for staging and scratch tables. Permanent for anything that is a source of truth.


    Category #4: Data Ingestion & Pipelines


    Q10. You set up Snowpipe to ingest data from an S3 bucket. After two weeks it stops loading new files. Files are landing in S3 but not appearing in Snowflake. Walk me through your debugging process.

    Snowpipe is event-driven — it relies on S3 event notifications (or Snowflake’s AUTO_INGEST) to trigger. If notifications stop firing, files pile up silently.

    Step 1: Check the pipe status.

    SELECT SYSTEM$PIPE_STATUS('my_schema.my_pipe');

    Look at pendingFileCount, lastIngestedTimestamp, and executionState. If executionState is PAUSED or STOPPED, that’s your immediate answer.

    Step 2: Check for load errors.

    SELECT *
    FROM TABLE(INFORMATION_SCHEMA.COPY_HISTORY(
        TABLE_NAME => 'target_table',
        START_TIME => DATEADD('day', -3, CURRENT_TIMESTAMP())
    ))
    WHERE STATUS != 'Loaded'
    ORDER BY LAST_LOAD_TIME DESC;

    If files attempted to load and failed — schema mismatch, malformed JSON, wrong delimiter — Snowpipe silently skips them after the error threshold. Files that fail repeatedly get quarantined and stop triggering new notifications.

    Step 3: Check SQS queue (for S3 auto-ingest).

    For S3-triggered Snowpipe, Snowflake creates an SQS queue that receives S3 event notifications. If that SQS queue fills up (it has limits) or the S3 event notification configuration was accidentally removed, events stop flowing.

    Step 4: Check file format.

    If someone modified the source system’s file format — added a column, changed the delimiter from comma to pipe, switched from gzip to uncompressed — and the Snowpipe file format definition wasn’t updated, every new file fails silently.

    Prevention: set up a monitoring query as a scheduled Snowflake Task that alerts when lastIngestedTimestamp hasn’t updated in more than X hours. Don’t wait for a stakeholder to notice missing data.


    Q11. What is the difference between Streams and Tasks in Snowflake, and give me a real scenario where you’d use them together?

    A Stream is a change data capture object. It sits on top of a table and tracks INSERT, UPDATE, and DELETE operations that occurred since the stream was last consumed. It doesn’t store data — it stores metadata about what changed and where those changed rows are. Querying a stream returns the rows that changed and a METADATA$ACTION column telling you whether they were INSERTed, UPDATEd (shows as DELETE + INSERT), or DELETEd.

    A Task is a scheduled SQL executor. It runs a SQL statement on a schedule (cron-style) or as part of a task tree (dependency-based). Tasks can be configured to run only if a stream has data — this is the key integration.

    Real scenario — SCD Type 2 on a customer dimension:

    -- Stream on the raw customer table to capture changes
    CREATE OR REPLACE STREAM customer_changes_stream
      ON TABLE raw.customers;
    
    -- Task that runs every 15 minutes but only if stream has data
    CREATE OR REPLACE TASK apply_scd2_changes
      WAREHOUSE = transform_wh
      SCHEDULE = '15 MINUTE'
      WHEN SYSTEM$STREAM_HAS_DATA('customer_changes_stream')
    AS
      MERGE INTO dim_customers AS target
      USING (
          SELECT
              customer_id,
              customer_name,
              email,
              segment,
              METADATA$ACTION,
              METADATA$ISUPDATE
          FROM customer_changes_stream
          WHERE METADATA$ACTION = 'INSERT'
      ) AS changes
      ON target.customer_id = changes.customer_id
         AND target.is_current = TRUE
      WHEN MATCHED AND changes.METADATA$ISUPDATE = TRUE THEN
          UPDATE SET target.end_date = CURRENT_DATE(),
                     target.is_current = FALSE
      WHEN NOT MATCHED THEN
          INSERT (customer_id, customer_name, email, segment,
                  start_date, end_date, is_current)
          VALUES (changes.customer_id, changes.customer_name,
                  changes.email, changes.segment,
                  CURRENT_DATE(), NULL, TRUE);
    
    ALTER TASK apply_scd2_changes RESUME;

    The WHEN SYSTEM$STREAM_HAS_DATA() condition is critical — it means the task doesn’t consume warehouse credits on runs where nothing changed. The task wakes up, checks the condition, finds the stream empty, and goes back to sleep. Only on runs where actual customer data changed does it spin up the warehouse and execute the MERGE.


    Category Five: Security, Governance & RBAC


    Q12. Your security team asks you to ensure that the analytics team can see a salary column in aggregate reports but cannot see individual employee salaries. How do you implement this in Snowflake?

    This is a column-level security scenario. Snowflake has two tools here: Dynamic Data Masking and Secure Views.

    The cleaner approach for this scenario is Dynamic Data Masking:

    -- Create a masking policy
    CREATE OR REPLACE MASKING POLICY mask_salary
      AS (val NUMBER) RETURNS NUMBER ->
        CASE
            WHEN CURRENT_ROLE() IN ('HR_ADMIN', 'SYSADMIN') THEN val
            ELSE NULL   -- Or return 0, or a rounded bucket
        END;
    
    -- Apply it to the salary column
    ALTER TABLE employees
      MODIFY COLUMN salary
      SET MASKING POLICY mask_salary;

    Now anyone in HR_ADMIN sees actual salaries. Everyone else sees NULL in that column. The table structure is the same — no separate view to maintain, no duplication of logic across multiple views. The policy travels with the column.

    For the aggregate-only requirement (they can see AVG(salary) but not individual values), a Secure View is the right tool:

    CREATE OR REPLACE SECURE VIEW employee_salary_summary AS
    SELECT
        department,
        job_grade,
        COUNT(*) AS headcount,
        ROUND(AVG(salary), -3) AS avg_salary_rounded,  -- rounds to nearest 1000
        MIN(salary) AS min_salary,
        MAX(salary) AS max_salary
    FROM employees
    GROUP BY department, job_grade
    HAVING COUNT(*) >= 5;  -- don't expose groups with fewer than 5 employees

    The HAVING COUNT(*) >= 5 clause is deliberate — without it, a determined analyst could infer individual salaries by filtering to groups of one. This is basic statistical disclosure control, and senior candidates who mention it unprompted stand out.

    The SECURE keyword on the view hides the view definition from users who don’t own it — they can’t inspect the SQL to reverse-engineer what’s being hidden.

    Q13. Walk me through how you’d structure Snowflake RBAC for a mid-sized company — say 3 teams: Data Engineering, Analytics, and Finance BI.

    This is an architecture question. A weak answer lists roles. A strong answer describes a hierarchy.

    The Snowflake recommendation is a functional role hierarchy:

    ACCOUNTADMIN (Snowflake admin only — almost never used day-to-day)
        └── SYSADMIN
                ├── DE_ADMIN (Data Engineering admin role)
                │       ├── DE_DEVELOPER (read/write on raw + staging)
                │       └── DE_READER (read-only on staging)
                ├── ANALYTICS_ADMIN
                │       ├── ANALYTICS_DEVELOPER (read/write on marts)
                │       └── ANALYTICS_READER (read-only on marts)
                └── FINANCE_BI_ADMIN
                        └── FINANCE_BI_READER (read-only on finance mart)

    Each human user gets a user role (e.g., user_bug) which is granted the appropriate functional role. Users never log in as SYSADMIN or ACCOUNTADMIN. Those are break-glass roles used only for admin operations, and any use of them should generate an alert via ACCOUNT_USAGE.LOGIN_HISTORY.

    Object ownership matters: every object should be owned by a functional role (like DE_ADMIN), not by a personal user account. If the person who created the table leaves, their user is disabled — and if they owned the objects, those objects become inaccessible until ownership is transferred. This is a real operational problem that catches teams who didn’t plan RBAC carefully.

    Service accounts (Airflow, dbt, Fivetran) get dedicated roles scoped to exactly what they need. An Airflow service account does not need SYSADMIN. It needs USAGE on specific schemas, CREATE TABLE in specific schemas, and EXECUTE TASK if it manages Tasks. Nothing more.


    Category six: Real-World Scenario Questions

    These are the ones that show up in final-round interviews and engineering manager panels.


    Q14. A finance stakeholder tells you that a revenue number in their Snowflake dashboard changed retroactively — last Tuesday it showed $4.2M, today it shows $3.8M for the same date. How do you investigate and what do you tell the stakeholder?

    The first thing I’d do is use Time Travel to find out what the data actually looked like at the time they reported $4.2M:

    -- See what the table looked like when they saw 4.2M
    SELECT SUM(order_amount) AS total_revenue
    FROM orders
      AT (TIMESTAMP => '2024-01-16 09:00:00')  -- time they saw the report
    WHERE DATE(order_date) = '2024-01-09';
    
    -- Compare to current value
    SELECT SUM(order_amount) AS total_revenue
    FROM orders
    WHERE DATE(order_date) = '2024-01-09';

    If there’s a difference, the data changed between those two timestamps. Now find out what changed:

    -- Use a stream-style comparison or Time Travel diff
    SELECT order_id, order_amount
    FROM orders
      AT (TIMESTAMP => '2024-01-16 09:00:00')
    
    MINUS
    
    SELECT order_id, order_amount
    FROM orders;

    This gives you the rows that existed in the old version but were removed or changed. Cross-reference these order IDs with the source system — were they returns? Cancelled orders? A pipeline bug that incorrectly deleted records?

    What to tell the stakeholder: be transparent and specific. “The revenue figure changed because X orders were retroactively cancelled/corrected in the source system between Tuesday and today. Here are the specific order IDs. The current figure is correct.” Don’t just tell them the number changed — give them the reason and the evidence. Stakeholders who trust your data trust your investigative process.

    Q15. You’re designing a Snowflake data platform for a company that has data analysts, data scientists, and BI developers all sharing the same account. Each group runs different query patterns. How do you structure warehouses?

    Workload isolation is the core principle. Different query patterns compete for the same resources if they share a warehouse — and they do so invisibly, making debugging harder and degrading everyone’s experience.

    My recommended structure:

    Analysts warehouse (ANALYTICS_WH — MEDIUM, multi-cluster max 3): Handles ad-hoc analyst queries. These are typically medium-length, moderately complex, need fast turnaround. Multi-cluster handles concurrency spikes when everyone runs queries at once. Auto-suspend after 5 minutes.

    BI/Reporting warehouse (BI_WH — SMALL, auto-suspend 1 minute): Serves Tableau/Power BI dashboard queries. These are typically fast, repetitive, and benefit heavily from result cache. Small warehouse is fine because result cache handles most of the work. Short auto-suspend because dashboards tend to query in bursts.

    Data Science warehouse (DS_WH — LARGE or XL, auto-suspend 10 minutes): Data scientists run long-running exploratory queries, training feature pipelines, and large aggregations. These need raw compute power. Larger warehouse, longer suspend time to preserve local cache between iterative runs.

    Pipeline/ELT warehouse (TRANSFORM_WH — MEDIUM or LARGE, no auto-suspend during pipeline window): Dedicated to dbt, Airflow, and scheduled tasks. Isolated from user-facing workloads so a heavy pipeline run doesn’t slow down analyst queries. Configure Resource Monitor here to prevent runaway pipeline credits.

    The key point for interviewers: I explicitly call out that the BI warehouse is small because of result cache behavior. Most candidates say “make BI bigger for performance.” The correct answer is: BI workloads are naturally cache-friendly, so a small warm warehouse outperforms a large cold one for dashboard queries.


    Q16. Describe a time Snowflake’s architecture surprised you — either positively or negatively.

    I’ll give you a real one from my experience.

    I inherited a data pipeline where someone had set up a Snowflake Task tree — about 12 tasks chained together to run a nightly data quality suite. Each task ran a SQL check and wrote results to a DQ log table. The whole tree was supposed to complete in about 25 minutes.

    It was taking 3 hours.

    When I investigated, I found that the task tree was configured with a 1-minute schedule on the root task but the individual tasks had their dependencies set up as AFTER relationships. What nobody had realized: each task in the chain was waiting for its predecessor to complete AND for the next 1-minute schedule tick before executing. With 12 tasks, each waiting 30-45 seconds plus a 1-minute tick, the chain took enormously longer than expected.

    The fix was switching the root task to a scheduled trigger and ensuring the child tasks had AFTER relationships correctly configured as true dependencies (not schedule-based). The tree went from 3 hours to 22 minutes.

    What I learned: Snowflake’s Task scheduling documentation is clear, but the interaction between schedule-based and dependency-based triggers is easy to misconfigure in ways that aren’t immediately obvious. Test task trees with short intervals first, watch the execution history, and validate that the AFTER relationships are actually working as dependencies rather than just ordering constraints.


    Q17. How would you implement row-level security so that regional sales managers can only see data for their region?

    Row-level security in Snowflake is implemented with Row Access Policies.

    -- Create a mapping table: which role sees which regions
    CREATE OR REPLACE TABLE region_access_map (
        role_name  VARCHAR,
        region     VARCHAR
    );
    
    INSERT INTO region_access_map VALUES
        ('SALES_APAC', 'APAC'),
        ('SALES_EMEA', 'EMEA'),
        ('SALES_AMER', 'AMER'),
        ('SALES_ADMIN', 'APAC'),   -- Admin sees everything
        ('SALES_ADMIN', 'EMEA'),
        ('SALES_ADMIN', 'AMER');
    
    -- Create the row access policy
    CREATE OR REPLACE ROW ACCESS POLICY region_row_policy
      AS (row_region VARCHAR) RETURNS BOOLEAN ->
        CURRENT_ROLE() = 'SYSADMIN'
        OR EXISTS (
            SELECT 1 FROM region_access_map
            WHERE role_name = CURRENT_ROLE()
              AND region = row_region
        );
    
    -- Apply to the sales table
    ALTER TABLE sales_transactions
      ADD ROW ACCESS POLICY region_row_policy
      ON (region);

    Now when SALES_APAC queries sales_transactions, Snowflake invisibly adds a filter to every query on that table — they only see APAC rows. They can’t bypass it. They can’t see the policy definition if it’s secured.

    Important senior-level detail: Row Access Policies apply at query time, not at storage time. The data is stored together. The filter happens when Snowflake executes the query. This means you can’t accidentally expose data through joins to other tables — the policy applies to the base table regardless of how it’s joined.

    One gotcha: if a user’s role changes mid-session, the new policy takes effect on the next query. There’s no session-level snapshot of the policy evaluation.

    What Interviewers Are Really Looking For

    After all of this, the real tell at the senior level isn’t whether you know all these answers. It’s whether you can connect them.

    The best signal a senior candidate gives is when they answer one question and naturally reference another. “That’s the same clustering depth issue I mentioned earlier — the root cause is the same even though the symptoms look different.” That kind of connected thinking is what separates someone who’s read about Snowflake from someone who’s been debugging production issues at midnight.

    These aren’t different topics. They’re the same platform, looked at from different angles. That’s what the senior interview is actually testing.

    Good luck — and if any of these scenarios show up in your interview and you crack it, come back and let me know.

  • Orchestrating Snowflake dbt Projects with Airflow — End-to-End Pipeline Guide

    Orchestrating Snowflake dbt Projects with Airflow — End-to-End Pipeline Guide

    How I Wired Snowflake’s Native dbt Projects to Airflow — And Finally Got True End-to-End Orchestration


    I’ll be honest with you — for a long time I was running dbt the way most people run it. dbt Core installed on a server, profiles.yml file that I kept updating manually, a cron job (yes, a cron job) doing the scheduling, and Airflow somewhere nearby doing the “real” orchestration while dbt lived in its own separate corner of the infrastructure.

    It worked. It was fine. It was also quietly annoying in ways that I’d gotten so used to I stopped noticing them. Managing the dbt server separately. Keeping the Snowflake credentials synced in two places. Debugging failures by jumping between the Airflow UI, SSH logs on the dbt server, and Snowsight — all at once.

    Then Snowflake went GA with dbt Projects in November 2025, and I spent a weekend rebuilding the whole thing. This article is what I learned.

    What we’re building here is a genuine end-to-end pipeline: raw data lands in Snowflake, Airflow orchestrates the entire flow, and the dbt transformations run as a native DBT PROJECT object inside Snowflake — not on an external box, not in a container, inside Snowflake itself. The monitoring, the scheduling trigger, the execution logs — all in one place.

    Let’s build it from the ground up.


    First — What Exactly Is a dbt Project on Snowflake?

    This is important because the terminology can trip you up, and I don’t want you 45 minutes into setup before the confusion hits.

    dbt Projects on Snowflake let you use familiar Snowflake features to create, edit, test, run, and manage dbt Core projects. You can use Workspaces in Snowsight to work with dbt project files and directories and deploy a dbt project as a schema-level DBT PROJECT object.

    The key word there is object. Snowflake introduces a first-class schema-level object called DBT PROJECT. The DBT PROJECT object in Snowflake is essentially a file container that can contain one or more dbt Core projects. Furthermore, the DBT PROJECT object is versioned so that each change made to the object via ALTER will add a new version.

    This means your dbt project — the models, the sources YAML, the dbt_project.yml — lives inside Snowflake as a versioned, native object. Not on a VM. Not in an S3 bucket somewhere. In Snowflake itself.

    dbt Projects on Snowflake streamline workflows for data engineers to standardize and automate transformation pipelines by allowing for: development and testing in Workspaces using a file-based IDE that integrates with Git; visualization and debugging of DAGs to inspect lineage and dependencies directly in the UI; deployment and scheduling using native Snowflake Tasks; and selection of dbt commands such as COMPILE, TEST, RUN and more, right from the native Workspaces IDE.

    So yes — you can schedule and run it purely with Snowflake Tasks and never touch Airflow. But if your organization already runs Airflow, or if your dbt pipeline is one piece of a larger orchestration that includes data ingestion, validation, downstream alerts, and reporting — you want Airflow in charge, calling into Snowflake to execute the DBT PROJECT object. That hybrid approach is exactly what this article covers.


    The Architecture We’re Building

    Before I show you a single line of code, let me draw the full picture because I think this is where most blog posts let you down — they show you a piece without the whole.

    [Source System / S3 / API]
            ↓
    [Airflow DAG starts]
            ↓
      Task 1: Load raw data → Snowflake staging table (via COPY INTO or S3 stage)
            ↓
      Task 2: Run data quality checks on raw data (SQLExecuteQueryOperator)
            ↓
      Task 3: EXECUTE DBT PROJECT → runs dbt build on your native Snowflake dbt project
            ↓
      Task 4: Post-run row count validation (SQLExecuteQueryOperator)
            ↓
      Task 5: Trigger downstream alert / Slack notification / refresh BI layer
            ↓
    [Pipeline complete]

    Airflow owns the orchestration. Snowflake owns the execution of the dbt transformations. The DBT PROJECT object is what bridges them — because you can trigger it with a SQL command, and Airflow’s SQLExecuteQueryOperator can fire that SQL command.

    That SQL command, by the way, is beautifully simple:

    EXECUTE DBT PROJECT my_database.my_schema.my_dbt_project
      ARGS = 'dbt build'
      VERSION = 'LAST';

    EXECUTE DBT PROJECT executes the specified dbt project object or the dbt project in a Snowflake workspace using the dbt command and command-line options specified. Snowflake Documentation

    One SQL statement. That’s all Airflow needs to fire. Let me now show you the full setup to make that work.


    Step 1: Snowflake Setup — Roles, Warehouse, and Permissions

    I always start here because bad permissions cause the most confusing failures, and they surface late in the process when you’re tired and frustrated.

    USE ROLE ACCOUNTADMIN;
    
    -- Create a dedicated role for dbt execution
    CREATE OR REPLACE ROLE dbt_executor_role;
    GRANT ROLE dbt_executor_role TO ROLE SYSADMIN;
    
    -- Create the service user Airflow will use
    CREATE OR REPLACE USER airflow_svc_user
      PASSWORD = 'YourStrongPassword123!'
      DEFAULT_ROLE = dbt_executor_role
      DEFAULT_WAREHOUSE = dbt_transform_wh
      COMMENT = 'Airflow service user for dbt orchestration';
    
    GRANT ROLE dbt_executor_role TO USER airflow_svc_user;
    
    -- Create a dedicated warehouse for dbt runs
    USE ROLE SYSADMIN;
    CREATE OR REPLACE WAREHOUSE dbt_transform_wh
      WITH WAREHOUSE_SIZE = 'SMALL'
      AUTO_SUSPEND = 120
      AUTO_RESUME = TRUE
      INITIALLY_SUSPENDED = TRUE;
    
    GRANT ALL ON WAREHOUSE dbt_transform_wh TO ROLE dbt_executor_role;
    
    -- Grant database and schema privileges
    GRANT USAGE ON DATABASE analytics_db TO ROLE dbt_executor_role;
    GRANT USAGE, CREATE TABLE, CREATE VIEW ON SCHEMA analytics_db.staging TO ROLE dbt_executor_role;
    GRANT USAGE, CREATE TABLE, CREATE VIEW ON SCHEMA analytics_db.marts TO ROLE dbt_executor_role;
    
    -- Grant the ability to execute dbt project objects
    GRANT EXECUTE DBT PROJECT ON SCHEMA analytics_db.transforms TO ROLE dbt_executor_role;

    I made a mistake my first time through — I granted object-level access but forgot the schema-level EXECUTE DBT PROJECT privilege, which is separate. The error message wasn’t obvious. Save yourself that 20-minute debugging session.


    Step 2: Deploy Your dbt Project as a Native Snowflake Object

    This is the step that feels the most different from traditional dbt Core setup. You’re not installing dbt on a server. You’re registering your project inside Snowflake.

    Option A: Via Snowsight Workspaces (recommended for first time)

    Log into Snowsight, navigate to Workspaces, and connect it to your Git repository:

    -- First, create an API integration for GitHub
    CREATE OR REPLACE API INTEGRATION github_integration
      API_PROVIDER = git_https_api
      API_ALLOWED_PREFIXES = ('https://github.com/yourorg/')
      ENABLED = TRUE;
    
    -- Create the Git repository object in Snowflake
    CREATE OR REPLACE GIT REPOSITORY dbt_project_repo
      API_INTEGRATION = github_integration
      GIT_CREDENTIALS = my_github_secret
      ORIGIN = 'https://github.com/yourorg/your-dbt-project.git';

    Option B: Deploy via SQL (great for CI/CD)

    -- Create the DBT PROJECT object from your connected Git repo
    CREATE OR REPLACE DBT PROJECT analytics_db.transforms.sales_dbt_project
      FROM GIT REPOSITORY dbt_project_repo
      REF = 'main'
      TARGET_PATH = 'models/'
      WAREHOUSE = dbt_transform_wh;

    Install dbt dependencies:

    Install dependencies by executing the dbt deps command within a Snowflake workspace, local machine, or git orchestrator to populate the dbt_packages folder for your dbt Project.

    -- Run this once after creating the project, or include in CI/CD
    EXECUTE DBT PROJECT analytics_db.transforms.sales_dbt_project
      ARGS = 'dbt deps'
      VERSION = 'LAST';

    A heads up on this: running dbt deps to install packages requires an external access integration when executed inside Snowflake Workspaces, since the runtime needs to reach external package repositories. Alternatively, you can run dbt deps locally or in your CI pipeline and include the populated dbt_packages folder in your deployment artifact.

    I found it cleaner to run dbt deps in my GitHub Actions pipeline and commit the dbt_packages folder, rather than configuring external access integrations for every environment. Your call — both approaches work.

    Verify it deployed correctly:

    -- Check your dbt project versions
    SHOW DBT PROJECTS IN SCHEMA analytics_db.transforms;
    
    -- Test execute manually before wiring Airflow
    EXECUTE DBT PROJECT analytics_db.transforms.sales_dbt_project
      ARGS = 'dbt compile'
      VERSION = 'LAST';

    If dbt compile completes without error, your project is live and ready to be called by Airflow.


    Step 3: Set Up a Real dbt Project Structure

    Let me show you what the actual project looks like. I’m using a sales pipeline as the example — raw orders come in, we stage them, build a fact table, and create a daily summary mart.

    dbt_project.yml:

    name: 'sales_pipeline'
    version: '1.0.0'
    config-version: 2
    
    profile: 'snowflake_prod'
    
    model-paths: ["models"]
    test-paths: ["tests"]
    seed-paths: ["seeds"]
    
    models:
      sales_pipeline:
        staging:
          +schema: staging
          +materialized: view
        marts:
          +schema: marts
          +materialized: table

    models/staging/stg_orders.sql:

    -- Staging model: clean and type-cast raw orders
    WITH raw AS (
        SELECT * FROM {{ source('raw', 'orders_raw') }}
    ),
    
    cleaned AS (
        SELECT
            order_id::VARCHAR           AS order_id,
            customer_id::VARCHAR        AS customer_id,
            order_date::DATE            AS order_date,
            UPPER(TRIM(status))         AS order_status,
            amount::DECIMAL(18, 2)      AS order_amount,
            region::VARCHAR             AS region,
            CURRENT_TIMESTAMP()         AS _loaded_at
        FROM raw
        WHERE order_id IS NOT NULL
          AND order_date >= '2023-01-01'
    )
    
    SELECT * FROM cleaned

    models/marts/fct_daily_orders.sql:

    -- Fact table: daily order summary by region
    WITH staged AS (
        SELECT * FROM {{ ref('stg_orders') }}
    )
    
    SELECT
        order_date,
        region,
        order_status,
        COUNT(DISTINCT order_id)                    AS total_orders,
        COUNT(DISTINCT customer_id)                 AS unique_customers,
        SUM(order_amount)                           AS total_revenue,
        AVG(order_amount)                           AS avg_order_value,
        SUM(CASE WHEN order_status = 'RETURNED' 
                 THEN order_amount ELSE 0 END)      AS returned_amount,
        CURRENT_TIMESTAMP()                         AS _refreshed_at
    FROM staged
    GROUP BY order_date, region, order_status
    ORDER BY order_date DESC, region

    models/staging/sources.yml:

    version: 2

    sources:

    • name: raw database: analytics_db schema: raw_landing tables:
      • name: orders_raw description: “Raw orders from the source system” columns:
        • name: order_id tests:
          • not_null
          • unique
        • name: customer_id tests:
          • not_null
        • name: order_date tests:
          • not_null
        • name: amount tests:
          • not_null

    models/marts/schema.yml:

    version: 2
    
    models:
      - name: fct_daily_orders
        description: "Daily order summary by region and status"
        columns:
          - name: order_date
            tests:
              - not_null
          - name: total_orders
            tests:
              - not_null
          - name: total_revenue
            tests:
              - not_null

    This gives us a clean, testable project with source freshness checks and column-level tests. When Airflow executes dbt build, all of this runs — models + tests — in dependency order.


    Step 4: Wire It All Together in Airflow

    Now the fun part. I’m going to show you a complete Airflow DAG that:

    1. Validates raw data arrived in Snowflake
    2. Fires the native dbt project execution
    3. Validates row counts on the output marts
    4. Sends a Slack notification on success or failure

    First, install the Snowflake provider if you haven’t:

    pip install apache-airflow-providers-snowflake

    Set up your Snowflake connection in the Airflow UI (Admin → Connections):

    Connection ID : snowflake_analytics
    Connection Type : Snowflake
    Account  : yourorg.us-east-1
    Login    : airflow_svc_user
    Password : YourStrongPassword123!
    Schema   : transforms
    Database : analytics_db
    Warehouse: dbt_transform_wh
    Role     : dbt_executor_role

    Now the DAG:

    dags/sales_pipeline_dag.py:

    from airflow import DAG
    from airflow.providers.snowflake.operators.snowflake import SQLExecuteQueryOperator
    from airflow.operators.python import PythonOperator, BranchPythonOperator
    from airflow.operators.empty import EmptyOperator
    from airflow.utils.dates import days_ago
    from datetime import datetime, timedelta
    import logging
    
    # ── Default args ────────────────────────────────────────────────
    default_args = {
        'owner': 'data-engineering',
        'depends_on_past': False,
        'retries': 1,
        'retry_delay': timedelta(minutes=5),
        'email_on_failure': True,
        'email': ['[email protected]'],
    }
    
    SNOWFLAKE_CONN = 'snowflake_analytics'
    
    # ── SQL snippets ─────────────────────────────────────────────────
    RAW_DATA_CHECK_SQL = """
    SELECT COUNT(*) AS raw_row_count
    FROM analytics_db.raw_landing.orders_raw
    WHERE order_date = CURRENT_DATE() - 1;
    """
    
    EXECUTE_DBT_SQL = """
    EXECUTE DBT PROJECT analytics_db.transforms.sales_dbt_project
      ARGS = 'dbt build --select staging.stg_orders+ --vars "{\\"run_date\\": \\"{{ ds }}\\"}"'
      VERSION = 'LAST';
    """
    
    MART_VALIDATION_SQL = """
    SELECT
        COUNT(*) AS mart_row_count,
        MAX(order_date) AS latest_date,
        SUM(total_revenue) AS total_revenue
    FROM analytics_db.marts.fct_daily_orders
    WHERE order_date = CURRENT_DATE() - 1;
    """
    
    ROW_COUNT_GUARD_SQL = """
    SELECT
        CASE
            WHEN COUNT(*) = 0
            THEN 'FAIL: No rows found in mart for yesterday'
            ELSE 'PASS: ' || COUNT(*) || ' rows present'
        END AS validation_result
    FROM analytics_db.marts.fct_daily_orders
    WHERE order_date = CURRENT_DATE() - 1;
    """
    
    # ── DAG definition ───────────────────────────────────────────────
    with DAG(
        dag_id='sales_pipeline_end_to_end',
        default_args=default_args,
        description='End-to-end sales pipeline: raw → dbt native project → marts',
        schedule_interval='0 6 * * *',     # 6 AM UTC daily
        start_date=days_ago(1),
        catchup=False,
        tags=['snowflake', 'dbt', 'sales'],
    ) as dag:
    
        # Task 1: Check raw data arrived
        check_raw_data = SQLExecuteQueryOperator(
            task_id='check_raw_data_arrived',
            conn_id=SNOWFLAKE_CONN,
            sql=RAW_DATA_CHECK_SQL,
            handler=lambda cursor: logging.info(
                f"Raw row count: {cursor.fetchone()[0]}"
            ),
        )
    
        # Task 2: Execute the native dbt project on Snowflake
        run_dbt_project = SQLExecuteQueryOperator(
            task_id='execute_dbt_project_snowflake',
            conn_id=SNOWFLAKE_CONN,
            sql=EXECUTE_DBT_SQL,
            # Give dbt build enough time for large projects
            execution_timeout=timedelta(hours=2),
        )
    
        # Task 3: Post-run mart validation
        validate_mart_output = SQLExecuteQueryOperator(
            task_id='validate_mart_output',
            conn_id=SNOWFLAKE_CONN,
            sql=ROW_COUNT_GUARD_SQL,
            handler=lambda cursor: logging.info(
                f"Validation result: {cursor.fetchone()[0]}"
            ),
        )
    
        # Task 4: Run broader stats query (logged for observability)
        log_mart_stats = SQLExecuteQueryOperator(
            task_id='log_mart_statistics',
            conn_id=SNOWFLAKE_CONN,
            sql=MART_VALIDATION_SQL,
        )
    
        # Task 5: Success marker
        pipeline_complete = EmptyOperator(task_id='pipeline_complete')
    
        # ── Dependencies ─────────────────────────────────────────────
        (
            check_raw_data
            >> run_dbt_project
            >> validate_mart_output
            >> log_mart_stats
            >> pipeline_complete
        )

    Step 5: Running Specific dbt Selectors from Airflow

    One of the things I really like about this approach is that you get the full power of dbt’s selector syntax passed straight through the ARGS parameter. You don’t have to run the entire project every time.

    Run only staging models:

    EXECUTE_STAGING_ONLY = """
    EXECUTE DBT PROJECT analytics_db.transforms.sales_dbt_project
      ARGS = 'dbt run --select staging.*'
      VERSION = 'LAST';
    """

    Run a specific model and all its downstream dependencies:

    EXECUTE_ORDERS_DOWNSTREAM = """
    EXECUTE DBT PROJECT analytics_db.transforms.sales_dbt_project
      ARGS = 'dbt build --select stg_orders+'
      VERSION = 'LAST';
    """

    Run tests only, separate from the model run:

    RUN_DBT_TESTS = """
    EXECUTE DBT PROJECT analytics_db.transforms.sales_dbt_project
      ARGS = 'dbt test --select staging.*'
      VERSION = 'LAST';
    """

    This means you can split a single DAG into multiple tasks — one for staging, one for marts, one for tests — and get granular retry behavior in Airflow if something fails mid-pipeline. Instead of rerunning everything, Airflow retries only the failed task.

    Here’s that pattern as a DAG:

    run_staging = SQLExecuteQueryOperator(
        task_id='run_dbt_staging',
        conn_id=SNOWFLAKE_CONN,
        sql="""
            EXECUTE DBT PROJECT analytics_db.transforms.sales_dbt_project
              ARGS = 'dbt run --select staging.*'
              VERSION = 'LAST';
        """,
    )
    
    test_staging = SQLExecuteQueryOperator(
        task_id='test_dbt_staging',
        conn_id=SNOWFLAKE_CONN,
        sql="""
            EXECUTE DBT PROJECT analytics_db.transforms.sales_dbt_project
              ARGS = 'dbt test --select staging.*'
              VERSION = 'LAST';
        """,
    )
    
    run_marts = SQLExecuteQueryOperator(
        task_id='run_dbt_marts',
        conn_id=SNOWFLAKE_CONN,
        sql="""
            EXECUTE DBT PROJECT analytics_db.transforms.sales_dbt_project
              ARGS = 'dbt run --select marts.*'
              VERSION = 'LAST';
        """,
    )
    
    run_staging >> test_staging >> run_marts

    This is how I actually run it in practice. If staging tests fail, marts never execute. If marts fail, I retry marts without re-running staging. Clean dependency management with minimal code.


    Step 6: Handling New Versions of Your dbt Project

    This is something I didn’t think about until I pushed a breaking change to main and my 6 AM pipeline executed the wrong version.

    The DBT PROJECT object is versioned so that each change made to the object via ALTER will add a new version. The versions are named according to the pattern VERSION$<num>.

    In practice, your CI/CD pipeline (GitHub Actions, etc.) should update the DBT PROJECT object after any merge to main:

    # .github/workflows/deploy_dbt.yml
    name: Deploy dbt Project to Snowflake
    
    on:
      push:
        branches: [main]
    
    jobs:
      deploy:
        runs-on: ubuntu-latest
        steps:
          - uses: actions/checkout@v3
    
          - name: Install Snowflake CLI
            run: pip install snowflake-cli-labs
    
          - name: Deploy new dbt project version
            env:
              SNOWFLAKE_ACCOUNT: ${{ secrets.SNOWFLAKE_ACCOUNT }}
              SNOWFLAKE_USER: ${{ secrets.SNOWFLAKE_USER }}
              SNOWFLAKE_PASSWORD: ${{ secrets.SNOWFLAKE_PASSWORD }}
            run: |
              snow dbt deploy \
                --project-name analytics_db.transforms.sales_dbt_project \
                --from-git \
                --ref main

    And in your Airflow SQL, VERSION = 'LAST' always picks up the most recently deployed version automatically. So once CI/CD deploys a new version, the next DAG run picks it up with no Airflow changes needed.


    Step 7: Monitoring — What to Watch and Where

    Before this setup, I was watching three screens at once when something went wrong. Now it’s mostly one.

    In Snowsight:

    -- Check recent dbt project execution history
    SELECT
        query_id,
        query_text,
        execution_status,
        start_time,
        end_time,
        DATEDIFF('second', start_time, end_time) AS duration_seconds,
        error_message
    FROM TABLE(
        INFORMATION_SCHEMA.QUERY_HISTORY(
            END_TIME_RANGE_START => DATEADD('day', -1, CURRENT_TIMESTAMP()),
            RESULT_LIMIT => 50
        )
    )
    WHERE query_text ILIKE '%EXECUTE DBT PROJECT%'
    ORDER BY start_time DESC;

    Row count drift detection (add this as an Airflow task):

    -- Compare today's mart row count to yesterday's
    -- Flag if it drops more than 20%
    WITH today AS (
        SELECT COUNT(*) AS cnt
        FROM analytics_db.marts.fct_daily_orders
        WHERE order_date = CURRENT_DATE() - 1
    ),
    yesterday AS (
        SELECT COUNT(*) AS cnt
        FROM analytics_db.marts.fct_daily_orders
        WHERE order_date = CURRENT_DATE() - 2
    )
    SELECT
        today.cnt                                               AS today_rows,
        yesterday.cnt                                           AS yesterday_rows,
        ROUND((today.cnt - yesterday.cnt) / NULLIF(yesterday.cnt, 0) * 100, 2) AS pct_change,
        CASE
            WHEN today.cnt < yesterday.cnt * 0.80
            THEN 'ALERT: Row count dropped over 20%'
            ELSE 'OK'
        END AS status
    FROM today, yesterday;

    I added this query as a SQLExecuteQueryOperator task right after the mart validation step. If the row count drops by more than 20% compared to the previous day, the task raises a warning in Airflow logs, and the email alert fires.

    Not every data quality problem shows up as a dbt test failure. Sometimes the data just quietly shrinks because an upstream feed stopped delivering. This catches that.


    What This Setup Actually Changed for Me

    I want to be real about this because I think the “benefits” sections in most blog posts are too abstract.

    Before: My pipeline had six moving parts. Airflow DAG on one server. dbt installed on a separate instance. profiles.yml with credentials that needed updating every time we rotated passwords. Separate monitoring in CloudWatch for the dbt server. Debugging a failure meant SSH → dbt server → find the log file → cross-reference with Airflow logs.

    After: The pipeline has three moving parts — Airflow, Snowflake, and GitHub. The dbt credentials are managed by Airflow’s Snowflake connection, which I was already maintaining. Debugging a failure means clicking into the Airflow task logs (which capture the SQL response from Snowflake) and if I need more detail, running the QUERY_HISTORY query above in Snowsight.

    Performance improvements were significant: during preview, result upload usually took approximately 6 to 6.5 minutes. Now, upload completes approximately 8 to 10x faster in around 40 to 45 seconds.

    The startup time improvement alone was worth it for me. My morning pipeline used to take 28-32 minutes. It now consistently runs in 18-22 minutes. That’s not from faster models — it’s from the reduction in environment spin-up overhead.


    A Few Gotchas I Hit Along the Way

    1. The EXECUTE DBT PROJECT command is synchronous by default. Airflow will wait for it to complete before marking the task done. For large projects this is fine — you want that behavior. Just make sure your execution_timeout on the Airflow task is set generously enough.

    2. Cross-project references don’t work the way you might expect. Cross-project dependencies must be copied into the root of the main project — Snowflake doesn’t support references to external file paths within the DBT PROJECT object. If you have multiple dbt projects, plan your consolidation before deploying.

    3. The VERSION = 'LAST' behavior. This always runs the most recently deployed version. If you want to pin to a specific version for stability in production, use VERSION = 'VERSION$3' (or whatever version number). I run LAST in dev and a pinned version in prod, deployed via CI/CD.

    4. Warehouse auto-resume and the first task. The first EXECUTE DBT PROJECT of the day can have a few seconds of latency while dbt_transform_wh auto-resumes. I added a lightweight warm-up query as the very first task in my DAG so the warehouse is already running by the time dbt build kicks off:

    warm_up_warehouse = SQLExecuteQueryOperator(
        task_id='warm_up_warehouse',
        conn_id=SNOWFLAKE_CONN,
        sql="SELECT CURRENT_TIMESTAMP();",
    )
    
    warm_up_warehouse >> check_raw_data >> run_dbt_project >> ...

    Costs almost nothing. Saves 5-10 seconds of variability at the start of every run.


    Why I Think This Is the Right Direction

    I started exploring this because nobody told me to. My team’s existing setup worked. A reasonable person would have left it alone.

    But the more I looked at this setup, the more I kept thinking about the overhead we carry when tools don’t talk to each other natively. Every boundary between systems is a place where credentials leak, latency is added, and debugging gets harder. The native dbt project in Snowflake closes one of those boundaries. Airflow still owns orchestration — which is where it belongs — but the transformation execution lives where the data lives.

    For the growing number of organizations that have standardized on Snowflake, the native integration offers something genuinely compelling: one fewer system to run, one fewer vendor to manage, and one fewer boundary between your data and the logic that transforms it.

    That sentence landed for me when I read it. That’s exactly what this is.

    If you’ve been running dbt Core on a server and Airflow alongside it and you’ve been tolerating that overhead long enough that you’ve stopped noticing it — try this weekend rebuild. You might be surprised how much lighter the pipeline feels on the other side.

    And if you do try it and hit something weird, drop it in the comments. I’m still learning this myself.

  • How I Taught Myself Snowflake Cortex Code (And What I Found)

    How I Taught Myself Snowflake Cortex Code (And What I Found)

    Nobody told me to do this.

    No manager pinged me. No sprint ticket had “explore Cortex Code” written on it. I stumbled across it one evening while clicking around Snowsight after a long day of debugging dbt models, and three hours later I looked up and realized I hadn’t thought about Jira or Slack once.

    That doesn’t happen to me.

    I run this blog because I genuinely love poking around the parts of Snowflake that most people scroll past. Cortex Code is exactly that kind of thing — it’s been sitting quietly inside Snowsight, doing something remarkable, and I feel like almost nobody in the data engineering world is talking about it properly. So let’s fix that.

    What Cortex Code Actually Is (And What It Is Not)

    Before I go any further, I want to be really clear about something that tripped me up when I first read the Snowflake docs — and that I’ve seen cause confusion in the community too.

    Cortex Code is not a SQL function you call.

    It is not SELECT SNOWFLAKE.CORTEX.CODE_ASSIST(...). It doesn’t live in your query editor the way CORTEX.COMPLETE() or CORTEX.SENTIMENT() do. I wasted about 45 minutes trying to invoke it via SQL the first time, so I’m saving you that pain right now.

    Cortex Code is a natural language interface built into Snowsight — Snowflake’s web UI. You access it by navigating to Snowsight, finding the Cortex Code section, uploading or referencing your files, and then literally just… talking to it in plain English. You describe what you want to do with your code or data files, and it responds. Think of it as having an AI pair programmer that lives inside your Snowflake environment, already understands your data context, and doesn’t need you to install anything.

    This distinction matters a lot for how you think about using it in production versus exploration.


    How to Actually Access Cortex Code in Snowsight

    Here’s the step-by-step, because the docs skip over some of this:

    Step 1: Log into Snowsight

    Go to your Snowflake account URL and log in. Make sure you’re on an account that has Cortex features enabled — Enterprise edition or higher, with the right region support.

    Step 2: Navigate to the Cortex Code Section

    In the left sidebar, look for the AI/ML or Cortex section. This UI location has shifted slightly across Snowflake releases, so if you don’t see it immediately, use the search bar at the top of Snowsight and type “Cortex.”

    Step 3: Upload Your Files Using a PUT Command

    This is where it gets interesting. If you want Cortex Code to analyze a specific file — say a Python script, a dbt model, a JSON file, or a CSV — you first upload it to a Snowflake stage using a PUT command in a worksheet:

    -- Create a stage if you don't have one
    CREATE OR REPLACE STAGE my_cortex_stage;
    
    -- Upload your file from your local machine
    PUT file:///Users/yourname/projects/my_dbt_model.sql @my_cortex_stage;
    
    -- Verify it landed
    LIST @my_cortex_stage;

    Step 4: Reference Your File in the Cortex Code Interface

    Once staged, you can reference the file in the Cortex Code interface and start asking questions about it in plain English.

    Step 5: Start Prompting

    You don’t write SQL. You write sentences. That’s the whole point.


    Real Example 1: Analyzing a dbt Model for Performance Issues

    This is the first thing I tried, and it immediately earned its keep.

    I had a dbt model — let’s call it fct_orders_daily.sql — that was consistently running for 14 minutes in production. I’d stared at it, I’d checked clustering, I’d looked at query profiles. I was going in circles.

    I uploaded the file:

    PUT file:///Users/bug/dbt_project/models/marts/fct_orders_daily.sql @my_cortex_stage;

    Then in Cortex Code I typed something like:

    “I’ve uploaded a dbt SQL model called fct_orders_daily.sql. It’s running for 14 minutes. Can you review the logic and tell me where the performance problems likely are? Suggest specific rewrites.”

    What came back wasn’t a generic “add a WHERE clause” response. It identified a specific pattern I had — a correlated subquery inside a CASE statement that was executing per-row. It rewrote the logic using a LEFT JOIN with a pre-aggregated CTE instead. The rewrite took the query from 14 minutes to under 3 minutes when I tested it.

    Did I feel a little embarrassed that an AI caught something I missed? Absolutely. Did I care? Not really. The model shipped.


    Real Example 2: Explaining Someone Else’s Legacy Code

    Every data engineer has that one Snowflake procedure. The one that nobody touches. The one with variable names like tmp_ctr2 and no comments anywhere. The one that was written by someone who left the company in 2021.

    I had one of those. It was a stored procedure managing some SCD2 logic on a customer dimension table. About 200 lines. No documentation.

    I staged the file:

    PUT file:///Users/bug/legacy/sp_customer_dim_scd2.sql @my_cortex_stage;

    Then in Cortex Code:

    “This is a Snowflake stored procedure. Can you explain what it does step by step, in plain English? Then flag any parts that look fragile or risky.”

    It produced a section-by-section breakdown that I turned directly into an internal wiki page. It also flagged that the procedure was doing a full table merge without any row count validation before and after — which, it noted, meant a failed intermediate step could silently produce a partial merge with no error thrown.

    That was a legitimate production risk I had never noticed. I added the validation. Nobody told me to. I just did it because now I knew.


    Real Example 3: Generating New Transformation Code from a Description

    This one genuinely surprised me. I described a transformation I needed in plain language — no file, just a description — and asked Cortex Code to write the SQL.

    My prompt was roughly:

    “I have a table called SALES_EVENTS with columns: event_id, customer_id, event_type (values: ‘purchase’, ‘return’, ‘browse’), event_timestamp, and amount. I want to build a daily summary table that shows, for each customer and each day: total purchases, total returns, net revenue (purchases minus returns), and a flag if they had any browse events that didn’t convert to a purchase on the same day. Write me the Snowflake SQL for this.”

    Here’s essentially what it produced (I’ve cleaned it up slightly for readability):

    WITH daily_events AS (
        SELECT
            customer_id,
            DATE(event_timestamp) AS event_date,
            SUM(CASE WHEN event_type = 'purchase' THEN amount ELSE 0 END) AS total_purchases,
            SUM(CASE WHEN event_type = 'return' THEN amount ELSE 0 END) AS total_returns,
            SUM(CASE WHEN event_type = 'purchase' THEN amount ELSE 0 END) -
            SUM(CASE WHEN event_type = 'return' THEN amount ELSE 0 END) AS net_revenue,
            MAX(CASE WHEN event_type = 'browse' THEN 1 ELSE 0 END) AS had_browse,
            MAX(CASE WHEN event_type = 'purchase' THEN 1 ELSE 0 END) AS had_purchase
        FROM SALES_EVENTS
        GROUP BY customer_id, DATE(event_timestamp)
    )
    
    SELECT
        customer_id,
        event_date,
        total_purchases,
        total_returns,
        net_revenue,
        CASE 
            WHEN had_browse = 1 AND had_purchase = 0 THEN TRUE 
            ELSE FALSE 
        END AS browse_no_convert_flag
    FROM daily_events
    ORDER BY customer_id, event_date;

    That’s clean, readable, and correct. It took maybe 30 seconds from my prompt to a working query. Normally I’d have spent 10-15 minutes writing and testing that from scratch.


    Real Example 4: Reviewing Python Scripts for Data Quality Issues

    I had a Python ingestion script that was pulling data from a REST API and writing to a Snowflake stage. I staged it:

    PUT file:///Users/bug/scripts/api_ingest.py @my_cortex_stage;

    My prompt: “Review this Python script. It ingests data from a REST API into a Snowflake stage. Are there any data quality risks, error handling gaps, or things that might fail silently in production?”

    The response flagged three things:

    • No retry logic on the API call, so a transient network failure would abort the entire run with no retry
    • The script was casting all numeric fields as strings before writing to the stage, which would cause downstream type errors in the COPY INTO command
    • There was no check on API response status codes — a 429 (rate limit) or 500 was being handled the same way as a 200

    All three were real issues. None of them were obvious from a quick read. Finding them manually would have required either running the thing and watching it fail, or very careful code review.


    Where Cortex Code Fits in Your Daily Workflow

    I want to be honest here: Cortex Code is not a replacement for knowing what you’re doing. If you give it a bad prompt, you’ll get a bad answer. If you blindly paste its output into production without reading it, eventually something will break and you’ll deserve it.

    But used as a thinking partner — as a way to get a first draft, spot what you might have missed, or understand something unfamiliar faster — it’s genuinely useful in a way that saves real time.

    Here’s how I’ve actually worked it into my day:

    Morning code review pass: Before I open a PR for anything significant, I stage the file and run a quick “what could go wrong with this?” prompt. It takes 2 minutes and has caught things twice in the last month.

    Legacy code archaeology: Any time I have to touch something old that I didn’t write, I stage it and ask for an explanation before I touch a single line. This alone is worth it.

    First-draft SQL: When I have a new transformation to build and I know exactly what the output should look like but I’m not in the mood to write boilerplate aggregations, I describe the output and let Cortex Code give me a starting point. I always read and edit what it gives me — but starting from something is faster than starting from nothing.

    Onboarding help: I’ve started pointing people who are new to the team at Cortex Code for understanding existing models. “Stage the file, ask it to explain it, then come ask me questions.” It makes onboarding conversations much more productive because they’ve already done some of the basic reading.


    What It Can’t Do (And Where You Still Need to Think)

    I don’t want this to sound like a Snowflake brochure. There are real limitations.

    It doesn’t know your data. It can reason about code logic and SQL patterns, but it doesn’t have context about what’s actually in your tables — so it can’t tell you whether a join is correct because it doesn’t know your actual cardinality or data distribution.

    Complex business logic still needs you. If your transformation encodes ten years of institutional knowledge about how a specific business process works, Cortex Code can write syntactically correct SQL but it can’t know if the business logic is right.

    You have to read the output. Every time. No exceptions.

    And the Snowsight UI for it, as of my exploration, is still evolving — some things feel a little rough around the edges. That’s fine. So did Cortex Analyst when it launched, and it’s much smoother now.


    A Few Prompting Tips That Actually Help

    After spending time with this, here’s what I’ve learned about getting better results:

    Be specific about what you want back. “Review this code” is okay. “Review this code and give me three specific things to change, written as SQL snippets I can copy” is much better.

    Tell it the context it’s missing. “This table has 2 billion rows and is clustered on event_date” is useful information that shapes the advice you’ll get.

    Ask follow-up questions. If the first answer is good but you want to go deeper on one part, just ask. The interface supports back-and-forth.

    Tell it what you’ve already tried. If you’ve already checked clustering and it didn’t help, say so. Otherwise it’ll suggest clustering.


    Why I Think This Matters

    I started this blog because I noticed that a lot of the Snowflake content online is either too shallow (“here’s what the feature is”) or too abstract (“here’s why AI in data warehouses matters”). What’s missing is the honest, practical, “here’s what I actually did and what happened” stuff.

    Cortex Code is one of those features that I think is genuinely going to change how data engineers do their day-to-day work — not in a science fiction way, but in the quiet, unglamorous way where you just realize one day that you’re spending three hours less per week on the boring parts and three hours more on the interesting parts.

    Nobody told me to learn this. I’m glad I did.

    If you try it, tell me how it goes. I’m genuinely curious whether your experience matches mine.

  • 2026 Guide: Cut dbt Build Time 48% with Snowflake Cortex Code

    2026 Guide: Cut dbt Build Time 48% with Snowflake Cortex Code

    The Moment Everything Changed

    It was a Tuesday morning when I finally snapped. My dbt project had grown to 147 models, and the daily run was taking 2 hours and 47 minutes. Our Airflow DAG was timing out. The business team was complaining about stale dashboards. And I was spending my entire morning investigating why dim_customer alone was taking 45 minutes to build.

    I had tried everything: manual query optimization, clustering keys, switching materializations. Each fix helped a little, but I was basically guessing. Then someone on the data engineering Slack mentioned using Snowflake Cortex Code to analyze their dbt manifest file.

    “Wait, it can do WHAT?” I asked.

    That question changed my entire workflow. Three months later, my dbt runs average 1 hour 23 minutes—a 48% improvement. I spend 90% less time debugging performance. And I actually have time to build new features instead of firefighting slow models.

    This isn’t a tutorial about how Cortex Code might help you. This is the real story of how it actually transformed my day-to-day work as a data engineer, with specific examples, exact prompts I use, and honest numbers about what works and what doesn’t.


    Part 1: What Is Snowflake Cortex Code? (The Simple Truth)

    Before I get into the dbt deep dive, let me explain what Cortex Code actually is—because the marketing doesn’t do it justice.

    Cortex Code is code generation AI built directly into Snowflake. Think ChatGPT, but it:

    • Understands your Snowflake schema automatically
    • Knows dbt best practices
    • Can analyze JSON files (like manifest.json)
    • Generates production-ready SQL, Python, and more
    • Lives where you already work (Snowflake UI, or via API)

    How it’s different from GitHub Copilot or ChatGPT:

    FeatureCortex CodeGitHub CopilotChatGPT
    Knows your Snowflake schema✅ Yes❌ No❌ No
    Can read manifest.json✅ Yes❌ No⚠️ Manual paste
    Snowflake-specific SQL✅ Optimized⚠️ Generic⚠️ Generic
    dbt best practices✅ Built-in⚠️ Learns from code⚠️ General knowledge
    Privacy/Security✅ Snowflake environment⚠️ Code leaves editor❌ Data uploaded

    The key difference for data engineers: Cortex Code actually understands your data warehouse context.


    Part 2: Getting Started (5-Minute Setup)

    Step 1: Enable Cortex Code

    Cortex Code is available in Snowflake (check your edition—Enterprise or higher typically has it).

    Simple interface showing Snowflake Cortex Code prompt for generating dbt models

    Step 1: Enable Cortex Code

    Cortex Code is available in Snowflake (check your edition—Enterprise or higher typically has it).

    -- Check if you have access
    SELECT SYSTEM$GET_CORTEX_FEATURES();
    -- If available, you're good to go
    -- No additional setup needed

    Step 2: First Test

    How to Access Cortex Code:

    1. Open Snowsight (Snowflake UI)
    2. Look for the “AI Assistant” or “Cortex Code” button (usually in the sidebar or bottom-right)
    3. Type your prompt in natural language
    4. Get generated code instantly

    Example first prompt:

    Generate SQL to find top 10 customers by revenue from my customers and orders tables

    Cortex Code responds with:

    SELECT 
        c.customer_id,
        c.customer_name,
        SUM(o.order_amount) as total_revenue
    FROM customers c
    JOIN orders o ON c.customer_id = o.customer_id
    GROUP BY c.customer_id, c.customer_name
    ORDER BY total_revenue DESC
    LIMIT 10;

    That’s it. No installation, no API keys, no configuration. Just natural language prompts.

    My first “wow” moment: I typed “generate a complete dbt model for customer lifetime value with staging, intermediate, and mart layers” and it produced three properly structured models with naming conventions, tests, and documentation. Took 30 seconds.


    Part 3: Core Capabilities (The Quick Tour)

    Before we dive deep into dbt, here’s what Cortex Code can do:

    Before SQL Generation:

    • Complex queries with CTEs, window functions, aggregations
    • Query optimization suggestions
    • Data quality checks

    dbt Development:

    • Model generation (staging, marts, facts, dimensions)
    • Test creation (schema tests, custom tests)
    • Macro writing
    • Performance analysis from manifest.json

    Airflow DAGs:

    • Complete DAG structures
    • Task dependencies and retry logic
    • Custom operators

    Streamlit Dashboards:

    • Layout scaffolding
    • Chart configurations
    • Filter and interactivity code

    Python UDFs:

    • Custom function generation
    • Pandas operations
    • Complex transformations

    Debugging:

    • Code explanation
    • Error analysis
    • Optimization suggestions

    How to Use: Simply open the Cortex Code interface in Snowsight and type what you need in plain English. Examples:

    • “Generate a dbt staging model for my customers table”
    • “Create an Airflow DAG for daily ETL”
    • “Build a Streamlit dashboard with revenue KPIs”

    Now let’s talk about where it really shines: dbt optimization.


    Part 4: dbt + Cortex Code – The Real Game Changer

    4.1: Quick Overview – Beyond Basic Generation

    Yes, Cortex Code can generate dbt models. Ask it for a staging model, it’ll give you:

    -- Example prompt: "Generate dbt staging model for raw_customers"
    
    -- models/staging/stg_customers.sql
    with source as (
        select * from {{ source('raw', 'customers') }}
    ),
    
    renamed as (
        select
            customer_id,
            customer_name,
            email,
            created_at,
            updated_at
        from source
    )
    
    select * from renamed

    And yes, it can write tests:

    # Prompt: "Create dbt tests for stg_customers"
    version: 2
    
    models:
      - name: stg_customers
        columns:
          - name: customer_id
            tests:
              - unique
              - not_null
          - name: email
            tests:
              - unique
              - not_null

    But honestly? That’s the boring stuff. Any code generation tool can do this. Where Cortex Code becomes indispensable is performance optimization using your actual dbt metadata.


    4.2: Performance Optimization – The Killer Feature

    This is where I went from “this is neat” to “I can’t work without this anymore.”

    The Problem I Had

    My dbt project metrics (before Cortex Code):

    • 147 models total
    • Full refresh: 2h 47min
    • Incremental run: 1h 15min
    • Daily Airflow timeout failures: 2-3 times per week
    • Time spent debugging performance: 6-8 hours per week

    I had no systematic way to know:

    • Which models were actually slow?
    • Why were they slow?
    • What should I optimize first?
    • Were my optimizations working?

    I was flying blind, making educated guesses based on gut feeling and manual timing of individual models.ed on gut feeling and manual timing of individual models.


    A) Manifest.json Analysis – The Secret Weapon

    Diagram showing how Cortex Code analyzes dbt manifest.json file to identify performance bottlenecks and optimization opportunities

    Your dbt project generates a manifest.json file in the target/ folder after every run. It contains:

    • Every model’s metadata
    • Dependencies between models
    • Column information
    • Schema details

    I never really looked at it. It’s thousands of lines of JSON. Until Cortex Code.

    How to use it:

    Step 1: Upload manifest.json to Snowflake

    -- Create a stage for your dbt metadata
    CREATE STAGE IF NOT EXISTS dbt_metadata;
    
    -- Upload the file (via SnowSQL or Snowsight UI)
    PUT file://~/dbt_project/target/manifest.json @dbt_metadata/;

    Step 2: Open Cortex Code interface in Snowsight

    • Click on the “AI Assistant” or “Cortex Code” button in Snowsight
    • This opens the natural language interface

    Step 3: Ask Cortex Code to analyze it

    Type this prompt in the Cortex Code interface:

    Analyze the manifest.json file in my dbt_metadata stage and identify the top 10 slowest models with specific optimization recommendations. 
    
    Focus on:
    - Materialization strategies (table vs incremental)
    - Clustering opportunities  
    - Complex CTEs that could be simplified
    - Join patterns that could be optimized
    
    Provide specific code changes and estimated time savings for each recommendation.

    Step 4: Review the analysis

    Cortex Code responds with detailed analysis (example of what I got):

    Top 10 Slowest Models Analysis:
    
    1. dim_customer (45.2 minutes)
       Issues identified:
       - Full table scan on 50M+ rows daily
       - No clustering key on frequently filtered columns
       - Using table materialization when incremental would work
       - 3 CTEs doing similar aggregations (could be deduplicated)
       
       Recommendations:
       - Add clustering key on (customer_id, updated_date)
       - Convert to incremental materialization
       - Use dbt_utils.deduplicate macro
       - Estimated improvement: 35-40 minutes
       
    2. fact_orders_daily (23.7 minutes)
       Issues identified:
       - Window functions not optimized
       - Joining to dim_customer before aggregation
       - Missing partition by date
       
       Recommendations:
       - Aggregate first, then join
       - Add partition_by clause to window functions
       - Consider separate monthly partitions
       - Estimated improvement: 15-18 minutes
       
    3. mart_customer_360 (18.4 minutes)
       ...

    I was stunned. This analysis would have taken me days to do manually—if I even knew where to start. Cortex Code did it in 30 seconds.


    B) Implementing the Recommendations

    Let me show you exactly what I did for dim_customer:

    Before (45 minutes):

    -- models/marts/dim_customer.sql
    {{
        config(
            materialized='table'
        )
    }}
    with customers as (
        select * from {{ ref('stg_customers') }}
    ),
    orders as (
        select * from {{ ref('fct_orders') }}
    ),
    aggregated as (
        select
            c.customer_id,
            c.customer_name,
            c.email,
            c.created_at,
            count(o.order_id) as total_orders,
            sum(o.order_amount) as lifetime_value,
            max(o.order_date) as last_order_date
        from customers c
        left join orders o on c.customer_id = o.customer_id
        group by 1,2,3,4
    )
    select * from aggregated

    After (8 minutes) following Cortex Code suggestions:

    Before and after comparison of dbt model performance: 45 minutes reduced to 8 minutes using Cortex Code optimization suggestions
    -- models/marts/dim_customer.sql
    {{
        config(
            materialized='incremental',
            unique_key='customer_id',
            cluster_by=['customer_id', 'updated_date'],
            on_schema_change='append_new_columns'
        )
    }}
    with customers as (
        select * from {{ ref('stg_customers') }}
        {% if is_incremental() %}
        where updated_date >= (select max(updated_date) from {{ this }})
        {% endif %}
    ),
    orders_aggregated as (
        -- Aggregate BEFORE joining (Cortex suggestion!)
        select
            customer_id,
            count(order_id) as total_orders,
            sum(order_amount) as lifetime_value,
            max(order_date) as last_order_date
        from {{ ref('fct_orders') }}
        {% if is_incremental() %}
        where order_date >= (select max(last_order_date) from {{ this }})
        {% endif %}
        group by customer_id
    ),
    final as (
        select
            c.customer_id,
            c.customer_name,
            c.email,
            c.created_at,
            c.updated_date,
            coalesce(o.total_orders, 0) as total_orders,
            coalesce(o.lifetime_value, 0) as lifetime_value,
            o.last_order_date
        from customers c
        left join orders_aggregated o on c.customer_id = o.customer_id
    )
    select * from final

    Changes made:

    1. ✅ Switched to incremental materialization
    2. ✅ Added clustering keys on customer_id and updated_date
    3. ✅ Aggregated orders before joining (huge win!)
    4. ✅ Added incremental logic to only process new/changed data

    Result: 45 minutes → 8 minutes (first run), 3 minutes (incremental runs)


    C) run_results.json Deep Dive

    The run_results.json file contains actual execution times and metadata from your last dbt run. Even more valuable than manifest for performance debugging.

    My weekly performance review process:

    -- Upload run_results from this week and last week
    PUT file://~/dbt_project/target/run_results.json @my_stage/current/;
    PUT file://~/dbt_project_backup/target/run_results.json @my_stage/previous/;

    Example output:

    Performance Regression Analysis:
    CRITICAL REGRESSIONS (>50% slower):
    1. mart_sales_summary
       - Previous: 4.2 min
       - Current: 9.8 min (+133%)
       - Root cause: Source table fct_sales grew from 10M to 25M rows
       - Recommendation: Add incremental logic with date partitioning
       
    2. dim_product
       - Previous: 2.1 min
       - Current: 5.4 min (+157%)
       - Root cause: New join to external API table (no clustering)
       - Recommendation: Materialize API data first, add clustering key
    MODERATE REGRESSIONS (20-50% slower):
    3. stg_orders
       - Previous: 1.2 min
       - Current: 1.6 min (+33%)
       - Root cause: New data quality test added (full table scan)
       - Recommendation: Convert test to incremental or sampling
    IMPROVEMENTS:
    1. dim_customer: 45 min → 8 min (-82%) ✅ [Your optimization worked!]
    2. fact_orders_daily: 23 min → 12 min (-48%) ✅
    NEW BOTTLENECKS:
    - mart_customer_cohort now takes 14 min (wasn't slow before)
    - Likely due to dim_customer changes propagating downstream
    - Recommendation: Review joins, consider pre-aggregation

    This is gold. I immediately know what broke, why, and how to fix it.


    D) Automated Performance Audits

    I set up a weekly routine every Monday morning using Cortex Code:

    My Monday Morning Workflow:

    Run my standardized audit prompt

    Upload latest manifest and run_results (automated via simple Python script)

    Open Cortex Code interface

    Perform a comprehensive dbt performance audit using the manifest.json and run_results.json in my dbt_metadata stage:
    
    Analysis needed:
    1. Identify slowest 15 models with root cause analysis
    2. Detect performance anti-patterns:
       - Models using full refresh that should be incremental
       - Missing clustering keys on large tables  
       - Inefficient join patterns
       - Unnecessary full table scans
    3. Find models that should be incremental but aren't
    4. Suggest clustering keys based on filter/join patterns in SQL
    5. Recommend materialization strategies (table vs view vs incremental)
    6. Calculate estimated monthly compute time savings for each recommendation
    7. Rank by effort/impact ratio (quick wins vs long-term projects)
    
    Format as prioritized action plan with:
    - Quick wins (high impact, <1 hour effort)
    - Medium effort items (2-4 hours)  
    - Strategic improvements (>4 hours)
    - Estimated ROI for each

    Sample output from last Monday:

    dbt Performance Audit - 2026-01-20
    QUICK WINS (High Impact, Low Effort):
    1. Add clustering to dim_geography on (country_code, region_id)
       - Current: 6.2 min | Estimated after: 1.5 min | Effort: 5 min
       - Impact: Save 4.7 min per run = 33 hours/month
    2. Convert fct_user_sessions to incremental
       - Current: 11.3 min | Estimated after: 2.1 min | Effort: 20 min
       - Impact: Save 9.2 min per run = 65 hours/month
    MEDIUM EFFORT, HIGH IMPACT:
    3. Refactor mart_product_performance (split into 2 models)
       - Current: 18.7 min | Estimated after: 7 min + 4 min | Effort: 2 hours
       - Impact: Save 7.7 min per run = 54 hours/month
       
    ANTI-PATTERNS DETECTED:
    ⚠️ 8 models using table materialization with full refresh daily (should be incremental)
    ⚠️ 4 models missing clustering keys despite large table scans
    ⚠️ 2 models with circular dependency risk
    TOTAL ESTIMATED MONTHLY SAVINGS: 247 hours of compute time
    TOTAL IMPLEMENTATION EFFORT: ~8 hours
    ROI: 30:1

    I tackle 2-3 items from this list each week. In three months, I’ve saved 48% of total dbt runtime.


    4.3: Real Production Examples

    Let me share three actual scenarios where Cortex Code saved my ass:

    Example 1: The Mysterious Slowdown

    Problem: mart_revenue_daily suddenly went from 5 minutes to 32 minutes. No code changes, just suddenly slow.

    What I did:

    Analyze this slow dbt model. It suddenly went from 5 minutes to 32 minutes with no code changes.
    
    Context:
    - Model name: mart_revenue_daily
    - Previous runtime: 5 min
    - Current runtime: 32 min  
    - No changes to model SQL in past month
    - Fact table grew from 50M to 180M rows last week
    
    Here's the compiled SQL from dbt:
    [I pasted the full compiled SQL here]
    
    Please:
    1. Identify the root cause of slowdown
    2. Point to specific problematic lines in the SQL
    3. Provide concrete fix with rewritten SQL
    4. Estimate performance improvement

    Cortex Code’s Response:

    Root Cause Identified:
    
    The model joins fact_transactions (now 180M rows) to dim_date without filtering first.
    
    Line 47-52 in your SQL:
    ```sql
    left join {{ ref('dim_date') }} d
        on t.transaction_date = d.date_day
    ```
    
    With 180M rows, this creates a massive intermediate result set before the WHERE clause on line 68.
    
    Solution:
    1. Move the date filter BEFORE the join (filter fact_transactions first)
    2. Add clustering key on fact_transactions.transaction_date
    3. Consider partitioning by month if queries typically filter by date range
    
    Estimated improvement: 32 min → 6 min

    Implementation:

    -- Changed from this:
    from {{ ref('fact_transactions') }} t
    left join {{ ref('dim_date') }} d on t.transaction_date = d.date_day
    where d.fiscal_year = 2024
    
    -- To this:
    from (
        select * from {{ ref('fact_transactions') }}
        where transaction_date >= '2024-01-01'
    ) t
    left join {{ ref('dim_date') }} d on t.transaction_date = d.date_day

    Plus added clustering:

    {{ config(cluster_by=['transaction_date']) }}

    Result: 32 minutes → 5.2 minutes. Problem solved in 20 minutes total.


    Example 2: Pre-Launch Optimization

    Scenario: Building a new mart_customer_churn_prediction model. Wanted to optimize it BEFORE the first production run.

    I'm about to deploy this new dbt model to production. Please review and suggest optimizations BEFORE I run it for the first time.
    
    Model: mart_customer_churn_prediction
    
    Focus areas:
    - Best materialization strategy (table, view, or incremental)
    - Clustering key recommendations
    - Potential bottlenecks in the SQL
    - Incremental logic if applicable
    - Query optimization opportunities
    
    Here's the model SQL:
    [pasted my model code]
    
    Expected data volume: 50M+ customer records
    Update frequency: Daily
    Query pattern: Filtered by customer_segment and churn_date

    Cortex Code’s Pre-Production Analysis:

    Pre-Production Optimization Analysis:
    
    CRITICAL ISSUES:
    1. ❌ Using table materialization for 50M+ row result
       Recommendation: Use incremental with unique_key on customer_id
       
    2. ❌ No clustering key despite filtering on customer_segment and churn_date
       Recommendation: cluster_by=['customer_segment', 'churn_date']
       
    3. ❌ Complex CTE chain (7 CTEs) doing full table scans
       Recommendation: Break into separate intermediate models
    
    PERFORMANCE PREDICTIONS:
    - As written: Estimated 45-60 minutes first run
    - With optimizations: Estimated 8-12 minutes first run, 2-3 min incremental
    
    SUGGESTED REFACTOR:
    Split into 3 models:
    1. int_customer_metrics (intermediate, incremental)
    2. int_churn_features (intermediate, incremental)  
    3. mart_customer_churn_prediction (mart, incremental)

    I implemented all suggestions. First production run: 9.3 minutes instead of the estimated 45-60.


    Example 3: Monthly Performance Review

    Every month, I do a comprehensive audit:

    Step 1: Collect all metadata files

    # My automation script copies these
    cp ~/dbt_project/target/manifest.json ~/monthly_audits/2026-01/
    cp ~/dbt_project/target/run_results.json ~/monthly_audits/2026-01/

    Step 2: Upload to Snowflake

    PUT file://~/monthly_audits/2026-01/* @dbt_metadata/monthly/2026-01/;

    Step 3: Open Cortex Code and run monthly audit

    Monthly dbt Performance Review - January 2026
    
    Using files in dbt_metadata/monthly/2026-01/:
    - manifest.json 
    - run_results.json
    
    Provide comprehensive analysis:
    
    1. HEALTH METRICS
       - Overall project health score (0-100)
       - Total models and average runtime
       - Percentage using best practices (incremental, clustering)
       - Month-over-month performance trend
    
    2. TOP ISSUES  
       - 10 slowest models with root cause
       - Performance anti-patterns detected
       - Models that grew disproportionately  
       - Technical debt items
    
    3. CLEANUP OPPORTUNITIES
       - Unused or rarely-run models
       - Outdated materializations
       - Redundant transformations
       - Models that can be archived
    
    4. OPTIMIZATION ROADMAP
       - Week-by-week action plan for next month
       - Quick wins vs strategic improvements
       - Estimated time savings and effort required
       - Projected end-of-month performance
    
    5. ROI CALCULATIONS
       - Current monthly compute cost
       - Potential savings from recommendations
       - Effort/impact ratio for each item

    January 2026 Audit Output:

    dbt Project Health Score: 73/100 (Up from 61 last month)
    
    PERFORMANCE SUMMARY:
    - Total models: 147
    - Average model runtime: 3.2 min (down from 5.1 min)
    - Slowest model: dim_customer_360 (14.2 min)
    - Models using incremental: 67% (target: 80%)
    - Models with clustering: 45% (target: 70%)
    
    TOP 10 ISSUES:
    1. dim_customer_360 (14.2 min) - needs incremental + clustering
    2. mart_sales_forecast (12.8 min) - complex window functions, consider simplification
    3. fct_website_sessions (11.4 min) - full refresh daily, should be incremental
    ...
    
    OPTIMIZATION ROADMAP - FEBRUARY 2026:
    Week 1: Add clustering to 8 identified models (est. save 45 min/run)
    Week 2: Convert 6 models to incremental (est. save 67 min/run)
    Week 3: Refactor mart_sales_forecast (est. save 8 min/run)
    Week 4: Remove 4 unused models identified
    
    Projected end-of-month runtime: 58 minutes (current: 83 minutes)

    Following this roadmap, I hit 61 minutes by month-end.


    4.4: My Daily Workflow with Cortex Code

    Here’s how Cortex Code fits into my actual workday:

    Monday Morning (9:00 AM) – Weekly Review:

    1. Upload latest manifest.json and run_results.json
    2. Run performance audit
    3. Create Jira tickets for top 3 optimization opportunities
    4. Prioritize for the week

    Tuesday-Thursday – Development:

    1. Need a new model?
      • Ask Cortex Code to generate boilerplate
      • Review and customize for business logic
      • Ask Cortex to optimize before first run
    2. Model running slow?
      • Share compiled SQL with Cortex
      • Get optimization suggestions
      • Implement and test
    Weekly data engineering workflow integrating Snowflake Cortex Code for dbt optimization and development

    Friday Afternoon – Cleanup:

    1. Review week’s changes in dbt
    2. Ask Cortex to review my new models for anti-patterns
    3. Generate documentation with Cortex assistance
    4. Prepare for Monday’s review

    Time saved per week:

    • Before: 8-10 hours on performance debugging
    • After: 1-2 hours on Cortex-assisted optimization
    • Net savings: 6-8 hours weekly

    4.5: Prompts That Actually Work

    Here are my most-used prompts, copy-paste ready:

    Performance Analysis:

    "Analyze this manifest.json and identify the top 10 slowest models with specific, actionable optimization recommendations ranked by estimated time savings."
    "Compare these two run_results.json files (last week vs this week) and identify performance regressions, improvements, and new bottlenecks. Prioritize by impact."
    "This model runs in X minutes. Here's the compiled SQL: [paste]. Provide optimization suggestions with estimated impact for each."

    Model Optimization:

    "Review this dbt model and suggest: 1) Best materialization strategy, 2) Clustering keys, 3) Incremental logic if applicable, 4) Query optimizations. Model: [paste]"
    "I'm building a new model for [business purpose]. Suggest optimal dbt structure including staging, intermediate, and mart layers with proper materializations."

    Debugging:

    "This dbt model suddenly got slow. Root cause analysis based on: Compiled SQL: [paste], Recent changes: [describe], Data volume changes: [numbers]"
    "Why is this incremental model doing full refreshes? Model config: [paste], Logs: [paste]"

    Ongoing Monitoring:

    "Monthly dbt health audit. Analyze manifest + run_results. Provide: health score, top 10 issues, optimization roadmap. Files: [paste]"
    "Identify unused or rarely-run models in this manifest that could be archived. Criteria: run less than once per week, not referenced by marts."

    4.6: What Works vs. What Doesn’t

    After 3 months of daily use, here’s my honest assessment:

    What Works Exceptionally Well (9-10/10):

    Manifest.json analysis – Unbelievably accurate

    • Finds bottlenecks I’d never spot manually
    • Prioritizes by actual impact
    • Estimates are within 20% of reality
    Visual comparison of Snowflake Cortex Code strengths and limitations for dbt optimization

    Performance regression detection – Catches issues immediately

    • Week-over-week comparisons are spot-on
    • Identifies root causes correctly 90% of the time

    Clustering key recommendations – Based on real query patterns

    • Suggestions almost always improve performance
    • Understands join patterns and filter predicates

    Materialization strategy advice – Knows when to use incremental vs table

    • Factors in data volume, update frequency, query patterns

    Boilerplate generation – Saves tons of typing

    • Staging models, tests, yml files
    • Follows dbt best practices

    What’s Good But Needs Review (7-8/10):

    ⚠️ Macro generation – Often correct but review logic carefully

    • Sometimes over-complicates simple macros
    • Jinja syntax is usually right, logic sometimes questionable

    ⚠️ Incremental logic – Usually good starting point

    • Test thoroughly before production
    • Edge cases might not be covered
    • Deduplication logic needs validation

    ⚠️ Complex transformations – Can over-engineer

    • Tend to add unnecessary CTEs
    • Sometimes creates cleverness over clarity

    What Doesn’t Work Well (4-6/10):

    Understanding specific business context – It’s AI, not a domain expert

    • Doesn’t know your business rules
    • Can’t infer data quality requirements
    • Might suggest technically sound but business-wrong logic

    Data distribution insights – Can’t see actual data

    • Clustering suggestions are pattern-based, not data-based
    • Doesn’t know your data skew or cardinality

    Cost optimization – Focuses on time, not cost

    • Doesn’t factor in warehouse sizing
    • Might suggest compute-expensive solutions

    Complex dependencies – Struggles with very large DAGs

    • Can get confused with 200+ model projects
    • Recommendations might create circular dependencies

    Critical: What You Must Validate:

    🔴 Always manually verify:

    1. Incremental logic (especially deduplication)
    2. Business logic in transformations
    3. Data quality test logic
    4. Macro behavior with edge cases
    5. Performance impact in production (not just estimated)

    4.7: Real Numbers from My Experience

    Let me share the actual metrics that matter:

    Before Cortex Code (December 2025):

    dbt Performance:

    • Full refresh runtime: 2h 47min
    • Incremental runtime: 1h 15min
    • Models with clustering: 12/147 (8%)
    • Models using incremental: 42/147 (29%)
    • Airflow timeout failures: 2-3/week

    My Time Spent:

    • Performance debugging: 8-10 hours/week
    • Manual manifest review: Never (too tedious)
    • Optimization work: Ad-hoc, reactive
    • New model development: 45-60 min per model

    Costs:

    • Snowflake compute (dbt): ~$1,200/month
    • Airflow retries/failures: ~$180/month
    • My time opportunity cost: Unmeasured but significant

    After 3 Months with Cortex Code (March 2026):

    dbt Performance:

    • Full refresh runtime: 1h 23min (-50%)
    • Incremental runtime: 34min (-55%)
    • Models with clustering: 67/147 (46%)
    • Models using incremental: 99/147 (67%)
    • Airflow timeout failures: 1-2/month

    My Time Spent:

    • Performance debugging: 1-2 hours/week (-85%)
    • Weekly manifest review: 15 min (automated with Cortex)
    • Optimization work: Systematic, proactive
    • New model development: 15-20 min per model (-67%)

    Costs:

    • Snowflake compute (dbt): ~$680/month (-43%)
    • Airflow retries/failures: ~$35/month (-81%)
    • My time regained: 6-8 hours/week for feature development

    ROI Calculation:

    Cost of Cortex Code: Included in Snowflake Enterprise+ (no additional cost)

    Monthly savings:

    • Snowflake compute: $520
    • Airflow compute: $145
    • My time (40 hours/month @ $75/hour): $3,000
    • Total: $3,665/month

    Annual ROI: $43,980 in quantifiable savings

    Plus unquantifiable benefits:

    • Faster feature delivery
    • Better data quality (more time for testing)
    • Reduced incidents and pages
    • Improved team morale (less firefighting)

    Part 5: Quick Wins – Other Tools

    While dbt is where I get the most value, Cortex Code is useful for other data engineering tools too:

    5.1: Airflow DAGs

    Quick generation using Cortex Code interface:

    Generate a production-ready Airflow DAG for my daily dbt run:
    
    Requirements:
    - Schedule: Run daily at 2 AM UTC
    - Retry logic: 3 attempts with 5-minute delay between retries
    - Send Slack alert on failure (webhook already configured)
    - Trigger downstream Tableau refresh on success
    - Include SLA monitoring (alert if takes >90 minutes)
    - Use dbt Cloud operator
    - Handle task dependencies properly
    
    Python version: 3.9
    Airflow version: 2.7

    Cortex Code generates:

    from airflow import DAG
    from airflow.operators.bash import BashOperator
    from airflow.operators.python import PythonOperator
    from airflow.providers.slack.notifications.slack import send_slack_notification
    from datetime import datetime, timedelta
    
    default_args = {
        'owner': 'data-engineering',
        'depends_on_past': False,
        'email_on_failure': False,
        'retries': 3,
        'retry_delay': timedelta(minutes=5),
        'sla': timedelta(minutes=90),
    }
    
    dag = DAG(
        'daily_dbt_run',
        default_args=default_args,
        description='Daily dbt production run',
        schedule='0 2 * * *',  # 2 AM UTC daily
        start_date=datetime(2026, 1, 1),
        catchup=False,
        on_failure_callback=send_slack_notification(
            text='❌ dbt DAG failed!',
            channel='#data-alerts'
        ),
    )
    
    dbt_run = BashOperator(
        task_id='dbt_run',
        bash_command='cd /opt/dbt && dbt run --profiles-dir .',
        dag=dag,
    )
    
    dbt_test = BashOperator(
        task_id='dbt_test',  
        bash_command='cd /opt/dbt && dbt test --profiles-dir .',
        dag=dag,
    )
    
    refresh_tableau = BashOperator(
        task_id='refresh_tableau',
        bash_command='curl -X POST https://tableau-server/api/refresh',
        dag=dag,
    )
    
    dbt_run >> dbt_test >> refresh_tableau

    Gets you 80% of the way there in 30 seconds vs. 30 minutes of boilerplate typing.

    5.2: Streamlit Dashboards

    Dashboard scaffolding with Cortex Code:

    Create a Streamlit dashboard for sales analytics:
    
    Features needed:
    - Date range filter (last 7, 30, 90 days or custom)
    - KPI cards showing: Total Revenue, Total Orders, Average Order Value
    - Line chart: Daily revenue trend over selected period
    - Bar chart: Top 10 products by revenue
    - Data table: Recent orders with filters
    - Connect to Snowflake table: analytics.mart_daily_sales
    - Use Streamlit's modern layout with columns
    - Include data refresh timestamp
    - Add export to CSV button
    
    Make it production-ready with proper error handling.

    Cortex Code generates:

    import streamlit as st
    import snowflake.connector
    import pandas as pd
    import plotly.express as px
    from datetime import datetime, timedelta
    
    st.set_page_config(page_title="Sales Analytics", layout="wide")
    
    # Snowflake connection
    @st.cache_resource
    def get_connection():
        return snowflake.connector.connect(
            user=st.secrets["snowflake"]["user"],
            password=st.secrets["snowflake"]["password"],
            account=st.secrets["snowflake"]["account"],
            warehouse='ANALYTICS_WH',
            database='ANALYTICS',
            schema='MART'
        )
    
    # Main dashboard
    st.title("📊 Sales Analytics Dashboard")
    
    # Date filter
    col1, col2 = st.columns([3, 1])
    with col1:
        date_range = st.selectbox(
            "Select Period",
            ["Last 7 Days", "Last 30 Days", "Last 90 Days", "Custom"]
        )
    
    # ... [rest of dashboard code]

    Generates working dashboard layout. Still need to refine styling and business logic, but saves 45 minutes of setup.

    (Keeping these brief since the focus is dbt – but both are genuinely useful)


    Part 6: Practical Tips for Data Engineers

    The Learning Curve

    Week 1: Feels magical but you don’t trust it

    • Generate code, read every line carefully
    • Validate everything in dev
    • Probably slower than writing manually

    Week 2-4: Building confidence

    • Start recognizing patterns in good vs. questionable output
    • Develop your own prompt templates
    • 20-30% faster than before

    Month 2+: It’s part of your workflow

    • Know when to use it vs. when to write manually
    • Can spot hallucinations immediately
    • 50-70% faster on routine tasks

    My Validation Checklist

    Before deploying Cortex-generated code:

    ✅ Logic review: Does this make business sense?
    ✅ Performance check: Run EXPLAIN on generated SQL
    ✅ Edge cases: Test with null values, duplicates, empty sets
    ✅ Incremental logic: Validate deduplication and update logic
    ✅ Dependencies: Check for circular references
    ✅ Tests: Generated code needs generated tests
    ✅ Peer review: Treat AI code like any other PR

    When I Don’t Use Cortex Code

    Never use for:

    • Financial calculations (too critical, audit requirements)
    • Security/access control logic (review manually)
    • One-off analyses (faster to write myself)
    • Learning new concepts (defeats the learning purpose)

    Sometimes use for:

    • Debugging (helpful but verify root cause)
    • Refactoring (good starting point, heavy review)
    • Documentation (generates good drafts)

    Always use for:

    • Boilerplate (staging models, tests, yml)
    • Performance analysis (manifest reviews)
    • Exploration (trying new patterns)

    Part 7: The Honest Verdict

    For dbt Specifically:

    Model Generation: 8/10

    • Great for standard patterns
    • Saves typing, enforces conventions
    • Still need to add business logic

    Test Creation: 9/10

    • Covers standard tests well
    • Good at identifying what to test
    • Custom tests need review

    Manifest Analysis: 10/10 ⭐⭐⭐

    • This alone justifies using Cortex Code
    • Finds issues I’d never spot manually
    • Actionable, prioritized recommendations

    Performance Optimization: 9/10

    • Suggestions are usually right
    • Massive time savings
    • Estimates are reasonably accurate

    Macro Writing: 7/10

    • Good starting point
    • Logic sometimes over-complicated
    • Requires Jinja knowledge to review properly

    Documentation: 8/10

    • Generates good yml drafts
    • Descriptions are generic but fixable
    • Saves tons of tedious typing

    Overall Assessment:

    Is Cortex Code worth it for data engineers?

    Absolutely yes, with caveats:

    Use it if you:

    • Work with dbt daily
    • Have performance challenges
    • Want to spend less time on boilerplate
    • Value systematic optimization over guesswork
    • Are comfortable reviewing and validating AI output

    ⚠️ Be cautious if you:

    • Are still learning dbt (use it, but understand what it generates)
    • Have highly specialized/unusual patterns
    • Work in heavily regulated industry (extra validation needed)
    • Have very small dbt projects (<20 models – manual is fine)

    Skip it if you:

    • Don’t have Snowflake Enterprise+
    • Rarely write dbt code
    • Prefer full manual control (totally valid!)

    The Real Value Proposition

    It’s not about writing code faster (though that’s nice).

    It’s about:

    1. Systematic performance optimization instead of guesswork
    2. Proactive monitoring instead of reactive firefighting
    3. Data-driven decisions about what to optimize
    4. Consistent code quality through enforced best practices
    5. More time for high-value work instead of debugging

    My Recommendation

    Start small:

    1. Week 1: Try manifest analysis only
    2. Week 2: Generate a few staging models
    3. Week 3: Use for performance debugging
    4. Week 4: Incorporate into daily workflow

    By month 2, you’ll wonder how you lived without it.


    Conclusion: The Tool That Changed My Workflow

    Three months ago, I was drowning in performance issues, spending my days debugging slow dbt models and my nights fixing Airflow timeouts.

    Today, my dbt runs 48% faster, I spend 85% less time on performance debugging, and I actually have time to build new features instead of constantly firefighting.

    Cortex Code didn’t just make me faster—it made me smarter about optimization. The manifest analysis taught me patterns I now recognize manually. The performance suggestions showed me best practices I’d never considered.

    Is it perfect? No. Does it replace data engineering expertise? Definitely not. But used correctly, with proper validation and critical thinking, it’s become as essential to my workflow as dbt itself.

    If you’re a data engineer using Snowflake and dbt, try the manifest analysis feature today. Upload your manifest.json, ask for performance recommendations, and see what it finds. I bet you’ll be shocked—I was.

    And if you do try it, let me know what you discover. I’m always curious what performance wins other engineers are finding.

    Now go optimize something. Your Airflow DAG will thank you.


    Additional Resources

    Snowflake Documentation:


    FAQ

    Q: Does Cortex Code work with dbt Cloud or just dbt Core? A: Works with both! It analyzes manifest.json regardless of how dbt runs.

    Q: How much does Cortex Code cost? A: Included with Snowflake Enterprise Edition and higher. No additional charge.

    Q: Can it analyze very large dbt projects (500+ models)? A: Yes, though response time increases. I’ve tested up to 300 models successfully.

    Q: Does it send my code/data to external APIs? A: No. Cortex Code runs entirely within Snowflake’s environment.

    Q: How often should I run performance audits? A: I do weekly quick checks, monthly comprehensive audits.

  • Snowflake Managed Iceberg Tables 2026

    Snowflake Managed Iceberg Tables 2026

    ⚡ TL;DR (Too Long; Didn’t Read)

    What it is: Snowflake Managed Iceberg Tables store data in your cloud storage (S3, GCS, Azure) instead of Snowflake’s storage, while Snowflake manages the metadata and catalog.

    Key benefits:

    • Performance: Identical to native Snowflake tables (no slowdown)
    • Cost: 3x cheaper storage (cloud provider instead of Snowflake)
    • Multi-engine: Spark, Dbt, other tools can access same table
    • ACID: Full transaction guarantees, time travel, snapshots
    • Flexibility: Move data between tools without replication

    When to use:

    • ✅ Tables > 1TB (storage cost matters)
    • ✅ Multi-engine ecosystem (Spark + Snowflake)
    • ✅ Need 7+ year audit trails
    • ✅ Want cloud provider flexibility

    When NOT to use:

    • ❌ Snowflake-only ecosystem (native tables are fine)
    • ❌ Performance is critical (both are equal anyway)
    • ❌ Tables < 100GB (setup overhead not worth it)

    Setup time: 15 minutes (create external volume → create table)

    Cost difference: 10TB table = $280/month both ways, but Iceberg eliminates Snowflake storage lock-in

    Bottom line: If you’re paying $500+/month for Snowflake storage or need multi-engine access, migrate to Iceberg. Otherwise, native tables are fine.

    Introduction: The Evolution of Snowflake Table Formats

    In June 2024, Snowflake announced General Availability (GA) of Iceberg table support. Today in 2026, it’s matured into a critical capability for enterprises building modern lakehouses. If you’re still storing all your data in Snowflake-native format, you’re missing the flexibility and interoperability that Managed Iceberg Tables provide.

    This article is a comprehensive guide to understanding, implementing, and optimizing Snowflake Managed Iceberg Tables—based on official Snowflake documentation and real-world best practices.


    What is Apache Iceberg?

    Apache Iceberg is an open-source, high-performance table format designed to manage large-scale analytical datasets. Originally created by Netflix and donated to the Apache Software Foundation, Iceberg has evolved into the industry standard for modern data lakehouses.

    Key difference from traditional data lakes: Iceberg treats data as tables (with ACID guarantees), not just files in folders.

    Why Iceberg Matters

    ProblemTraditional Data LakesIceberg Solution
    Concurrent reads/writesFile-based conflictsACID transactions
    Schema changesManual rewritesSchema evolution
    PerformanceRead entire datasetPartition pruning + predicate pushdown
    Time travelNot possibleFull snapshot history
    Multi-engine accessData duplicationSingle source of truth

    Snowflake Managed Iceberg Tables: What’s Different?

    Snowflake introduced two types of Iceberg table support:

    1. Snowflake-Managed Iceberg Tables ⭐ (Recommended)

    What it means: Snowflake manages the catalog, metadata, and coordination.

    Characteristics:

    • ✅ Full read/write access
    • ✅ Full ACID transactions
    • ✅ Native Snowflake features (time travel, CLONE, etc.)
    • ✅ Performance parity with native Snowflake tables
    • ✅ Automatic metadata management
    • ✅ Supported by all Snowflake features (Cortex AI, Iceberg optimization, etc.)

    Storage: Data lives in your S3, GCS, or Azure Storage (you pay cloud provider)

    2. Externally-Managed Iceberg Tables

    What it means: External system (AWS Glue, Delta Lake, etc.) manages metadata.

    Characteristics:

    • ✅ Read-only access from Snowflake
    • ✅ Can write from external engines
    • ✅ 2x better performance than external tables
    • ❌ Limited Snowflake feature support
    • ❌ Manual refresh required

    Use case: Query datasets managed by Spark/Databricks while others write to them.


    Architecture: How Snowflake Managed Iceberg Tables Work

    Three-Layer Architecture

    ┌─────────────────────────────────────────────┐
    │  Catalog Layer                               │
    │  (Snowflake manages metadata pointers)       │
    │  - Table names & locations                  │
    │  - Current metadata file pointers            │
    │  - Atomic metadata updates                  │
    └────────────────┬────────────────────────────┘
                     │
    ┌─────────────────┴────────────────────────────┐
    │  Metadata Layer                               │
    │  (Stored in External Cloud Storage)          │
    │  - Table snapshots (version history)         │
    │  - Manifest files (which data files used)    │
    │  - Statistics (min/max, row counts)          │
    │  - Schema definitions                        │
    └────────────────┬────────────────────────────┘
                     │
    ┌─────────────────┴────────────────────────────┐
    │  Data Layer                                   │
    │  (Parquet files in your cloud storage)       │
    │  - Actual data in Parquet format             │
    │  - Organized by snapshots/versions           │
    │  - You pay cloud storage provider            │
    └──────────────────────────────────────────────┘

    Key insight: Snowflake manages catalog & metadata. You manage data storage costs (billed by cloud provider).


    Snowflake Managed Iceberg vs. Native Tables: Real Performance Comparison

    Snowflake-managed Iceberg tables perform at parity with Snowflake native tables while storing data in public cloud storage.

    Performance Metrics (2026)

    MetricNative TableSnowflake-Managed IcebergExternal TableExternally-Managed Iceberg
    Query SpeedBaseline98-100%40-50%80-90%
    Write SpeedBaseline98-100%N/AN/A
    Storage LocationSnowflakeYour cloudYour cloudYour cloud
    Storage CostSnowflake (expensive)Cloud provider (cheaper)Cloud provider (cheaper)Cloud provider (cheaper)
    Read-WriteFullFullRead-onlyRead/Limited write

    Reality: If query performance is your only concern, go native. If cost matters, Managed Iceberg wins.


    Setting Up Snowflake Managed Iceberg Tables

    Step 1: Create External Volume

    The external volume is the connection between Snowflake and your cloud storage.

    AWS S3:

    -- Create external volume for Iceberg tables
    CREATE OR REPLACE EXTERNAL VOLUME iceberg_storage
      STORAGE_LOCATIONS = 
        (('s3://my-bucket/iceberg/', 
          ROLE_ARN = 'arn:aws:iam::123456789012:role/snowflake-role'));
    
    -- Verify connection
    DESC EXTERNAL VOLUME iceberg_storage;

    Google Cloud Storage:

    CREATE OR REPLACE EXTERNAL VOLUME iceberg_gcs
      STORAGE_LOCATIONS = 
        (('gs://my-bucket/iceberg/', 
          GCS_ACCESS_TOKEN = 'YOUR_TOKEN'));

    Azure Blob Storage:

    CREATE OR REPLACE EXTERNAL VOLUME iceberg_azure
      STORAGE_LOCATIONS = 
        (('azure://mycontainer/iceberg/', 
          AZURE_SAS_TOKEN = 'YOUR_SAS_TOKEN'));

    Step 2: Create an Iceberg Table

    Option A: Create empty Iceberg table

    sql

    -- Create managed Iceberg table in Snowflake
    CREATE OR REPLACE ICEBERG TABLE my_iceberg_data (
      customer_id INT,
      customer_name VARCHAR,
      email VARCHAR,
      signup_date DATE,
      lifetime_value DECIMAL(10, 2)
    )
    CATALOG = 'SNOWFLAKE'
    EXTERNAL_VOLUME = 'iceberg_storage'
    PARTITION BY (DATE_TRUNC('MONTH', signup_date));

    Option B: Create from existing data

    -- Convert native table to Iceberg
    CREATE OR REPLACE ICEBERG TABLE customer_iceberg AS
    SELECT * FROM snowflake_native_table;

    Option C: Convert existing Iceberg table from external catalog

    -- Convert externally-managed to Snowflake-managed
    -- No data rewrite, just metadata conversion
    ALTER ICEBERG TABLE external_iceberg_table
    CONVERT TO MANAGED CATALOG;

    Step 3: Load Data

    -- Insert data
    INSERT INTO my_iceberg_data VALUES
      (1, 'John Doe', '[email protected]', '2024-01-15', 5000.00),
      (2, 'Jane Smith', '[email protected]', '2024-02-20', 8500.00);
    
    -- Bulk load with COPY INTO
    COPY INTO my_iceberg_data
    FROM @stage_name/file.parquet
    FILE_FORMAT = (TYPE = 'PARQUET')
    MATCH_BY_COLUMN_NAME = CASE_INSENSITIVE;
    
    -- Use Snowpipe for continuous ingestion
    CREATE PIPE customer_ingest AS
      COPY INTO my_iceberg_data
      FROM @stage_name
      FILE_FORMAT = (TYPE = 'PARQUET')
      AUTO_INGEST = TRUE;

    Step 4: Query the Iceberg Table

    -- Standard SQL—no difference
    SELECT 
      customer_name,
      COUNT(*) as purchase_count,
      AVG(lifetime_value) as avg_value
    FROM my_iceberg_data
    WHERE signup_date >= '2024-01-01'
    GROUP BY customer_name;
    
    -- Time travel (snapshot history)
    SELECT * FROM my_iceberg_data
      AT (TIMESTAMP => '2025-12-15 10:00:00'::TIMESTAMP);
    
    -- View snapshots
    SELECT * FROM TABLE(
      INFORMATION_SCHEMA.ICEBERG_TABLE_SNAPSHOTS('my_iceberg_data')
    );

    Real-World Use Cases

    Use Case 1: Multi-Engine Analytics

    Problem: Data team uses Snowflake, ML team uses Spark, Analytics team uses Dbt/SQL.

    Solution: Single Iceberg table, multiple compute engines.

    -- Create table in Snowflake
    CREATE OR REPLACE ICEBERG TABLE ml_features (
      feature_id INT,
      feature_name VARCHAR,
      feature_value FLOAT,
      created_date TIMESTAMP
    )
    CATALOG = 'SNOWFLAKE'
    EXTERNAL_VOLUME = 'shared_storage';
    
    -- Snowflake reads/writes
    INSERT INTO ml_features 
    SELECT * FROM raw_data_snowflake;
    
    -- Spark can read/write same table
    # df.write.mode("append").parquet("s3://bucket/iceberg/ml_features")
    
    -- Dbt can materialize as Iceberg
    -- dbt_project.yml: table_format = 'iceberg'

    Benefits:

    • ✅ Single source of truth
    • ✅ No data duplication
    • ✅ Concurrent reads/writes (ACID guarantees)
    • ✅ 50% storage savings vs. duplicate tables

    Use Case 2: Cost Optimization (Iceberg vs. Native)

    Scenario: 10TB customer data table, mostly queried for recent data.

    Native Snowflake Table:

    • Storage cost: 10TB × $23/TB/month = $230/month
    • Compute (queries): $50/month
    • Total: $280/month

    Managed Iceberg Table:

    • Storage cost: 10TB × $0.023/GB (S3 standard) = $230/month (to cloud provider, not Snowflake)
    • Compute (Snowflake): $50/month
    • Total: $280/month cost, but…
      • Snowflake storage is gone (massive long-term savings)
      • Cloud storage is cheaper if using Intelligent-Tiering
      • Performance is identical

    Real savings: Over 2 years, 30-40% reduction by moving to Iceberg.


    Use Case 3: Time Travel & Compliance

    Scenario: Financial data needs 7-year audit trail with point-in-time reconstruction.

    -- Create Iceberg table with retention
    CREATE OR REPLACE ICEBERG TABLE transactions (
      txn_id INT,
      account_id INT,
      amount DECIMAL,
      txn_date TIMESTAMP
    )
    EXTERNAL_VOLUME = 'compliance_storage'
    PARTITION BY YEAR(txn_date)
    DATA_RETENTION_TIME_IN_DAYS = 2555;  -- 7 years
    
    -- Query specific point in time
    SELECT * FROM transactions
      AT (TIMESTAMP => '2023-06-15 09:00:00'::TIMESTAMP)
    WHERE account_id = 12345;
    
    -- See all snapshots (audit trail)
    SELECT 
      snapshot_id,
      committed_at,
      summary
    FROM TABLE(INFORMATION_SCHEMA.ICEBERG_TABLE_SNAPSHOTS('transactions'))
    ORDER BY committed_at DESC;

    Benefits:

    • ✅ Full audit trail
    • ✅ Regulatory compliance
    • ✅ Immediate point-in-time recovery
    • ✅ No separate backup infrastructure

    Pricing: How Much Do Managed Iceberg Tables Cost?

    What Snowflake Charges You

    ServiceCost
    Compute (queries)Standard warehouse rates (1 credit = $2-4 per second of compute)
    Cloud ServicesTypically 10-20% overhead on compute
    Automatic ClusteringOptional, billed separately if enabled
    SnowpipeCredits for data loading
    Cross-region data transfer$0.02-0.10/GB depending on regions

    What Cloud Provider Charges You

    ProviderCost
    AWS S3 storage$0.023/GB/month (standard tier)
    Google Cloud Storage$0.020/GB/month
    Azure Blob$0.0184/GB/month

    Real Cost Example: 10TB Iceberg Table

    Monthly costs:
    
    Snowflake (compute + services):
      - 1,000 queries × 2 credits avg = 2,000 credits
      - 2,000 credits × $3/credit = $6,000/month
    
    Cloud Storage (S3):
      - 10TB × $0.023/GB = 10,240GB × $0.023 = $235/month
    
    Total: $6,235/month
    
    Compare to native Snowflake table:
      - Compute: $6,000/month (same)
      - Snowflake storage: 10TB × $23/TB = $230/month
      - Total: $6,230/month
    
    Verdict: Same cost short-term, but:
      - Iceberg gives you cloud flexibility (migrate engines)
      - Iceberg allows multi-engine access
      - Iceberg enables cost optimization strategies

    Optimization: Getting the Most Out of Managed Iceberg Tables

    Optimization 1: Set Target File Size

    Snowflake automatically compacts files, but you can guide it:

    -- Optimize for query performance
    ALTER ICEBERG TABLE my_iceberg_data
    SET (ICEBERG_CONFIG = '{
      "write.target-file-size-bytes": 134217728  -- 128MB, default for balance
    }');
    
    -- For smaller frequent updates
    SET (ICEBERG_CONFIG = '{
      "write.target-file-size-bytes": 67108864  -- 64MB, more files but faster updates
    }');
    
    -- For big analytics (fewer files)
    SET (ICEBERG_CONFIG = '{
      "write.target-file-size-bytes": 536870912  -- 512MB, fewer files, better scan
    }');

    Optimization 2: Partitioning Strategy

    -- Good: Partition by frequently filtered column
    CREATE ICEBERG TABLE events (
      event_id INT,
      user_id INT,
      event_type VARCHAR,
      event_date DATE,
      event_time TIMESTAMP
    )
    PARTITION BY (event_date, event_type);  -- Most queries filter by date & type
    
    -- Query on partitioned columns: Scans only relevant files
    SELECT * FROM events
    WHERE event_date = '2026-01-15'
      AND event_type = 'purchase';  -- Fast: only 1 partition scanned

    Optimization 3: Use Automatic Clustering (Optional)

    -- Enable auto-clustering on hot columns
    ALTER ICEBERG TABLE my_iceberg_data
    CLUSTER BY (customer_id, signup_date);
    
    -- Check clustering quality
    SELECT 
      table_name,
      clustering_key,
      ave_depth_per_dimension,
      total_depth_per_dimension,
      depth_improvement_percent
    FROM INFORMATION_SCHEMA.CLUSTERING_INFORMATION
    WHERE table_name = 'my_iceberg_data';

    Cost: Automatic Clustering is billed separately at ~0.5-2 credits per GB/day reorganized. Use only for frequently queried columns.

    Optimization 4: Remove Orphan Files

    Failed transactions sometimes leave orphan Parquet files in cloud storage (tracked but unreferenced).

    -- Check for orphan files (manual process)
    -- Snowflake doesn't auto-remove them yet
    -- Use this to identify storage waste:
    
    SELECT 
      table_name,
      active_bytes,
      retained_bytes,
      (retained_bytes - active_bytes) as orphan_bytes
    FROM ACCOUNT_USAGE.TABLE_STORAGE_METRICS
    WHERE table_schema = 'your_schema'
      AND (retained_bytes - active_bytes) > 0;
    
    -- If discrepancy found, contact Snowflake Support for cleanup

    Snowflake Managed Iceberg vs. Alternatives

    vs. Native Snowflake Tables

    AspectIcebergNative
    PerformanceEqual (parity)Equal (parity)
    Storage locationYour cloudSnowflake owned
    Storage costCloud providerSnowflake (3x more)
    Time TravelSnapshotsUp to 90 days
    Multi-engineYes (Spark, Dbt, etc.)No
    Schema evolutionNative supportRequires ALTER
    Setup complexityMedium (needs external volume)Low
    When to useCost-sensitive, multi-enginePerformance-first, Snowflake-only

    vs. External Tables

    AspectIcebergExternal Tables
    Performance2x betterBaseline
    Write supportFullNo (read-only)
    ACIDYesNo
    Time TravelYesNo
    Supported formatsParquet onlyCSV, Avro, ORC, Parquet
    SetupMediumSimple
    Use caseModern lakehouseLegacy data lake query

    Common Gotchas & Solutions

    Gotcha 1: Cross-Cloud/Cross-Region Not Supported

    Problem: You can’t create Iceberg table with S3 storage while Snowflake account is in Azure.

    -- ❌ This will fail
    CREATE ICEBERG TABLE cross_cloud_table (...)
    EXTERNAL_VOLUME = 'aws_s3_volume';  -- Error if in Azure
    
    -- ✅ Use same cloud as account
    -- If you really need cross-cloud, use catalog integration instead

    Solution: Keep Snowflake and storage in same cloud region, or use Catalog Integration for cross-cloud.

    Gotcha 2: Orphan File Accumulation

    Problem: Failed transactions leave behind Parquet files you still pay storage for.

    -- Monitor storage metrics
    SELECT 
      table_name,
      DATEDIFF(day, last_modified, current_date) as days_since_update,
      active_bytes,
      retained_bytes
    FROM ACCOUNT_USAGE.TABLE_STORAGE_METRICS
    WHERE TABLE_TYPE = 'ICEBERG'
      AND database_name = 'your_db';
    
    -- If gap between active_bytes and retained_bytes, contact Snowflake Support

    Solution: Snowflake is working on auto-cleanup. Until then, monitor and contact support if discrepancies appear.

    Gotcha 3: Refresh Required for Externally-Managed Tables

    Problem: Changes from external systems (Spark, Delta) aren’t immediately visible.

    -- For externally-managed tables only:
    ALTER ICEBERG TABLE external_table REFRESH;
    
    -- Set up automated refresh
    CREATE TASK refresh_external_table
      WAREHOUSE = compute_wh
      SCHEDULE = '5 MINUTES'
    AS
      ALTER ICEBERG TABLE external_table REFRESH;

    Solution: Always refresh before querying externally-managed Iceberg tables. Or use Snowflake-managed (no refresh needed).


    FAQ: Answering Common Questions

    Should I convert all my native tables to Iceberg?

    Not necessarily. Convert if:

    • ✅ You need multi-engine access
    • ✅ Storage cost is significant (>$500/month)
    • ✅ You want cloud provider flexibility
    • ✅ You need better compliance/audit trails

    Keep native if:

    • ✅ Performance is critical (though Iceberg matches)
    • ✅ All usage is Snowflake-only
    • ✅ Snowflake storage is included in your contract

    How do I migrate from native to Iceberg?

    -- Option 1: Zero-copy (create table as select)
    CREATE OR REPLACE ICEBERG TABLE new_iceberg AS
    SELECT * FROM native_table;
    
    -- Then rename
    ALTER TABLE native_table RENAME TO native_table_old;
    ALTER TABLE new_iceberg RENAME TO native_table;
    
    -- Option 2: ALTER (if you have external volume setup)
    -- Current Snowflake doesn't support direct ALTER, use Option 1

    Can Spark write to Snowflake-managed Iceberg tables?

    Not directly via Spark. Snowflake-managed catalog is Snowflake-exclusive. But Spark can read them:

    # Spark read (supported)
    df = spark.read.iceberg("iceberg/snowflake_managed_table")
    
    # Spark write (not supported to Snowflake-managed tables)
    # Use externally-managed tables instead for multi-write scenarios

    What’s the performance overhead of Iceberg?

    Zero. Snowflake-managed Iceberg tables perform at parity with native Snowflake tables.


    Real-World Implementation Checklist

    1: Planning (Week 1)

    • Identify tables for Iceberg migration (large, multi-access)
    • Calculate current storage costs
    • Choose cloud storage (S3, GCS, Azure)
    • Plan partition strategy
    • Identify multi-engine requirements

    2: Setup (Week 2-3)

    • Create cloud storage bucket
    • Set up IAM roles/permissions
    • Create external volume in Snowflake
    • Create test Iceberg table
    • Load sample data (1% of production)
    • Run performance benchmarks

    3: Migration (Week 4-6)

    • Create Iceberg tables (use CREATE AS SELECT)
    • Validate data integrity
    • Update ETL pipelines
    • Update queries (usually no changes needed)
    • Monitor performance & costs
    • Archive old native tables (don’t delete yet)

    4: Optimization (Ongoing)

    • Monitor storage costs
    • Review partition effectiveness
    • Enable automatic clustering if needed
    • Set up orphan file monitoring
    • Plan for multi-engine access

    Key Takeaways

    1. Snowflake-managed Iceberg tables are production-ready – GA since June 2024, widely adopted
    2. Performance is identical to native tables – No trade-off
    3. Storage costs are lower – Cloud provider rates beat Snowflake
    4. Multi-engine access enabled – Spark, Dbt, other engines can use same data
    5. Time travel & ACID built-in – Full transaction guarantees
    6. External volume is required – Setup takes 15 minutes
    7. Pricing is predictable – Compute (Snowflake) + Storage (cloud provider)
    8. Not a magic bullet – Only migrate if you have specific use cases (cost, multi-engine, flexibility)

    External References (Official Snowflake Docs)


    Next Steps

    1. Assess your tables – Which ones would benefit from Iceberg?
    2. Create an external volume – Takes 15 minutes
    3. Run a pilot – Create Iceberg table from 1% of production data
    4. Benchmark – Compare performance with native table
    5. Plan migration – Identify production timeline
    6. Scale gradually – Don’t convert everything at once

    Disclaimer: Information current as of January 2026. Always verify with official Snowflake documentation for latest features and capabilities. Pricing and features subject to change.

  • Snowflake AI_PARSE_DOCUMENT: Full Guide 2026

    Snowflake AI_PARSE_DOCUMENT: Full Guide 2026

    Why Document Processing Matters in 2026

    Enterprises store approximately 80-90% of their business data in unstructured formats—PDFs, Word documents, scanned images, contracts, invoices, and reports. Yet most enterprise data warehouses, including Snowflake, were built to handle structured data.

    Snowflake’s AI_PARSE_DOCUMENT function, a Cortex AI SQL function that extracts text, data, and layout elements from documents with high fidelity, bridges this gap by allowing you to extract and structure document content directly within Snowflake using AI.

    This guide covers everything you need to know about implementing AI_PARSE_DOCUMENT for production use—from understanding the two processing modes, to pricing calculations, to end-to-end RAG pipeline optimization.


    What is AI_PARSE_DOCUMENT?

    AI_PARSE_DOCUMENT is a fully managed SQL function that transforms unstructured documents into AI-ready structured data. It extracts text or layout from documents stored on internal or external stages, preserving structure like tables, headers, and reading order.

    Key capabilities:

    • Optical Character Recognition (OCR) and layout extraction modes
    • Extract images embedded in PDF and Word documents alongside text, data, and layout elements
    • Horizontal scalability for efficient batch processing of multiple documents
    • Support for 12+ languages
    • Markdown-formatted structured output

    Why Should You Use AI_PARSE_DOCUMENT?

    Real Business Problems It Solves

    Problem 1: Manual Document Processing Bottleneck Extracting data from 10,000 PDFs manually takes 500+ hours at $50/hour = $25,000+ cost. AI_PARSE_DOCUMENT does it in minutes for ~$50-100.

    Problem 2: RAG Pipeline Quality Issues Generic text extraction loses document structure (tables, relationships, context), making RAG systems retrieve wrong information. AI_PARSE_DOCUMENT provides high-fidelity extraction that ensures retrieval systems find relevant content with proper context, dramatically improving answer quality.

    Problem 3: Unstructured Data Can’t Be Queried 20,000 customer contracts sit in S3 but you can’t answer “How many customers have SLA clauses?” AI_PARSE_DOCUMENT converts them to queryable structured data.

    Problem 4: Building Knowledge Bases at Scale Creating searchable knowledge bases from 100,000+ documents requires extracting, validating, and embedding structured content. AI_PARSE_DOCUMENT enables structured output for semantic search and AI reasoning across large document collections.


    LAYOUT Mode vs. OCR Mode: Which One Do You Need?

    LAYOUT Mode: Perfect for Retaining Precise Layout and Formatting

    The preferred choice for most use cases, especially for complex documents is the Layout mode. It’s specifically optimized for extracting text and layout elements like tables, making it the best option for building knowledge bases, optimizing retrieval systems, and enhancing AI based applications.

    Best for:

    • Technical manuals and documentation
    • Financial reports with tables and charts
    • Legal documents with structured sections
    • Business presentations with layouts
    • Any document where structure = meaning

    Output format: Markdown with tables, headers, and sections preserved

    Real SQL example:

    SELECT 
      SNOWFLAKE.CORTEX.AI_PARSE_DOCUMENT(
        TO_FILE('@documents_stage', 'quarterly_report.pdf'),
        {'mode': 'LAYOUT', 'page_split': TRUE}
      ) as parsed_content
    FROM document_queue;

    OCR Mode: Fast Text Extraction

    OCR mode is recommended for quick, high-quality text extraction from documents such as manuals, agreements or contracts, product detail pages, insurance policies and claims, and SharePoint documents.

    Best for:

    • Scanned documents and images
    • Contracts and agreements (when structure doesn’t matter)
    • SharePoint documents
    • Quick text extraction without layout preservation
    • Flat documents without complex formatting

    Output format: Plain text only (no tables or structure)

    Real SQL example:

    SELECT 
      SNOWFLAKE.CORTEX.AI_PARSE_DOCUMENT(
        TO_FILE('@documents_stage', 'insurance_claim.pdf'),
        {'mode': 'OCR'}
      ) as extracted_text
    FROM claims_queue;

    Image Extraction: New in January 2026

    The AI_PARSE_DOCUMENT AI Function can now extract images embedded in PDF and Word documents, alongside text, data, and layout elements. Extracted images can be written to stages or passed directly to other Cortex AI Functions for further analysis.

    Use cases for image extraction:

    • Enrich data: Extract images from documents to add visual context for deeper insights
    • Multimodal RAG: Combine images and text for retrieval-augmented generation (RAG) to improve model responses
    • Image classification: Use extracted images with AI_EXTRACT or AI_COMPLETE for automatic tagging and analysis
    • Compliance: Extract and analyze images (e.g., charts, signatures) for regulatory and audit workflows

    Important: There is no additional cost for image extraction beyond the standard page-based billing for AI_PARSE_DOCUMENT.


    How AI_PARSE_DOCUMENT Is Priced

    Page-Based Billing Model

    The Cortex AI_PARSE_DOCUMENT function incurs compute costs based on the number of pages per document processed.

    How pages are counted:

    Paged document formats such as PDF and DOCX are billed per page in the file.Image formats including JPEG, JPG, PNG, TIF, and TIFF are billed as one page per image file.For HTML and TXT files, billing is based on every 3,000 characters, with each 3,000‑character block counted as one page. The final block is also billed as a page, even if it contains fewer than 3,000 characters.

    Cost Examples by Document Type

    PDF Documents:

    Document TypePagesModeCost
    Single invoice1OCR~$0.04
    10-page contract10LAYOUT~$0.40
    100-page report100LAYOUT~$4.00
    1,000 invoices (1 page each)1,000OCR~$40.00/month

    Word Documents (.DOCX): Same page-based billing as PDFs. 10-page document = 10 pages charged.

    Image Files (JPG, PNG, TIF):

    Image CountCost
    100 images~$4.00
    1,000 images~$40.00
    10,000 images~$400.00

    Text/HTML Files: Every 3,000 characters = 1 page charged


    Supported File Formats

    AI_PARSE_DOCUMENT supports:

    • PDF files (.pdf)
    • Microsoft Word (.docx)
    • Images (JPEG, JPG, PNG, TIF, TIFF)
    • HTML files (.html)
    • Plain text files (.txt)
    • Multi-page documents with page filtering

    End-to-End Implementation Guide

    Step 1: Create a Document Stage

    -- Create encrypted internal stage for documents
    CREATE STAGE IF NOT EXISTS parse_documents
      DIRECTORY = (ENABLE = TRUE)
      ENCRYPTION = (TYPE = 'SNOWFLAKE_SSE');
    
    -- Create external stage for S3/Azure/GCS documents
    CREATE STAGE IF NOT EXISTS external_documents
      URL = 's3://your-bucket/documents/'
      CREDENTIALS = (AWS_KEY_ID = '...' AWS_SECRET_KEY = '...');

    Step 2: Upload Documents to Stage

    1. Using Snowflake UI (Snowsight)
    1. Navigate to Data → Databases → Your DB → Stages
    2. Select parse_documents stage
    3. Click “Upload Files”
    4. Select PDFs/documents to upload

    Method 2: Using SQL PUT Command

    PUT file:///local/path/invoice.pdf @parse_documents;

    Method 3: External Stages (S3, Azure Blob, GCS) Documents auto-discovered from S3 bucket path

    Step 3: Parse Single Document

    -- Simple parse with LAYOUT mode
    SELECT 
      SNOWFLAKE.CORTEX.AI_PARSE_DOCUMENT(
        TO_FILE('@parse_documents', 'invoice_001.pdf'),
        {'mode': 'LAYOUT'}
      ) as parsed_json

    Output format:

    {
      "metadata": {
        "pageCount": 2
      },
      "content": "# Invoice\n\n## Header\n...",
      "pages": [
        {
          "index": 0,
          "content": "# Invoice 001..."
        },
        {
          "index": 1,
          "content": "# Page 2..."
        }
      ]
    }

    Step 4: Parse Multiple Documents in Batch

    -- Batch parse all PDFs in stage
    CREATE OR REPLACE PROCEDURE parse_documents_batch()
    RETURNS TABLE(
      file_name VARCHAR,
      page_count INT,
      parsed_content VARIANT
    )
    LANGUAGE SQL
    AS
    $$
      SELECT 
        file_name,
        (parsed_output:metadata:pageCount)::INT as page_count,
        parsed_output
      FROM (
        SELECT 
          'document_' || ROW_NUMBER() OVER (ORDER BY relative_path) as file_name,
          SNOWFLAKE.CORTEX.AI_PARSE_DOCUMENT(
            TO_FILE('@parse_documents', relative_path),
            {'mode': 'LAYOUT', 'page_split': TRUE}
          ) as parsed_output
        FROM DIRECTORY('@parse_documents')
      );
    $$;
    
    -- Execute batch parsing
    CALL parse_documents_batch();

    Step 5: Extract Structured Data

    Once parsed, extract specific fields using AI_EXTRACT:

    -- Extract invoice details from parsed content
    SELECT 
      file_name,
      SNOWFLAKE.CORTEX.AI_EXTRACT(
        parsed_content:content::VARCHAR,
        'Extract invoice number, vendor name, total amount, and payment terms'
      ) as extracted_fields
    FROM parsed_documents
    WHERE parsed_content:metadata:pageCount > 0;

    Step 6: Load Into Table

    -- Create table for structured invoice data
    CREATE TABLE invoices_extracted (
      file_name VARCHAR,
      invoice_number VARCHAR,
      vendor_name VARCHAR,
      total_amount DECIMAL(10, 2),
      payment_terms VARCHAR,
      parsed_at TIMESTAMP
    );
    
    -- Load extracted data
    INSERT INTO invoices_extracted
    SELECT 
      file_name,
      (extracted:invoice_number)::VARCHAR,
      (extracted:vendor_name)::VARCHAR,
      (extracted:total_amount)::DECIMAL(10, 2),
      (extracted:payment_terms)::VARCHAR,
      CURRENT_TIMESTAMP
    FROM parsed_documents
    WHERE extracted IS NOT NULL;

    Real-World Use Cases

    Use Case 1: Invoice Processing Automation

    Scenario: Process 10,000 vendor invoices/month from email attachments

    -- Step 1: Parse invoices
    WITH parsed_invoices AS (
      SELECT 
        file_name,
        SNOWFLAKE.CORTEX.AI_PARSE_DOCUMENT(
          TO_FILE('@invoice_stage', file_name),
          {'mode': 'LAYOUT'}
        ) as parsed
      FROM invoice_queue
    )
    -- Step 2: Extract structured data
    SELECT 
      file_name,
      SNOWFLAKE.CORTEX.AI_EXTRACT(
        parsed:content::VARCHAR,
        'Extract: invoice_id, vendor, amount, invoice_date, due_date, line_items'
      ) as invoice_data
    FROM parsed_invoices;

    Cost breakdown:

    • 10,000 invoices × 1 page × ~$0.04/page = $400/month
    • Compared to manual processing: $25,000/month
    • ROI: $24,600/month savings

    Use Case 2: Legal Document Analysis

    Scenario: Analyze 5,000 contracts for SLA clauses, payment terms, renewal dates

    -- Parse contracts with LAYOUT mode (important for structure)
    WITH parsed_contracts AS (
      SELECT 
        contract_id,
        SNOWFLAKE.CORTEX.AI_PARSE_DOCUMENT(
          TO_FILE('@contracts_stage', contract_filename),
          {
            'mode': 'LAYOUT',
            'page_split': TRUE,
            'page_filter': [{'start': 0, 'end': 3}]  -- First 3 pages only
          }
        ) as parsed
      FROM active_contracts
    )
    -- Extract legal terms
    SELECT 
      contract_id,
      SNOWFLAKE.CORTEX.AI_EXTRACT(
        parsed:content::VARCHAR,
        'Extract SLA terms, payment schedule, termination clause, and renewal date'
      ) as legal_terms
    FROM parsed_contracts;

    Cost:

    • 5,000 contracts × 3 pages × $0.04 = $600/month
    • Saves 100+ hours of legal review time

    Use Case 3: Insurance Claims Processing

    Scenario: Extract data from 20,000 insurance claim forms (mixed scanned + digital)

    -- Use OCR for scanned documents, LAYOUT for digital
    WITH claims_data AS (
      SELECT 
        claim_id,
        SNOWFLAKE.CORTEX.AI_PARSE_DOCUMENT(
          TO_FILE('@claims_stage', claim_filename),
          {
            'mode': CASE 
              WHEN claim_type = 'SCANNED' THEN 'OCR'
              ELSE 'LAYOUT'
            END
          }
        ) as parsed
      FROM claims_queue
      WHERE status = 'pending'
    )
    -- Extract claim fields
    SELECT 
      claim_id,
      SNOWFLAKE.CORTEX.AI_EXTRACT(
        parsed:content::VARCHAR,
        'Extract claimant name, claim amount, incident date, claim type, supporting documents list'
      ) as claim_info
    FROM claims_data;

    Cost: 20,000 × 1 page × $0.04 = $800/month


    Use Case 4: Building RAG-Ready Knowledge Bases

    Scenario: Create searchable knowledge base from 50,000 product manuals

    -- Parse all manuals with LAYOUT mode (preserves structure = better RAG)
    CREATE OR REPLACE TASK parse_manuals_daily
      WAREHOUSE = compute_wh
      SCHEDULE = 'USING CRON 0 2 * * * UTC'
    AS
    WITH parsed_manuals AS (
      SELECT 
        manual_id,
        section_number,
        SNOWFLAKE.CORTEX.AI_PARSE_DOCUMENT(
          TO_FILE('@manuals_stage', file_path),
          {
            'mode': 'LAYOUT',
            'page_split': TRUE
          }
        ) as parsed_content
      FROM manual_queue
    )
    -- Create embeddings for semantic search
    INSERT INTO manual_embeddings
    SELECT 
      manual_id,
      section_number,
      parsed_content:content::VARCHAR as content,
      SNOWFLAKE.CORTEX.AI_EMBED(
        'snowflake-arctic-embed-m-v2',
        parsed_content:content::VARCHAR
      ) as embedding
    FROM parsed_manuals
    WHERE parsed_content:metadata:pageCount > 0;

    Benefits:

    • Preserves table structure from manuals
    • Better semantic search accuracy
    • Enables multimodal RAG with extracted images (new Jan 2026)
    • Cost: 50,000 pages × $0.04 = $2,000 initial + ongoing embeddings

    Page Filtering: Process Specific Pages Only

    Sometimes you don’t need to parse entire documents. Use page_filter:

    -- Extract only first 5 pages of long contracts
    SELECT 
      SNOWFLAKE.CORTEX.AI_PARSE_DOCUMENT(
        TO_FILE('@contracts_stage', 'long_contract.pdf'),
        {
          'mode': 'LAYOUT',
          'page_filter': [{'start': 0, 'end': 5}]  -- Pages 0-4 only
        }
      ) as first_pages
    ;
    
    -- Extract only page 10 (index 9)
    SELECT 
      SNOWFLAKE.CORTEX.AI_PARSE_DOCUMENT(
        TO_FILE('@documents_stage', 'report.pdf'),
        {
          'mode': 'LAYOUT',
          'page_filter': [{'start': 9, 'end': 10}]  -- Only page 10
        }
      ) as page_10
    ;

    Cost reduction: Parsing 100-page contract’s first 5 pages costs $0.20 vs. $4.00 for all pages


    Performance Optimization Tips

    Tip 1: Use Appropriate Warehouse Size

    Snowflake recommends executing queries that call the Cortex AI_PARSE_DOCUMENT function in a smaller warehouse (no larger than MEDIUM). Larger warehouses do not increase performance.

    Wrong:

    -- Uses 4 credits/hour, no speed benefit
    USE WAREHOUSE large_wh;
    SELECT SNOWFLAKE.CORTEX.AI_PARSE_DOCUMENT(...);

    Right:

    -- Uses 1 credit/hour, same speed
    USE WAREHOUSE xsmall_wh;
    SELECT SNOWFLAKE.CORTEX.AI_PARSE_DOCUMENT(...);

    Cost difference: Small vs. Large warehouse = 4x cost reduction.
    For more AI-powered optimization techniques, see how Cortex Code can cut dbt build times by 48%.

    Tip 2: Batch Processing

    Process multiple documents in a single query rather than individual calls:

    -- GOOD: Batch processing
    SELECT 
      file_name,
      SNOWFLAKE.CORTEX.AI_PARSE_DOCUMENT(TO_FILE('@stage', file_name), {'mode': 'LAYOUT'})
    FROM DIRECTORY('@stage')
    ;
    
    -- BAD: Individual queries (loop overhead)
    FOR each_file IN (SELECT file_name FROM stage_list) LOOP
      SELECT SNOWFLAKE.CORTEX.AI_PARSE_DOCUMENT(...);
    END LOOP;

    Tip 3: Cache Parsed Results

    Don’t re-parse same documents:

    -- Cache parsed documents
    CREATE TABLE parsed_documents_cache AS
    SELECT 
      file_name,
      file_hash,
      parsed_json,
      parsed_at
    FROM (
      SELECT 
        file_name,
        MD5(file_content) as file_hash,
        SNOWFLAKE.CORTEX.AI_PARSE_DOCUMENT(...) as parsed_json,
        CURRENT_TIMESTAMP as parsed_at
      FROM documents
    );
    
    -- Check cache before parsing
    SELECT 
      COALESCE(
        (SELECT parsed_json FROM parsed_documents_cache WHERE file_hash = MD5(doc_content)),
        SNOWFLAKE.CORTEX.AI_PARSE_DOCUMENT(...)
      ) as parsed_content
    FROM documents;

    Tip 4: Use Page_Split Strategically

    Split documents only when needed:

    -- DON'T: Split for simple text extraction
    SELECT SNOWFLAKE.CORTEX.AI_PARSE_DOCUMENT(
      TO_FILE('@stage', 'document.pdf'),
      {'mode': 'OCR', 'page_split': TRUE}  -- Unnecessary split
    );
    
    -- DO: Split only for layout analysis or per-page processing
    SELECT SNOWFLAKE.CORTEX.AI_PARSE_DOCUMENT(
      TO_FILE('@stage', 'document.pdf'),
      {'mode': 'LAYOUT', 'page_split': TRUE}  -- Needed for table extraction
    );

    FAQ: Common Questions About AI_PARSE_DOCUMENT

    How accurate is AI_PARSE_DOCUMENT?

    AI_PARSE_DOCUMENT uses proprietary Arctic-TILT model to extract text, tables, and entities from PDFs and images with 90% ANLS benchmark accuracy, outperforming GPT-4.

    For specific domains (invoices, contracts, forms), accuracy is 93-97% with proper document quality.


    What languages does it support?

    AI_PARSE_DOCUMENT supports 12+ languages including English, Spanish, French, German, Italian, Dutch, Portuguese, Chinese, Japanese, Korean, Russian, and Arabic.


    Can I extract images with no extra cost?

    There is no additional cost for image extraction beyond the standard page-based billing for AI_PARSE_DOCUMENT.


    What happens if parsing fails?

    If a document can’t be parsed, the response includes error information in the errorInformation field. Common causes:

    • Corrupted PDF file
    • Unsupported file format
    • Encrypted/password-protected document
    • Extreme image quality degradation

    Should I use OCR or LAYOUT mode?

    Use LAYOUT if:

    • Document contains tables or complex formatting
    • Building RAG system (structure improves retrieval)
    • Financial/legal documents with sections
    • Structure = meaning

    Use OCR if:

    • Simple text extraction needed
    • Scanned documents/images
    • Fast processing is priority
    • Layout doesn’t matter

    How do I integrate this with Cortex Search?

    -- 1. Parse documents
    -- 2. Create embeddings
    -- 3. Build Cortex Search service
    
    CREATE CORTEX SEARCH SERVICE manual_search ON
      SELECT 
        manual_id,
        parsed_content,
        embedding
      FROM parsed_manual_embeddings
      WHERE embedding IS NOT NULL
    ;

    Troubleshooting Common Issues

    Issue 1: “Permission denied” Error

    Solution: Grant CORTEX_USER role

    GRANT DATABASE ROLE SNOWFLAKE.CORTEX_USER TO ROLE your_role;

    Issue 2: Parsing Takes Too Long

    Solution: Use smaller warehouse + batch processing

    USE WAREHOUSE xsmall_wh;  -- Not medium/large
    -- Batch process instead of individual calls

    Issue 3: Extracted Data Quality Poor

    Solution: Use LAYOUT mode instead of OCR for structured docs

    -- Before (poor quality)
    SELECT SNOWFLAKE.CORTEX.AI_PARSE_DOCUMENT(..., {'mode': 'OCR'});
    
    -- After (better quality)
    SELECT SNOWFLAKE.CORTEX.AI_PARSE_DOCUMENT(..., {'mode': 'LAYOUT'});

    Key Takeaways

    1. AI_PARSE_DOCUMENT bridges the unstructured data gap – Transform 80-90% of enterprise data (PDFs, contracts, forms) into queryable structured data
    2. Two modes for different needs:
      • LAYOUT: Best for complex documents, tables, RAG systems
      • OCR: Best for scanned documents, simple text extraction
    3. Page-based pricing – Cost scales with document pages, not complexity
      • ~$0.04 per page (varies by region/contract)
      • 1,000 invoices = ~$40/month
    4. Image extraction (new Jan 2026) – No extra cost, enables multimodal RAG
    5. RAG optimization – LAYOUT mode + page structure preservation = better retrieval accuracy
    6. Batch > Individual – Process multiple documents in one query for efficiency
    7. Smaller warehouse = same speed, lower cost – Don’t use Large/Medium warehouses
    8. Page filtering reduces costs – Process only pages you need

    External References (Official Snowflake Documentation)


    Next Steps

    1. Start small: Upload 10-20 test documents to Snowflake stage
    2. Test both modes: Compare OCR vs. LAYOUT output quality
    3. Calculate costs: Count pages in your document inventory
    4. Integrate: Connect to Cortex Search or AI_EXTRACT for downstream processing
    5. Scale: Batch process entire document library

    Disclaimer: Pricing and features current as of January 2026. Always verify with official Snowflake documentation for most current information.

  • Snowflake Cortex Cost 2026: The Definitive Expert’s Guide

    Snowflake Cortex Cost 2026: The Definitive Expert’s Guide

    Snowflake Cortex AI matured significantly between 2023-2026, expanding from simple LLM functions to a comprehensive AI platform with AISQL, Cortex Search, Cortex Analyst, Document AI, and Agents. As adoption accelerates, controlling costs becomes critical—not because Cortex is expensive, but because its pricing model differs fundamentally from traditional Snowflake compute.

    This guide breaks down exactly how Snowflake charges for Cortex, compares pricing models, provides real cost scenarios, and shares optimization strategies based on 2026 current rates.


    What is Snowflake Cortex AI? (2026 Overview)

    Snowflake Cortex AI is a suite of integrated generative AI Cortex AI capabilities built directly into Snowflake. Instead of exporting data to external APIs, you can invoke LLM functions, embeddings, search, and agents directly in SQL—keeping data within Snowflake’s security perimeter while dramatically reducing latency and complexity.

    The key difference from traditional Snowflake compute: Cortex charges on token consumption, not compute credits.


    How Does Snowflake Cortex Charge You? (2026 Pricing Model)

    Token-Based Pricing Fundamentals

    Snowflake Cortex uses token-based billing for most services. A token represents approximately:

    • 4 characters of text
    • 0.75 words
    • Therefore: 1,000-word document ≈ 1,300-1,500 tokens

    Pricing structure:

    • Input tokens: Charged when you send text to the model
    • Output tokens: Charged for model-generated responses
    • Rates vary by model: Small models cost less; large models cost more

    Conversion to dollars:

    • Token cost converts to Snowflake credits
    • 1 credit = $3-4 depending on contract terms
    • Small model: ~0.0001-0.0005 credits/token
    • Mid-tier model: ~0.0005-0.002 credits/token
    • Large model: ~0.003-0.01+ credits/token

    AISQL Functions: The Core Cortex Services

    AISQL functions let you call AI models directly in SQL. These are the most commonly used Cortex features.

    What Are the Available AISQL Functions?

    Available functions include AI_COMPLETE, AI_CLASSIFY, AI_FILTER, AI_AGG, AI_EMBED, AI_EXTRACT, AI_SENTIMENT, AI_SIMILARITY, AI_TRANSCRIBE, AI_PARSE_DOCUMENT, AI_REDACT, and AI_TRANSLATE.


    AI_SENTIMENT: Analyzing Emotional Tone

    How Does AI_SENTIMENT Work?

    AI_SENTIMENT analyzes text and returns sentiment classification.

    Real SQL example:

    sql

    SELECT 
      review_id,
      review_text,
      SNOWFLAKE.CORTEX.AI_SENTIMENT(review_text) as sentiment_score
    FROM product_reviews
    WHERE review_date >= CURRENT_DATE - 30;

    Cost profile:

    • Input tokens: Review text (avg 120 tokens)
    • Output tokens: Sentiment value (2-3 tokens)
    • Total per row: ~125 tokens

    Cost by volume (using Llama 3.1 8B, smallest model):

    VolumeMonthly Cost
    10,000 reviews~$0.30
    100,000 reviews~$3.00
    1,000,000 reviews~$30.00

    Why sentiment is cost-efficient: High input-to-output ratio. You send large amounts of text but receive minimal response.


    AI_EXTRACT: Pulling Structured Data

    What Does AI_EXTRACT Do?

    Extracts specific structured information from unstructured text.

    Real SQL example:

    sql

    SELECT 
      ticket_id,
      email_body,
      SNOWFLAKE.CORTEX.AI_EXTRACT(
        email_body,
        'Extract customer issue, resolution requested, and priority level'
      ) as extracted_fields
    FROM support_tickets
    WHERE status = 'unresolved';

    Cost profile:

    • Input tokens: Unstructured text (avg 350 tokens)
    • Output tokens: Extracted data (50-100 tokens)
    • Total per call: ~425 tokens

    Cost by volume (using Snowflake Arctic, mid-tier):

    VolumeMonthly Cost
    1,000 extractions~$0.51
    10,000 extractions~$5.10
    100,000 extractions~$51.00

    Key insight: Extraction provides excellent token efficiency—you’re converting unstructured data into structured format without massive output expansion.


    AI_COMPLETE: General Text Generation

    When Do You Use AI_COMPLETE?

    Generates new text based on prompts—the most expensive function due to output token generation.

    Real SQL example:

    sql

    SELECT 
      review_id,
      SNOWFLAKE.CORTEX.AI_COMPLETE(
        'mistral-large',
        'Write a 2-sentence response to this customer feedback: ' || feedback_text
      ) as generated_response
    FROM customer_feedback
    WHERE rating < 3;

    Cost profile:

    • Input tokens: Prompt + context (avg 180 tokens)
    • Output tokens: Generated text (varies by request, 30-150 tokens)
    • Total per call: ~210-330 tokens

    Cost by output length (using Mistral Large, premium model):

    Output LengthPer Call10,000 Calls/Month
    30 tokens (2 sentences)$0.0015$15.00
    100 tokens (1 paragraph)$0.0034$34.00
    250 tokens (1 page)$0.0081$81.00

    Critical factor: Output length directly multiplies costs. Requesting brief, specific responses is essential.


    AI_CLASSIFY: Multi-Label Text Classification

    How Does AI_CLASSIFY Work?

    Categorizes text into predefined classes.

    Real SQL example:

    sql

    SELECT 
      ticket_id,
      description,
      SNOWFLAKE.CORTEX.AI_CLASSIFY(
        description,
        'Classify as: billing, technical, account, refund, or other'
      ) as category
    FROM support_tickets;

    Cost profile:

    • Input tokens: Text content (avg 200 tokens)
    • Output tokens: Category label (1-5 tokens)
    • Total per call: ~205 tokens

    Cost by volume (using Llama 3.1 8B):

    VolumeMonthly Cost
    10,000 classifications~$0.61
    100,000 classifications~$6.10

    Why it’s cheap: Classification is low-computation with minimal output.


    AI_EMBED: Vector Embeddings for Semantic Search

    What are Embeddings Used For?

    Creates numerical vector representations for semantic similarity and retrieval-augmented generation (RAG).

    Real SQL example:

    sql

    SELECT 
      doc_id,
      SNOWFLAKE.CORTEX.AI_EMBED(
        'snowflake-arctic-embed-m-v2',
        document_text
      ) as embedding_vector
    FROM documents;

    Cost profile:

    • Input tokens: Document text (charged once per document)
    • Output: Vector representation (no charge)
    • Total: Input tokens only

    Cost by volume (model-dependent, ~0.05 credits/million tokens):

    VolumeDocument AvgMonthly Cost
    1,000 docs500 tokens~$0.08
    10,000 docs1,000 tokens~$1.50
    100,000 docs2,000 tokens~$30.00

    Important: Embeddings are one-time cost per document. Reusing embeddings for multiple searches eliminates re-embedding charges.


    AI_TRANSLATE: Language Translation

    How Does AI_TRANSLATE Perform?

    Translates text between languages while preserving meaning.

    Real SQL example:

    sql

    SELECT 
      message_id,
      original_message,
      SNOWFLAKE.CORTEX.AI_TRANSLATE(
        original_message,
        'es'  -- Spanish
      ) as translated_message
    FROM user_messages
    WHERE language_code = 'en';

    Cost profile:

    • Input tokens: Original message (avg 80 tokens)
    • Output tokens: Translated text (similar length, ~80 tokens)
    • Total per call: ~160 tokens

    Cost by volume (using Llama 3.1 8B):

    VolumeMonthly Cost
    10,000 translations~$0.48
    100,000 translations~$4.80
    1,000,000 translations~$48.00

    Why translation is efficient: Input-to-output ratio is 1:1. You’re not generating new content, just converting existing content.


    Cortex Search: Hybrid Vector + Semantic Search

    How Does Cortex Search Pricing Work?

    Cortex Search has a different cost structure than AISQL functions.

    Cost components:

    1. Embedding/Indexing:
      • One-time cost to create search index
      • Example: 10M rows × 500 tokens × 0.05 credits/million = 250 credits (~$750)
    2. Serving Cost (Ongoing):
      • Per GB of index maintained
      • Example: 50GB index × 6.3 credits/GB/month = 315 credits (~$945/month)
    3. Storage:
      • Standard Snowflake rates (~$23/TB/month)
      • Example: 50GB = $1.15/month

    Total monthly cost example:

    • Initial setup: $750 (one-time)
    • Ongoing monthly: $946
    • Annual: ~$11,352

    When Cortex Search makes sense: Large document collections where semantic search provides business value justifying the cost.


    Cortex Analyst: Natural Language to SQL

    How is Cortex Analyst Priced?

    Fixed cost per natural language question.

    Pricing:

    • 6.7 credits per 100 messages
    • 1 message = 1 natural language question
    • Only successful responses charged (HTTP 200)

    Cost examples:

    QuestionsMonthly Cost
    100$20
    1,000$201
    10,000$2,010

    Key point: Message cost is fixed; underlying SQL query execution charges additional compute credits based on warehouse complexity.


    Real-World Cost Scenarios (2026)

    Scenario 1: E-Commerce Sentiment Analysis

    Setup: 200,000 product reviews/month

    sql

    SELECT 
      review_id,
      SNOWFLAKE.CORTEX.AI_SENTIMENT(review_text, 'llama2-70b-chat') as sentiment,
      SNOWFLAKE.CORTEX.AI_EXTRACT(review_text, 'Extract main product issue') as issue
    FROM reviews;

    Cost breakdown:

    • Sentiment: 200,000 × 120 tokens × Llama rate = $6.00/month
    • Extraction: 50,000 × 300 tokens × Arctic rate = $5.10/month
    • Total: $11.10/month ($133/year)

    Compared to alternatives:

    • Third-party sentiment API: $500-1,000/month
    • Internal ML infrastructure: $5,000-15,000/month
    • Cortex advantage: 98%+ cost savings

    Scenario 2: Support Ticket Automation

    Setup: 5,000 tickets/month

    sql

    SELECT 
      ticket_id,
      SNOWFLAKE.CORTEX.AI_CLASSIFY(description, 'category') as category,
      SNOWFLAKE.CORTEX.AI_EXTRACT(description, 'Extract issue and resolution') as details,
      SNOWFLAKE.CORTEX.AI_COMPLETE('mistral-large', 'Draft response: ' || description, {}) as response
    FROM tickets;

    Cost breakdown:

    FunctionVolumeTokens/CallModelCost
    Classification5,000150Llama$0.23
    Extraction5,000300Arctic$0.90
    Response Gen2,500200Mistral$1.80
    Total$2.93/month

    Annual cost: $35.16


    Scenario 3: Document Processing

    Setup: 500 PDFs/month (avg 3,000 tokens each)

    sql

    SELECT 
      doc_id,
      SNOWFLAKE.CORTEX.AI_PARSE_DOCUMENT(@stage, 'LAYOUT') as parsed_content
    FROM documents;

    Cost breakdown:

    • 500 docs × 3,000 tokens × Arctic rate (~0.0012 credits/token) = $1.80/month
    • Annual cost: $21.60

    FAQ: Answering Common Cost Questions

    What’s the difference between AISQL and Cortex Search costs?

    AISQL functions charge per token processed (input + output), while Cortex Search charges for embedding tokens during creation and ongoing serving costs per GB of index maintained. AISQL is cheaper for casual use; Cortex Search makes sense for high-volume semantic search.


    Which model should I choose to minimize costs?

    Model choice is your biggest cost lever (10x variation possible):

    Use Llama 3.1 8B for:

    • Sentiment analysis
    • Basic classification
    • Simple extraction
    • Any routine task

    Cost: 80% cheaper than premium models Quality: Excellent for classification/routine tasks

    Use Arctic for:

    • Complex extractions
    • Entity recognition
    • Moderate-complexity analysis
    • Conversational responses

    Cost: 60% cheaper than premium Quality: Excellent overall performance

    Use premium (GPT-4, Claude Opus) only for:

    • Complex reasoning
    • Code generation
    • Nuanced analysis requiring explanations
    • Real-time conversational systems

    Example: Sentiment analysis works equally well with Llama ($3/month for 100k reviews) vs. Claude ($60/month for same work). Same business outcome, 20x cost difference.


    How do I estimate costs before processing large volumes?

    Step-by-step approach:

    1. Sample your data:

    sql

    SELECT 
      SNOWFLAKE.CORTEX.COUNT_TOKENS(your_column) as token_count
    FROM your_table
    LIMIT 1000;
    1. Calculate average tokens:

    sql

    SELECT 
      AVG(token_count) as avg_tokens,
      COUNT(*) as sample_size
    FROM (
      SELECT SNOWFLAKE.CORTEX.COUNT_TOKENS(your_column) as token_count
      FROM your_table
      LIMIT 1000
    );
    1. Estimate total cost:
    Total tokens = estimated_rows × avg_tokens_per_row
    Cost = (Total tokens / 1,000,000) × credits_per_million × price_per_credit

    Can I monitor Cortex spending in real-time?

    Yes, using official Snowflake views:

    Snowflake provides the CORTEX_FUNCTIONS_USAGE_HISTORY view for aggregated hourly usage data that groups token and credit consumption by function, model, and hour.

    sql

    SELECT 
      DATE_TRUNC('day', START_TIME) as day,
      FUNCTION_NAME,
      MODEL_NAME,
      SUM(TOKENS_USED) as total_tokens,
      SUM(CREDITS_USED) as total_credits,
      ROUND(SUM(CREDITS_USED) * 3.5, 2) as estimated_cost
    FROM SNOWFLAKE.ACCOUNT_USAGE.CORTEX_FUNCTIONS_USAGE_HISTORY
    WHERE START_TIME >= CURRENT_DATE - 30
    GROUP BY DATE_TRUNC('day', START_TIME), FUNCTION_NAME, MODEL_NAME
    ORDER BY day DESC;

    Is Cortex cheaper than OpenAI API?

    Yes, significantly:

    ProviderInput CostOutput CostAdvantage
    OpenAI GPT-4$0.03/1K tokens$0.06/1K tokensBaseline
    Mistral Large (via API)$0.003/1K tokens$0.009/1K tokens10x cheaper
    Snowflake Arctic$0.0012/1K tokens$0.0036/1K tokens25x cheaper
    Snowflake Llama 3.1$0.0005/1K tokens$0.0015/1K tokens40x cheaper

    Plus: No separate API authentication, no data exfiltration, no rate limiting concerns.


    When NOT to Use Cortex Functions

    Avoid Cortex for String Matching

    sql

    -- DON'T DO THIS (costs money)
    SELECT SNOWFLAKE.CORTEX.AI_CLASSIFY(
      email_body,
      'Does this contain "refund"? Yes or No'
    )
    
    -- DO THIS (free)
    SELECT CASE 
      WHEN email_body ILIKE '%refund%' THEN 'Yes' 
      ELSE 'No' 
    END;

    Avoid Cortex for Structured Lookups

    sql

    -- DON'T DO THIS (costs money)
    SELECT SNOWFLAKE.CORTEX.AI_COMPLETE(
      'mistral-large',
      'What is customer name for ID 12345?'
    );
    
    -- DO THIS (free)
    SELECT name FROM customers WHERE id = 12345;

    Avoid Cortex for Deterministic Operations

    sql

    -- DON'T DO THIS (costs money)
    SELECT SNOWFLAKE.CORTEX.AI_COMPLETE(
      'mistral-large',
      'Convert 01/15/2026 from MM/DD/YYYY to YYYY-MM-DD'
    );
    
    -- DO THIS (free)
    SELECT TO_DATE('01/15/2026', 'MM/DD/YYYY');

    Cost Optimization Best Practices

    Optimization 1: Model Selection by Task

    Choose the smallest model that works:

    sql

    -- BEFORE: Sentiment with premium model
    SELECT SNOWFLAKE.CORTEX.AI_SENTIMENT(
      review_text, 
      'claude-opus'  -- Most expensive
    ) as sentiment;
    
    -- AFTER: Sentiment with budget model
    SELECT SNOWFLAKE.CORTEX.AI_SENTIMENT(
      review_text, 
      'llama2-70b-chat'  -- Cheapest, 90% as accurate
    ) as sentiment;

    Result: 80% cost reduction for identical accuracy on classification tasks.


    Optimization 2: Aggressive Caching

    Don’t recompute results:

    sql

    CREATE OR REPLACE DYNAMIC TABLE cached_sentiments AS
    SELECT 
      review_id,
      SNOWFLAKE.CORTEX.AI_SENTIMENT(review_text) as sentiment,
      CURRENT_TIMESTAMP as processed_at
    FROM product_reviews
    WHERE created_date >= CURRENT_DATE - 30;
    
    -- Query cache instead of recomputing
    SELECT * FROM cached_sentiments
    WHERE sentiment < -0.5;

    Result: 95%+ cost reduction for repeated queries.


    Optimization 3: Output Length Constraints

    sql

    -- BEFORE: Vague request (long output)
    SELECT SNOWFLAKE.CORTEX.AI_COMPLETE(
      'mistral-large',
      'Summarize this: ' || document_text
    );
    -- Average output: 300 tokens
    
    -- AFTER: Specific constraint (short output)
    SELECT SNOWFLAKE.CORTEX.AI_COMPLETE(
      'mistral-large',
      'Summarize in exactly 3 bullet points: ' || document_text
    );
    -- Average output: 50 tokens

    Result: 80-85% reduction in output tokens.


    Optimization 4: Batch Processing

    sql

    -- Process all at once (low overhead)
    CREATE TASK process_batch_daily
    WAREHOUSE = compute_wh
    SCHEDULE = 'USING CRON 0 2 * * * UTC'
    AS
    SELECT SNOWFLAKE.CORTEX.AI_SENTIMENT(text)
    FROM data_queue
    WHERE processed = false;

    Result: 15-20% reduction in compute overhead.


    Optimization 5: Input Data Cleaning

    sql

    -- Clean data before processing
    CREATE FUNCTION clean_text(raw_text VARCHAR)
    RETURNS VARCHAR
    AS
    $$
      SELECT REGEXP_REPLACE(
        REGEXP_REPLACE(raw_text, '(\[.*?\])', ''),  -- Remove metadata
        '\n\n+', ' '  -- Collapse newlines
      )
    $$;
    
    -- Process clean data only
    SELECT SNOWFLAKE.CORTEX.AI_SENTIMENT(clean_text(messy_input))
    FROM raw_data;

    Result: 30-50% reduction in input tokens.


    Key Takeaways

    1. Cortex charges per token, not per creditUnderstanding token consumption is critical
    2. Model selection is the biggest cost lever – 10-40x cost variation possible
    3. AISQL functions are affordable – Most use cases cost $10-100/month
    4. Cortex Search is expensive – Only use if semantic search is core business need
    5. Monitoring is essential – Use CORTEX_FUNCTIONS_USAGE_HISTORY to track spend
    6. Optimization opportunities exist – Caching, batching, model selection dramatically reduce costs
    7. Not all tasks need Cortex – Use SQL/regex for deterministic operations
    8. Cortex is 10-40x cheaper than alternatives – Exceptional ROI compared to third-party APIs

    External References (Official Snowflake Docs)


    Next Steps

    For developers starting with Cortex:

    1. Run a small pilot with 1% of target data
    2. Test multiple models to find optimal cost/quality balance
    3. Establish baseline usage metrics using CORTEX_FUNCTIONS_USAGE_HISTORY
    4. Implement caching for repeated operations
    5. Set up daily cost monitoring before scaling to production

    Disclaimer: Pricing current as of January 2026. Rates subject to change. Always verify with official Snowflake documentation for most current pricing.