Tag: dbt

Orchestrating Snowflake dbt Projects with Airflow — End-to-End Pipeline Guide
How I Wired Snowflake’s Native dbt Projects to Airflow — And Finally Got True End-to-End Orchestration

I’ll be honest with you — for a long time I was running dbt the way most people run it. dbt Core installed on a server, profiles.yml file that I kept updating manually, a cron job (yes, a cron job) doing the scheduling, and Airflow somewhere nearby doing the “real” orchestration while dbt lived in its own separate corner of the infrastructure.

It worked. It was fine. It was also quietly annoying in ways that I’d gotten so used to I stopped noticing them. Managing the dbt server separately. Keeping the Snowflake credentials synced in two places. Debugging failures by jumping between the Airflow UI, SSH logs on the dbt server, and Snowsight — all at once.

Then Snowflake went GA with dbt Projects in November 2025, and I spent a weekend rebuilding the whole thing. This article is what I learned.

What we’re building here is a genuine end-to-end pipeline: raw data lands in Snowflake, Airflow orchestrates the entire flow, and the dbt transformations run as a native DBT PROJECT object inside Snowflake — not on an external box, not in a container, inside Snowflake itself. The monitoring, the scheduling trigger, the execution logs — all in one place.

Let’s build it from the ground up.

First — What Exactly Is a dbt Project on Snowflake?

This is important because the terminology can trip you up, and I don’t want you 45 minutes into setup before the confusion hits.

dbt Projects on Snowflake let you use familiar Snowflake features to create, edit, test, run, and manage dbt Core projects. You can use Workspaces in Snowsight to work with dbt project files and directories and deploy a dbt project as a schema-level DBT PROJECT object.

The key word there is object. Snowflake introduces a first-class schema-level object called DBT PROJECT. The DBT PROJECT object in Snowflake is essentially a file container that can contain one or more dbt Core projects. Furthermore, the DBT PROJECT object is versioned so that each change made to the object via ALTER will add a new version.

This means your dbt project — the models, the sources YAML, the dbt_project.yml — lives inside Snowflake as a versioned, native object. Not on a VM. Not in an S3 bucket somewhere. In Snowflake itself.

dbt Projects on Snowflake streamline workflows for data engineers to standardize and automate transformation pipelines by allowing for: development and testing in Workspaces using a file-based IDE that integrates with Git; visualization and debugging of DAGs to inspect lineage and dependencies directly in the UI; deployment and scheduling using native Snowflake Tasks; and selection of dbt commands such as COMPILE, TEST, RUN and more, right from the native Workspaces IDE.

So yes — you can schedule and run it purely with Snowflake Tasks and never touch Airflow. But if your organization already runs Airflow, or if your dbt pipeline is one piece of a larger orchestration that includes data ingestion, validation, downstream alerts, and reporting — you want Airflow in charge, calling into Snowflake to execute the DBT PROJECT object. That hybrid approach is exactly what this article covers.

The Architecture We’re Building

Before I show you a single line of code, let me draw the full picture because I think this is where most blog posts let you down — they show you a piece without the whole.
```
[Source System / S3 / API]
        ↓
[Airflow DAG starts]
        ↓
  Task 1: Load raw data → Snowflake staging table (via COPY INTO or S3 stage)
        ↓
  Task 2: Run data quality checks on raw data (SQLExecuteQueryOperator)
        ↓
  Task 3: EXECUTE DBT PROJECT → runs dbt build on your native Snowflake dbt project
        ↓
  Task 4: Post-run row count validation (SQLExecuteQueryOperator)
        ↓
  Task 5: Trigger downstream alert / Slack notification / refresh BI layer
        ↓
[Pipeline complete]
```
Airflow owns the orchestration. Snowflake owns the execution of the dbt transformations. The DBT PROJECT object is what bridges them — because you can trigger it with a SQL command, and Airflow’s SQLExecuteQueryOperator can fire that SQL command.

That SQL command, by the way, is beautifully simple:
```
EXECUTE DBT PROJECT my_database.my_schema.my_dbt_project
  ARGS = 'dbt build'
  VERSION = 'LAST';
```
EXECUTE DBT PROJECT executes the specified dbt project object or the dbt project in a Snowflake workspace using the dbt command and command-line options specified. Snowflake Documentation

One SQL statement. That’s all Airflow needs to fire. Let me now show you the full setup to make that work.

Step 1: Snowflake Setup — Roles, Warehouse, and Permissions

I always start here because bad permissions cause the most confusing failures, and they surface late in the process when you’re tired and frustrated.
```
USE ROLE ACCOUNTADMIN;

-- Create a dedicated role for dbt execution
CREATE OR REPLACE ROLE dbt_executor_role;
GRANT ROLE dbt_executor_role TO ROLE SYSADMIN;

-- Create the service user Airflow will use
CREATE OR REPLACE USER airflow_svc_user
  PASSWORD = 'YourStrongPassword123!'
  DEFAULT_ROLE = dbt_executor_role
  DEFAULT_WAREHOUSE = dbt_transform_wh
  COMMENT = 'Airflow service user for dbt orchestration';

GRANT ROLE dbt_executor_role TO USER airflow_svc_user;

-- Create a dedicated warehouse for dbt runs
USE ROLE SYSADMIN;
CREATE OR REPLACE WAREHOUSE dbt_transform_wh
  WITH WAREHOUSE_SIZE = 'SMALL'
  AUTO_SUSPEND = 120
  AUTO_RESUME = TRUE
  INITIALLY_SUSPENDED = TRUE;

GRANT ALL ON WAREHOUSE dbt_transform_wh TO ROLE dbt_executor_role;

-- Grant database and schema privileges
GRANT USAGE ON DATABASE analytics_db TO ROLE dbt_executor_role;
GRANT USAGE, CREATE TABLE, CREATE VIEW ON SCHEMA analytics_db.staging TO ROLE dbt_executor_role;
GRANT USAGE, CREATE TABLE, CREATE VIEW ON SCHEMA analytics_db.marts TO ROLE dbt_executor_role;

-- Grant the ability to execute dbt project objects
GRANT EXECUTE DBT PROJECT ON SCHEMA analytics_db.transforms TO ROLE dbt_executor_role;
```
I made a mistake my first time through — I granted object-level access but forgot the schema-level EXECUTE DBT PROJECT privilege, which is separate. The error message wasn’t obvious. Save yourself that 20-minute debugging session.

Step 2: Deploy Your dbt Project as a Native Snowflake Object

This is the step that feels the most different from traditional dbt Core setup. You’re not installing dbt on a server. You’re registering your project inside Snowflake.

Option A: Via Snowsight Workspaces (recommended for first time)

Log into Snowsight, navigate to Workspaces, and connect it to your Git repository:
```
-- First, create an API integration for GitHub
CREATE OR REPLACE API INTEGRATION github_integration
  API_PROVIDER = git_https_api
  API_ALLOWED_PREFIXES = ('https://github.com/yourorg/')
  ENABLED = TRUE;

-- Create the Git repository object in Snowflake
CREATE OR REPLACE GIT REPOSITORY dbt_project_repo
  API_INTEGRATION = github_integration
  GIT_CREDENTIALS = my_github_secret
  ORIGIN = 'https://github.com/yourorg/your-dbt-project.git';
```
Option B: Deploy via SQL (great for CI/CD)
```
-- Create the DBT PROJECT object from your connected Git repo
CREATE OR REPLACE DBT PROJECT analytics_db.transforms.sales_dbt_project
  FROM GIT REPOSITORY dbt_project_repo
  REF = 'main'
  TARGET_PATH = 'models/'
  WAREHOUSE = dbt_transform_wh;
```
Install dbt dependencies:

Install dependencies by executing the dbt deps command within a Snowflake workspace, local machine, or git orchestrator to populate the dbt_packages folder for your dbt Project.
```
-- Run this once after creating the project, or include in CI/CD
EXECUTE DBT PROJECT analytics_db.transforms.sales_dbt_project
  ARGS = 'dbt deps'
  VERSION = 'LAST';
```
A heads up on this: running dbt deps to install packages requires an external access integration when executed inside Snowflake Workspaces, since the runtime needs to reach external package repositories. Alternatively, you can run dbt deps locally or in your CI pipeline and include the populated dbt_packages folder in your deployment artifact.

I found it cleaner to run dbt deps in my GitHub Actions pipeline and commit the dbt_packages folder, rather than configuring external access integrations for every environment. Your call — both approaches work.

Verify it deployed correctly:
```
-- Check your dbt project versions
SHOW DBT PROJECTS IN SCHEMA analytics_db.transforms;

-- Test execute manually before wiring Airflow
EXECUTE DBT PROJECT analytics_db.transforms.sales_dbt_project
  ARGS = 'dbt compile'
  VERSION = 'LAST';
```
If dbt compile completes without error, your project is live and ready to be called by Airflow.

Step 3: Set Up a Real dbt Project Structure

Let me show you what the actual project looks like. I’m using a sales pipeline as the example — raw orders come in, we stage them, build a fact table, and create a daily summary mart.

dbt_project.yml:
```
name: 'sales_pipeline'
version: '1.0.0'
config-version: 2

profile: 'snowflake_prod'

model-paths: ["models"]
test-paths: ["tests"]
seed-paths: ["seeds"]

models:
  sales_pipeline:
    staging:
      +schema: staging
      +materialized: view
    marts:
      +schema: marts
      +materialized: table
```
models/staging/stg_orders.sql:
```
-- Staging model: clean and type-cast raw orders
WITH raw AS (
    SELECT * FROM {{ source('raw', 'orders_raw') }}
),

cleaned AS (
    SELECT
        order_id::VARCHAR           AS order_id,
        customer_id::VARCHAR        AS customer_id,
        order_date::DATE            AS order_date,
        UPPER(TRIM(status))         AS order_status,
        amount::DECIMAL(18, 2)      AS order_amount,
        region::VARCHAR             AS region,
        CURRENT_TIMESTAMP()         AS _loaded_at
    FROM raw
    WHERE order_id IS NOT NULL
      AND order_date >= '2023-01-01'
)

SELECT * FROM cleaned
```
models/marts/fct_daily_orders.sql:
```
-- Fact table: daily order summary by region
WITH staged AS (
    SELECT * FROM {{ ref('stg_orders') }}
)

SELECT
    order_date,
    region,
    order_status,
    COUNT(DISTINCT order_id)                    AS total_orders,
    COUNT(DISTINCT customer_id)                 AS unique_customers,
    SUM(order_amount)                           AS total_revenue,
    AVG(order_amount)                           AS avg_order_value,
    SUM(CASE WHEN order_status = 'RETURNED' 
             THEN order_amount ELSE 0 END)      AS returned_amount,
    CURRENT_TIMESTAMP()                         AS _refreshed_at
FROM staged
GROUP BY order_date, region, order_status
ORDER BY order_date DESC, region
```
models/staging/sources.yml:

version: 2

sources:
- name: raw database: analytics_db schema: raw_landing tables:
  - name: orders_raw description: “Raw orders from the source system” columns:
    
    name: order_id tests:
    
    not_null
    
    unique
    
    name: customer_id tests:
    
    not_null
    
    name: order_date tests:
    
    not_null
    
    name: amount tests:
    
    not_null
models/marts/schema.yml:
```
version: 2

models:
  - name: fct_daily_orders
    description: "Daily order summary by region and status"
    columns:
      - name: order_date
        tests:
          - not_null
      - name: total_orders
        tests:
          - not_null
      - name: total_revenue
        tests:
          - not_null
```
This gives us a clean, testable project with source freshness checks and column-level tests. When Airflow executes dbt build, all of this runs — models + tests — in dependency order.

Step 4: Wire It All Together in Airflow

Now the fun part. I’m going to show you a complete Airflow DAG that:
1. Validates raw data arrived in Snowflake
2. Fires the native dbt project execution
3. Validates row counts on the output marts
4. Sends a Slack notification on success or failure
First, install the Snowflake provider if you haven’t:
```
pip install apache-airflow-providers-snowflake
```
Set up your Snowflake connection in the Airflow UI (Admin → Connections):
```
Connection ID : snowflake_analytics
Connection Type : Snowflake
Account  : yourorg.us-east-1
Login    : airflow_svc_user
Password : YourStrongPassword123!
Schema   : transforms
Database : analytics_db
Warehouse: dbt_transform_wh
Role     : dbt_executor_role
```
Now the DAG:

dags/sales_pipeline_dag.py:
```
from airflow import DAG
from airflow.providers.snowflake.operators.snowflake import SQLExecuteQueryOperator
from airflow.operators.python import PythonOperator, BranchPythonOperator
from airflow.operators.empty import EmptyOperator
from airflow.utils.dates import days_ago
from datetime import datetime, timedelta
import logging

# ── Default args ────────────────────────────────────────────────
default_args = {
    'owner': 'data-engineering',
    'depends_on_past': False,
    'retries': 1,
    'retry_delay': timedelta(minutes=5),
    'email_on_failure': True,
    'email': ['[email protected]'],
}

SNOWFLAKE_CONN = 'snowflake_analytics'

# ── SQL snippets ─────────────────────────────────────────────────
RAW_DATA_CHECK_SQL = """
SELECT COUNT(*) AS raw_row_count
FROM analytics_db.raw_landing.orders_raw
WHERE order_date = CURRENT_DATE() - 1;
"""

EXECUTE_DBT_SQL = """
EXECUTE DBT PROJECT analytics_db.transforms.sales_dbt_project
  ARGS = 'dbt build --select staging.stg_orders+ --vars "{\\"run_date\\": \\"{{ ds }}\\"}"'
  VERSION = 'LAST';
"""

MART_VALIDATION_SQL = """
SELECT
    COUNT(*) AS mart_row_count,
    MAX(order_date) AS latest_date,
    SUM(total_revenue) AS total_revenue
FROM analytics_db.marts.fct_daily_orders
WHERE order_date = CURRENT_DATE() - 1;
"""

ROW_COUNT_GUARD_SQL = """
SELECT
    CASE
        WHEN COUNT(*) = 0
        THEN 'FAIL: No rows found in mart for yesterday'
        ELSE 'PASS: ' || COUNT(*) || ' rows present'
    END AS validation_result
FROM analytics_db.marts.fct_daily_orders
WHERE order_date = CURRENT_DATE() - 1;
"""

# ── DAG definition ───────────────────────────────────────────────
with DAG(
    dag_id='sales_pipeline_end_to_end',
    default_args=default_args,
    description='End-to-end sales pipeline: raw → dbt native project → marts',
    schedule_interval='0 6 * * *',     # 6 AM UTC daily
    start_date=days_ago(1),
    catchup=False,
    tags=['snowflake', 'dbt', 'sales'],
) as dag:

    # Task 1: Check raw data arrived
    check_raw_data = SQLExecuteQueryOperator(
        task_id='check_raw_data_arrived',
        conn_id=SNOWFLAKE_CONN,
        sql=RAW_DATA_CHECK_SQL,
        handler=lambda cursor: logging.info(
            f"Raw row count: {cursor.fetchone()[0]}"
        ),
    )

    # Task 2: Execute the native dbt project on Snowflake
    run_dbt_project = SQLExecuteQueryOperator(
        task_id='execute_dbt_project_snowflake',
        conn_id=SNOWFLAKE_CONN,
        sql=EXECUTE_DBT_SQL,
        # Give dbt build enough time for large projects
        execution_timeout=timedelta(hours=2),
    )

    # Task 3: Post-run mart validation
    validate_mart_output = SQLExecuteQueryOperator(
        task_id='validate_mart_output',
        conn_id=SNOWFLAKE_CONN,
        sql=ROW_COUNT_GUARD_SQL,
        handler=lambda cursor: logging.info(
            f"Validation result: {cursor.fetchone()[0]}"
        ),
    )

    # Task 4: Run broader stats query (logged for observability)
    log_mart_stats = SQLExecuteQueryOperator(
        task_id='log_mart_statistics',
        conn_id=SNOWFLAKE_CONN,
        sql=MART_VALIDATION_SQL,
    )

    # Task 5: Success marker
    pipeline_complete = EmptyOperator(task_id='pipeline_complete')

    # ── Dependencies ─────────────────────────────────────────────
    (
        check_raw_data
        >> run_dbt_project
        >> validate_mart_output
        >> log_mart_stats
        >> pipeline_complete
    )
```
Step 5: Running Specific dbt Selectors from Airflow

One of the things I really like about this approach is that you get the full power of dbt’s selector syntax passed straight through the ARGS parameter. You don’t have to run the entire project every time.

Run only staging models:
```
EXECUTE_STAGING_ONLY = """
EXECUTE DBT PROJECT analytics_db.transforms.sales_dbt_project
  ARGS = 'dbt run --select staging.*'
  VERSION = 'LAST';
"""
```
Run a specific model and all its downstream dependencies:
```
EXECUTE_ORDERS_DOWNSTREAM = """
EXECUTE DBT PROJECT analytics_db.transforms.sales_dbt_project
  ARGS = 'dbt build --select stg_orders+'
  VERSION = 'LAST';
"""
```
Run tests only, separate from the model run:
```
RUN_DBT_TESTS = """
EXECUTE DBT PROJECT analytics_db.transforms.sales_dbt_project
  ARGS = 'dbt test --select staging.*'
  VERSION = 'LAST';
"""
```
This means you can split a single DAG into multiple tasks — one for staging, one for marts, one for tests — and get granular retry behavior in Airflow if something fails mid-pipeline. Instead of rerunning everything, Airflow retries only the failed task.

Here’s that pattern as a DAG:
```
run_staging = SQLExecuteQueryOperator(
    task_id='run_dbt_staging',
    conn_id=SNOWFLAKE_CONN,
    sql="""
        EXECUTE DBT PROJECT analytics_db.transforms.sales_dbt_project
          ARGS = 'dbt run --select staging.*'
          VERSION = 'LAST';
    """,
)

test_staging = SQLExecuteQueryOperator(
    task_id='test_dbt_staging',
    conn_id=SNOWFLAKE_CONN,
    sql="""
        EXECUTE DBT PROJECT analytics_db.transforms.sales_dbt_project
          ARGS = 'dbt test --select staging.*'
          VERSION = 'LAST';
    """,
)

run_marts = SQLExecuteQueryOperator(
    task_id='run_dbt_marts',
    conn_id=SNOWFLAKE_CONN,
    sql="""
        EXECUTE DBT PROJECT analytics_db.transforms.sales_dbt_project
          ARGS = 'dbt run --select marts.*'
          VERSION = 'LAST';
    """,
)

run_staging >> test_staging >> run_marts
```
This is how I actually run it in practice. If staging tests fail, marts never execute. If marts fail, I retry marts without re-running staging. Clean dependency management with minimal code.

Step 6: Handling New Versions of Your dbt Project

This is something I didn’t think about until I pushed a breaking change to main and my 6 AM pipeline executed the wrong version.

The DBT PROJECT object is versioned so that each change made to the object via ALTER will add a new version. The versions are named according to the pattern VERSION$<num>.

In practice, your CI/CD pipeline (GitHub Actions, etc.) should update the DBT PROJECT object after any merge to main:
```
# .github/workflows/deploy_dbt.yml
name: Deploy dbt Project to Snowflake

on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Install Snowflake CLI
        run: pip install snowflake-cli-labs

      - name: Deploy new dbt project version
        env:
          SNOWFLAKE_ACCOUNT: ${{ secrets.SNOWFLAKE_ACCOUNT }}
          SNOWFLAKE_USER: ${{ secrets.SNOWFLAKE_USER }}
          SNOWFLAKE_PASSWORD: ${{ secrets.SNOWFLAKE_PASSWORD }}
        run: |
          snow dbt deploy \
            --project-name analytics_db.transforms.sales_dbt_project \
            --from-git \
            --ref main
```
And in your Airflow SQL, VERSION = 'LAST' always picks up the most recently deployed version automatically. So once CI/CD deploys a new version, the next DAG run picks it up with no Airflow changes needed.

Step 7: Monitoring — What to Watch and Where

Before this setup, I was watching three screens at once when something went wrong. Now it’s mostly one.

In Snowsight:
```
-- Check recent dbt project execution history
SELECT
    query_id,
    query_text,
    execution_status,
    start_time,
    end_time,
    DATEDIFF('second', start_time, end_time) AS duration_seconds,
    error_message
FROM TABLE(
    INFORMATION_SCHEMA.QUERY_HISTORY(
        END_TIME_RANGE_START => DATEADD('day', -1, CURRENT_TIMESTAMP()),
        RESULT_LIMIT => 50
    )
)
WHERE query_text ILIKE '%EXECUTE DBT PROJECT%'
ORDER BY start_time DESC;
```
Row count drift detection (add this as an Airflow task):
```
-- Compare today's mart row count to yesterday's
-- Flag if it drops more than 20%
WITH today AS (
    SELECT COUNT(*) AS cnt
    FROM analytics_db.marts.fct_daily_orders
    WHERE order_date = CURRENT_DATE() - 1
),
yesterday AS (
    SELECT COUNT(*) AS cnt
    FROM analytics_db.marts.fct_daily_orders
    WHERE order_date = CURRENT_DATE() - 2
)
SELECT
    today.cnt                                               AS today_rows,
    yesterday.cnt                                           AS yesterday_rows,
    ROUND((today.cnt - yesterday.cnt) / NULLIF(yesterday.cnt, 0) * 100, 2) AS pct_change,
    CASE
        WHEN today.cnt < yesterday.cnt * 0.80
        THEN 'ALERT: Row count dropped over 20%'
        ELSE 'OK'
    END AS status
FROM today, yesterday;
```
I added this query as a SQLExecuteQueryOperator task right after the mart validation step. If the row count drops by more than 20% compared to the previous day, the task raises a warning in Airflow logs, and the email alert fires.

Not every data quality problem shows up as a dbt test failure. Sometimes the data just quietly shrinks because an upstream feed stopped delivering. This catches that.

What This Setup Actually Changed for Me

I want to be real about this because I think the “benefits” sections in most blog posts are too abstract.

Before: My pipeline had six moving parts. Airflow DAG on one server. dbt installed on a separate instance. profiles.yml with credentials that needed updating every time we rotated passwords. Separate monitoring in CloudWatch for the dbt server. Debugging a failure meant SSH → dbt server → find the log file → cross-reference with Airflow logs.

After: The pipeline has three moving parts — Airflow, Snowflake, and GitHub. The dbt credentials are managed by Airflow’s Snowflake connection, which I was already maintaining. Debugging a failure means clicking into the Airflow task logs (which capture the SQL response from Snowflake) and if I need more detail, running the QUERY_HISTORY query above in Snowsight.

Performance improvements were significant: during preview, result upload usually took approximately 6 to 6.5 minutes. Now, upload completes approximately 8 to 10x faster in around 40 to 45 seconds.

The startup time improvement alone was worth it for me. My morning pipeline used to take 28-32 minutes. It now consistently runs in 18-22 minutes. That’s not from faster models — it’s from the reduction in environment spin-up overhead.

A Few Gotchas I Hit Along the Way

1. The EXECUTE DBT PROJECT command is synchronous by default. Airflow will wait for it to complete before marking the task done. For large projects this is fine — you want that behavior. Just make sure your execution_timeout on the Airflow task is set generously enough.

2. Cross-project references don’t work the way you might expect. Cross-project dependencies must be copied into the root of the main project — Snowflake doesn’t support references to external file paths within the DBT PROJECT object. If you have multiple dbt projects, plan your consolidation before deploying.

3. The VERSION = 'LAST' behavior. This always runs the most recently deployed version. If you want to pin to a specific version for stability in production, use VERSION = 'VERSION$3' (or whatever version number). I run LAST in dev and a pinned version in prod, deployed via CI/CD.

4. Warehouse auto-resume and the first task. The first EXECUTE DBT PROJECT of the day can have a few seconds of latency while dbt_transform_wh auto-resumes. I added a lightweight warm-up query as the very first task in my DAG so the warehouse is already running by the time dbt build kicks off:
```
warm_up_warehouse = SQLExecuteQueryOperator(
    task_id='warm_up_warehouse',
    conn_id=SNOWFLAKE_CONN,
    sql="SELECT CURRENT_TIMESTAMP();",
)

warm_up_warehouse >> check_raw_data >> run_dbt_project >> ...
```
Costs almost nothing. Saves 5-10 seconds of variability at the start of every run.

Why I Think This Is the Right Direction

I started exploring this because nobody told me to. My team’s existing setup worked. A reasonable person would have left it alone.

But the more I looked at this setup, the more I kept thinking about the overhead we carry when tools don’t talk to each other natively. Every boundary between systems is a place where credentials leak, latency is added, and debugging gets harder. The native dbt project in Snowflake closes one of those boundaries. Airflow still owns orchestration — which is where it belongs — but the transformation execution lives where the data lives.

For the growing number of organizations that have standardized on Snowflake, the native integration offers something genuinely compelling: one fewer system to run, one fewer vendor to manage, and one fewer boundary between your data and the logic that transforms it.

That sentence landed for me when I read it. That’s exactly what this is.

If you’ve been running dbt Core on a server and Airflow alongside it and you’ve been tolerating that overhead long enough that you’ve stopped noticing it — try this weekend rebuild. You might be surprised how much lighter the pipeline feels on the other side.

And if you do try it and hit something weird, drop it in the comments. I’m still learning this myself.
April 7, 2026

Snowflake Native dbt Integration: Complete 2025 Guide

Run dbt Core Directly in Snowflake Without Infrastructure

Snowflake native dbt integration announced at Summit 2025 eliminates the need for separate containers or VMs to run dbt Core. Data teams can now execute dbt transformations directly within Snowflake, with built-in lineage tracking, logging, and job scheduling through Snowsight. This breakthrough simplifies data pipeline architecture and reduces operational overhead significantly.

For years, running dbt meant managing separate infrastructure—deploying containers, configuring CI/CD pipelines, and maintaining compute resources outside your data warehouse. The Snowflake native dbt integration changes everything by bringing dbt Core execution inside Snowflake’s secure environment.

What Is Snowflake Native dbt Integration?

Snowflake native dbt integration allows data teams to run dbt Core transformations directly within Snowflake without external orchestration tools. The integration provides a managed environment where dbt projects execute using Snowflake’s compute resources, with full visibility through Snowsight.

Key Benefits

The native integration delivers:

Zero infrastructure management – No containers, VMs, or separate compute
Built-in lineage tracking – Automatic data flow visualization
Native job scheduling – Schedule dbt runs using Snowflake Tasks
Integrated logging – Debug pipelines directly in Snowsight
No licensing costs – dbt Core runs free within Snowflake

Organizations using Snowflake Dynamic Tables can now complement those automated refreshes with sophisticated dbt transformations, creating comprehensive data pipeline solutions entirely within the Snowflake ecosystem.

How Native dbt Integration Works

Execution Architecture

When you deploy a dbt project to Snowflake native dbt integration, the platform:

Stores project files in Snowflake’s internal stage
Compiles dbt models using Snowflake’s compute
Executes SQL transformations against your data
Captures lineage automatically for all dependencies
Logs results to Snowsight for debugging

Similar to how real-time data pipeline architectures require proper orchestration, dbt projects benefit from Snowflake’s native task scheduling and dependency management.

-- Create a dbt job in Snowflake
CREATE OR REPLACE TASK run_dbt_models
  WAREHOUSE = transform_wh
  SCHEDULE = 'USING CRON 0 2 * * * America/Los_Angeles'
AS
  CALL DBT.RUN_DBT_PROJECT('my_analytics_project');

-- Enable the task
ALTER TASK run_dbt_models RESUME;

Setting Up Native dbt Integration

Prerequisites

Before deploying dbt projects natively:

Snowflake account with ACCOUNTADMIN or appropriate role
Existing dbt project with proper structure
Git repository containing dbt code (optional but recommended)

A flowchart showing dbt Project Files leading to Snowflake Stage, then dbt Core Execution, Data Transformation, and finally Output Tables, with SQL noted below dbt Core Execution.

Step-by-Step Implementation

1: Prepare Your dbt Project

Ensure your project follows standard dbt structure:

my_dbt_project/
├── models/
├── macros/
├── tests/
├── dbt_project.yml
└── profiles.yml

2: Upload to Snowflake

-- Create stage for dbt files
CREATE STAGE dbt_projects
  DIRECTORY = (ENABLE = true);

-- Upload project files
PUT file://my_dbt_project/* @dbt_projects/my_project/;

3: Configure Execution

-- Set up dbt execution environment
CREATE OR REPLACE PROCEDURE run_my_dbt()
  RETURNS STRING
  LANGUAGE PYTHON
  RUNTIME_VERSION = 3.8
  PACKAGES = ('dbt-core', 'dbt-snowflake')
  HANDLER = 'run_dbt'
AS
$$
def run_dbt(session):
    import dbt.main
    results = dbt.main.run(['run'])
    return f"dbt run completed with {results} models"
$$;

4: Schedule with Tasks

Link dbt execution to data quality validation processes by scheduling regular runs:

CREATE TASK daily_dbt_refresh
  WAREHOUSE = analytics_wh
  SCHEDULE = 'USING CRON 0 3 * * * UTC'
AS
  CALL run_my_dbt();

Lineage and Observability

Built-in Lineage Tracking

Snowflake native dbt integration automatically captures data lineage across:

Source tables referenced in models
Intermediate transformation layers
Final output tables and views
Test dependencies and validations

Access lineage through Snowsight’s graphical interface, similar to monitoring API integration workflows in modern data architectures.

Debugging Capabilities

The platform provides:

Real-time execution logs showing compilation and run details
Error stack traces pointing to specific model failures
Performance metrics for each transformation step
Query history for all generated SQL

Best Practices for Native dbt

Optimize Warehouse Sizing

Match warehouse sizes to transformation complexity:

-- Small warehouse for lightweight models
CREATE WAREHOUSE dbt_small_wh
  WAREHOUSE_SIZE = 'SMALL'
  AUTO_SUSPEND = 60
  AUTO_RESUME = TRUE;

-- Large warehouse for heavy aggregations
CREATE WAREHOUSE dbt_large_wh
  WAREHOUSE_SIZE = 'LARGE'
  AUTO_SUSPEND = 60;

Implement Incremental Strategies

Leverage dbt’s incremental models for efficiency:

-- models/incremental_sales.sql
{{ config(
    materialized='incremental',
    unique_key='sale_id'
) }}

SELECT *
FROM {{ source('raw', 'sales') }}
{% if is_incremental() %}
WHERE sale_date > (SELECT MAX(sale_date) FROM {{ this }})
{% endif %}

Use Snowflake-Specific Features

Take advantage of native capabilities when using machine learning integrations or advanced analytics:

-- Use Snowflake clustering for large tables
{{ config(
    materialized='table',
    cluster_by=['sale_date', 'region']
) }}

Migration from External dbt

Moving from dbt Cloud

Organizations migrating from dbt Cloud to Snowflake native dbt integration should:

Export existing projects from dbt Cloud repositories
Review connection profiles and update for Snowflake native execution
Migrate schedules to Snowflake Tasks
Update CI/CD pipelines to trigger native execution
Train teams on Snowsight-based monitoring

Moving from Self-Hosted dbt

Teams running dbt in containers or VMs benefit from:

Eliminated infrastructure costs (no more EC2 instances or containers)
Reduced maintenance burden (Snowflake manages runtime)
Improved security (execution stays within Snowflake perimeter)
Better integration with Snowflake features

Cost Considerations

Compute Consumption

Snowflake native dbt integration uses standard warehouse compute:

Charged per second of active execution
Auto-suspend reduces idle costs
Share warehouses across multiple jobs for efficiency

Comparison with External Solutions

Aspect	External dbt	Native dbt Integration
Infrastructure	EC2/VM costs	Only Snowflake compute
Maintenance	Manual updates	Managed by Snowflake
Licensing	dbt Cloud fees	Free (dbt Core)
Integration	External APIs	Native Snowflake

Organizations using automation strategies across their data stack can consolidate tools and reduce total cost of ownership.

Real-World Use Cases

Use Case 1: Financial Services Reporting

A fintech company moved 200+ dbt models from AWS containers to Snowflake native dbt integration, achieving:

60% reduction in infrastructure costs
40% faster transformation execution
Zero downtime migrations using blue-green deployment

Use Case 2: E-commerce Analytics

An online retailer consolidated their data pipeline by combining native dbt with Dynamic Tables:

dbt handles complex business logic transformations
Dynamic Tables maintain real-time aggregations
Both execute entirely within Snowflake

Use Case 3: Healthcare Data Warehousing

A healthcare provider simplified compliance by keeping all transformations inside Snowflake’s secure perimeter:

HIPAA compliance maintained without data egress
Audit logs automatically captured
PHI never leaves Snowflake environment

Advanced Features

Git Integration

Connect dbt projects directly to repositories:

CREATE GIT REPOSITORY dbt_repo
  ORIGIN = 'https://github.com/myorg/dbt-project.git'
  API_INTEGRATION = github_integration;

-- Run dbt from specific branch
CALL run_dbt_from_git('dbt_repo', 'production');

Testing and Validation

Native integration supports full dbt testing:

Schema tests validate data structure
Data tests check business rules
Custom tests enforce specific requirements

Multi-Environment Support

Manage dev, staging, and production through Snowflake databases:

sql

-- Development environment
USE DATABASE dev_analytics;
CALL run_dbt('dev_project');

-- Production environment
USE DATABASE prod_analytics;
CALL run_dbt('prod_project');

Troubleshooting Common Issues

Issue 1: Slow Model Compilation

Solution: Pre-compile dbt projects and cache results:

sql

-- Cache compiled SQL for faster execution
ALTER TASK dbt_refresh SET
  SUSPEND_TASK_AFTER_NUM_FAILURES = 3;

Issue 2: Dependency Conflicts

Solution: Use Snowflake’s Python environment isolation:

sql

-- Specify exact package versions
PACKAGES = ('dbt-core==1.7.0', 'dbt-snowflake==1.7.0')

Future Roadmap

Snowflake plans to enhance native dbt integration with:

Visual dbt model builder for low-code transformations
Automatic optimization suggestions using AI
Enhanced collaboration features for team workflows
Deeper integration with Snowflake’s AI capabilities

Organizations exploring autonomous AI agents in other platforms will find similar intelligence coming to dbt optimization.

Conclusion: Simplified Data Transformation

Snowflake native dbt integration represents a significant evolution in data transformation architecture. By eliminating external infrastructure and bringing dbt Core inside Snowflake, data teams achieve simplified operations, reduced costs, and enhanced security.

The integration is production-ready today, with thousands of organizations already migrating their dbt workloads. Teams should evaluate their current dbt architecture and plan migrations to take advantage of this native capability.

Start with non-critical projects, validate performance, and progressively move production workloads. The combination of zero infrastructure overhead, built-in observability, and seamless Snowflake integration makes native dbt integration the future of transformation pipelines.

🔗 External Resources

October 17, 2025

Structuring dbt Projects in Snowflake: The Definitive Guide
If you’ve ever inherited a dbt project, you know there are two kinds: the clean, logical, and easy-to-navigate project, and the other kind—a tangled mess of models that makes you question every life choice that led you to that moment. The difference between the two isn’t talent; it’s structure. For high-performing data teams, a well-defined structure for dbt projects in Snowflake isn’t just a nice-to-have, it’s the very foundation of a scalable, maintainable, and trustworthy analytics workflow.

While dbt and Snowflake are a technical match made in heaven, simply putting them together doesn’t guarantee success. Without a clear and consistent project structure, even the most powerful tools can lead to chaos. Dependencies become circular, model names become ambiguous, and new team members spend weeks just trying to understand the data flow.

This guide provides a battle-tested blueprint for structuring dbt projects in Snowflake. We’ll move beyond the basics and dive into a scalable, multi-layered framework that will save you and your team countless hours of rework and debugging.

Why dbt and Snowflake Are a Perfect Match

Before we dive into project structure, it’s crucial to understand why this combination has become the gold standard for the modern data stack. Their synergy comes from a shared philosophy of decoupling, scalability, and performance.
- Snowflake’s Decoupled Architecture: Its separation of storage and compute is revolutionary. This means you can run massive dbt transformations using a dedicated, powerful virtual warehouse without slowing down your BI tools.
- dbt’s Transformation Power: dbt focuses on the “T” in ELT—transformation. It allows you to build, test, and document your data models using simple SQL, which it then compiles and runs directly inside Snowflake’s powerful engine.
- Cost and Performance Synergy: Running dbt models in Snowflake is incredibly efficient. You can spin up a warehouse for a dbt run and spin it down the second it’s finished, meaning you only pay for the exact compute you use.
- Zero-Copy Cloning for Development: Instantly create a zero-copy clone of your entire production database for development. This allows you to test your dbt project against production-scale data without incurring storage costs or impacting the production environment.
In short, Snowflake provides the powerful, elastic engine, while dbt provides the organized, version-controlled, and testable framework to harness that engine.

The Layered Approach: From Raw Data to Actionable Insights

A scalable dbt project is like a well-organized factory. Raw materials come in one end, go through a series of refined production stages, and emerge as a finished product. We achieve this by structuring our models into distinct layers, each with a specific job.

Our structure will follow this flow: Sources -> Staging -> Intermediate -> Marts.

Layer 1: Declaring Your Sources (The Contract with Raw Data)

Before you write a single line of transformation SQL, you must tell dbt where your raw data lives in Snowflake. This is done in a .yml file. Think of this file as a formal contract that declares your raw tables, allows you to add data quality tests, and serves as a foundation for your data lineage graph.

Example: models/staging/sources.yml

Let’s assume we have a RAW_DATA database in Snowflake with schemas from a jaffle_shop and stripe.

YAML
```
version: 2

sources:
  - name: jaffle_shop
    database: raw_data 
    schema: jaffle_shop
    description: "Raw data from the primary application database."
    tables:
      - name: customers
        columns:
          - name: id
            tests:
              - unique
              - not_null
      - name: orders
        loaded_at_field: _etl_loaded_at
        freshness:
          warn_after: {count: 12, period: hour}

  - name: stripe
    database: raw_data
    schema: stripe
    tables:
      - name: payment
        columns:
          - name: orderid
            tests:
              - relationships:
                  to: source('jaffle_shop', 'orders')
                  field: id
```
Layer 2: Staging Models (Clean and Standardize)

Staging models are the first line of transformation. They should have a 1:1 relationship with your source tables. The goal here is strict and simple:
- DO: Rename columns, cast data types, and perform very light cleaning.
- DO NOT: Join to other tables.
This creates a clean, standardized version of each source table, forming a reliable foundation for the rest of your project.

Example: models/staging/stg_customers.sql

SQL
```
-- models/staging/stg_customers.sql
with source as (
    select * from {{ source('jaffle_shop', 'customers') }}
),

renamed as (
    select
        id as customer_id,
        first_name,
        last_name
    from source
)

select * from renamed
```
Layer 3: Intermediate Models (Build, Join, and Aggregate)

This is where the real business logic begins. Intermediate models are the “workhorses” of your dbt project. They take the clean data from your staging models and start combining them.
- DO: Join different staging models together.
- DO: Perform complex calculations, aggregations, and business-specific logic.
- Materialize them as tables if they are slow to run or used by many downstream models.
These models are not typically exposed to business users. They are building blocks for your final data marts.

Example: models/intermediate/int_orders_with_payments.sql

SQL
```
-- models/intermediate/int_orders_with_payments.sql
with orders as (
    select * from {{ ref('stg_orders') }}
),

payments as (
    select * from {{ ref('stg_payments') }}
),

order_payments as (
    select
        order_id,
        sum(case when payment_status = 'success' then amount else 0 end) as total_amount
    from payments
    group by 1
),

final as (
    select
        orders.order_id,
        orders.customer_id,
        orders.order_date,
        coalesce(order_payments.total_amount, 0) as amount
    from orders
    left join order_payments 
      on orders.order_id = order_payments.order_id
)

select * from final
```
Layer 4: Data Marts (Ready for Analysis)

Finally, we arrive at the data marts. These are the polished, final models that power your dashboards, reports, and analytics. They should be clean, easy to understand, and built for a specific business purpose (e.g., finance, marketing, product).
- DO: Join intermediate models.
- DO: Have clear, business-friendly column names.
- DO NOT: Contain complex, nested logic. All the heavy lifting should have been done in the intermediate layer.
These models are the “products” of your data factory, ready for consumption by BI tools like Tableau, Looker, or Power BI.

Example: models/marts/fct_customer_orders.sql

SQL
```
-- models/marts/fct_customer_orders.sql
with customers as (
    select * from {{ ref('stg_customers') }}
),

orders as (
    select * from {{ ref('int_orders_with_payments') }}
),

customer_orders as (
    select
        customers.customer_id,
        min(orders.order_date) as first_order_date,
        max(orders.order_date) as most_recent_order_date,
        count(orders.order_id) as number_of_orders,
        sum(orders.amount) as lifetime_value
    from customers
    left join orders 
      on customers.customer_id = orders.customer_id
    group by 1
)

select * from customer_orders
```
Conclusion: Structure is Freedom

By adopting a layered approach to your dbt projects in Snowflake, you move from a chaotic, hard-to-maintain process to a scalable, modular, and efficient analytics factory. This structure gives you:
- Maintainability: When logic needs to change, you know exactly which model to edit.
- Scalability: Onboarding new data sources or team members becomes a clear, repeatable process.
- Trust: With testing at every layer, you build confidence in your data and empower the entire organization to make better, faster decisions.
This framework isn’t just about writing cleaner code—it’s about building a foundation for a mature and reliable data culture.
September 25, 2025

Tag: dbt

Orchestrating Snowflake dbt Projects with Airflow — End-to-End Pipeline Guide

First — What Exactly Is a dbt Project on Snowflake?

The Architecture We’re Building

Step 1: Snowflake Setup — Roles, Warehouse, and Permissions

Step 2: Deploy Your dbt Project as a Native Snowflake Object

Step 3: Set Up a Real dbt Project Structure

Step 4: Wire It All Together in Airflow

Step 5: Running Specific dbt Selectors from Airflow

Step 6: Handling New Versions of Your dbt Project

Step 7: Monitoring — What to Watch and Where

What This Setup Actually Changed for Me

A Few Gotchas I Hit Along the Way

Why I Think This Is the Right Direction

Snowflake Native dbt Integration: Complete 2025 Guide

Run dbt Core Directly in Snowflake Without Infrastructure

What Is Snowflake Native dbt Integration?

Key Benefits

How Native dbt Integration Works

Execution Architecture

Setting Up Native dbt Integration

Prerequisites

Step-by-Step Implementation

Lineage and Observability

Built-in Lineage Tracking

Debugging Capabilities

Best Practices for Native dbt

Optimize Warehouse Sizing

Implement Incremental Strategies

Use Snowflake-Specific Features

Migration from External dbt

Moving from dbt Cloud

Moving from Self-Hosted dbt

Cost Considerations

Compute Consumption

Comparison with External Solutions

Real-World Use Cases

Use Case 1: Financial Services Reporting

Use Case 2: E-commerce Analytics

Use Case 3: Healthcare Data Warehousing

Advanced Features

Git Integration

Testing and Validation

Multi-Environment Support

Troubleshooting Common Issues

Issue 1: Slow Model Compilation

Issue 2: Dependency Conflicts

Future Roadmap

Conclusion: Simplified Data Transformation

🔗 External Resources

Structuring dbt Projects in Snowflake: The Definitive Guide

Why dbt and Snowflake Are a Perfect Match

The Layered Approach: From Raw Data to Actionable Insights

Layer 1: Declaring Your Sources (The Contract with Raw Data)

Example: models/staging/sources.yml

Layer 2: Staging Models (Clean and Standardize)

Example: models/staging/stg_customers.sql

Layer 3: Intermediate Models (Build, Join, and Aggregate)

Example: models/intermediate/int_orders_with_payments.sql

Layer 4: Data Marts (Ready for Analysis)

Example: models/marts/fct_customer_orders.sql

Conclusion: Structure is Freedom

Example: `models/staging/sources.yml`

Example: `models/staging/stg_customers.sql`

Example: `models/intermediate/int_orders_with_payments.sql`

Example: `models/marts/fct_customer_orders.sql`