Author: Sainath Reddy

Snowflake CoCo Desktop — What It Is, How It Works, and Whether It’s Worth It

Snowflake just announced a lot at Summit 2026. Most of it was the usual conference noise. CoCo Desktop isn’t.

I’ve been following Cortex Code — now officially rebranded as CoCo — since it first shipped in Snowsight. The Summit 2026 announcement on June 2 changed the scope significantly. A native desktop IDE, Cloud Agents that run async without keeping your machine on, a Slackbot, mobile app, and integrations with VS Code, Excel, and Claude Code. That’s not a feature update. That’s a platform play.

Here’s my honest take on what CoCo Desktop actually is, how it compares to what you’re probably already using, and whether data engineers should care.

TL;DR

→ Snowflake CoCo is the official rebrand of Cortex Code — same product, bigger vision, launched at Summit 2026 on June 2
→ CoCo Desktop is a native IDE that reads your Snowflake schemas, RBAC policies, and lineage before generating any code
→ It scored 72.1% on dbt’s ADE-Bench vs 65.1% for Claude Code — but benchmarks and production are different things
→ New at Summit: Cloud Agents run tasks async in Snowflake’s cloud, Automations handle recurring workflows, Skill Catalog shares reusable flows
→ It integrates with VS Code, Slack, Excel, and Claude Code — so you don’t have to abandon your existing tools
→ Worth evaluating if your team lives in Snowflake — not worth migrating to if you’re happy with Claude Code or Cursor

What CoCo Actually Is (And What Changed at Summit 2026)

CoCo (formerly Cortex Code) is Snowflake’s data-native AI coding agent. The key word is data-native. Unlike general-purpose coding assistants, CoCo reads your live Snowflake environment — schemas, RBAC policies, lineage — before generating anything. It doesn’t generate SQL and hope it matches your tables. It knows your tables.

That’s been true since Cortex Code. What changed at Summit 2026:

CoCo Desktop — a native desktop IDE, not just a Snowsight panel. Full agentic development, local environment, MCP integrations.

Cloud Agents — launch tasks that run async in Snowflake’s cloud. Your machine doesn’t need to stay on. This is the biggest shift.

Automations — recurring, event-driven workflows. CoCo monitors, validates, and triggers without manual intervention.

Skill Catalog — teams can share and reuse proven CoCo workflows. Institutional knowledge, codified.

The rebranding from Cortex Code to CoCo also signals something: Snowflake is positioning this as a platform, not a feature. The Cortex branding was burying it inside a product umbrella. CoCo gets its own identity, its own downloads page, and its own development roadmap.

The Data-Native Context Advantage

This is where CoCo genuinely differentiates from Claude Code, Cursor, or GitHub Copilot.

Every general-purpose coding assistant has the same fundamental problem with data engineering work: it doesn’t know your data. You ask it to write a dbt model and it writes syntactically valid SQL against tables that may or may not exist in your warehouse. You have to paste schema context, explain relationships, describe RBAC constraints. It’s friction.

CoCo eliminates that friction for Snowflake environments. Before generating anything, it reads your live schema, understands which tables your role can access, and traces lineage to understand dependencies. The output is grounded in your actual environment.

Here’s a real example of the difference — a natural language prompt to CoCo and what it generates knowing your schema:

CoCo Desktop — Schema-Grounded dbt Model Generation

-- Example: What you say to CoCo Desktop
"Create an incremental dbt model that calculates 
7-day rolling revenue by customer segment, 
partitioned by signup_date"

-- What CoCo generates (reads your actual schema first):
{{ config(
    materialized='incremental',
    unique_key='customer_segment || date_trunc(day, event_date)',
    on_schema_change='sync_all_columns'
) }}

WITH revenue_base AS (
    SELECT
        c.segment                          AS customer_segment,
        DATE_TRUNC('DAY', o.created_at)    AS event_date,
        SUM(o.revenue_usd)                 AS daily_revenue
    FROM {{ ref('orders') }} o
    JOIN {{ ref('customers') }} c ON o.customer_id = c.id
    {% if is_incremental() %}
    WHERE o.created_at >= (
        SELECT DATEADD(DAY, -8, MAX(event_date)) FROM {{ this }}
    )
    {% endif %}
    GROUP BY 1, 2
)
SELECT
    customer_segment,
    event_date,
    daily_revenue,
    AVG(daily_revenue) OVER (
        PARTITION BY customer_segment
        ORDER BY event_date
        ROWS BETWEEN 6 PRECEDING AND CURRENT ROW
    ) AS rolling_7d_revenue
FROM revenue_base

The output references your actual table names, your actual columns, and respects your actual partitioning strategy — because CoCo read your schema before writing a single line.

Cloud Agents — Async Scheduled Tasks

The second genuinely new capability is Cloud Agents. Here’s what that looks like in practice:

-- Cloud Agent example: schedule a validation job
-- that runs without keeping your laptop open

-- In CoCo Desktop → New Cloud Agent:
{
  "name": "daily_revenue_validation",
  "trigger": "schedule",
  "cron": "0 6 * * *",
  "task": "Run data quality checks on orders table,
           flag anomalies > 2 std deviations,
           post summary to #data-alerts Slack channel",
  "context": ["orders", "customers", "revenue_daily"],
  "on_failure": "notify_slack"
}

-- CoCo generates, schedules, and monitors this
-- entirely within Snowflake's governed environment

This is the shift from coding assistant to autonomous agent. CoCo doesn’t just help you write the job — it runs the job, in Snowflake’s governed environment, on a schedule, and reports back.

The Benchmark Reality Check

Snowflake claims CoCo scored 72.1% on dbt’s ADE-Bench versus 65.1% for Claude Code. That’s a real benchmark on real analytics engineering tasks — 145 queries, statistically significant.

I want to be honest about what this means and doesn’t mean.

It means CoCo is genuinely better at Snowflake-specific SQL and dbt model generation than Claude Code in a controlled evaluation. That’s not surprising — CoCo has live schema context and was purpose-built for this use case.

It doesn’t mean CoCo is better for all the work you actually do. ADE-Bench measures analytics engineering tasks specifically. It doesn’t measure debugging Python pipeline errors, writing Airflow DAGs, reviewing infrastructure-as-code, or any of the other things Claude Code or Cursor handle in a typical data engineering workday.

If 80% of your coding work is Snowflake SQL and dbt models, CoCo’s benchmark advantage is real and production-relevant. If you’re a generalist data engineer working across multiple systems, that 7-point advantage on analytics SQL is a smaller part of your actual workflow.

Where CoCo Desktop Has Limits

No offline mode. CoCo Desktop requires a Snowflake account connection. If you’re working without internet access or in an environment where outbound connections are restricted, it doesn’t work.

Snowflake-only context. CoCo understands your Snowflake environment deeply. It doesn’t understand your Postgres database, your Kafka topics, or your Airflow DAG structure unless you give it that context manually — at which point you’ve lost the data-native advantage.

Token-based pricing. Cloud Agents and Automations consume Snowflake credits. For high-frequency automation workflows, the cost model needs evaluation before you commit. This is a brand new product — pricing behaviour at scale is unknown.

MCP ecosystem is smaller than Claude Code’s. CoCo supports GitHub, Jira, Google Workspace via MCP. Claude Code’s MCP ecosystem is broader. If your workflow relies on specific MCP integrations, check the current list before assuming coverage.

The Comparison You Actually Need

Feature	CoCo Desktop	Claude Code / Cursor
Data context	Reads live Snowflake schema, RBAC, lineage automatically	No native warehouse context — you provide manually
SQL generation	72.1% ADE-Bench — purpose-built for analytics SQL	65.1% ADE-Bench — strong general coding
dbt support	Native — reads dbt project structure and models	Good — but no automatic schema grounding
Pipeline authoring	Snowflake-native — Snowpark, Streams, Tasks	General Python — works but no Snowflake operators
Cloud Agents	Run tasks async in Snowflake cloud	Local execution only
MCP integrations	GitHub, Jira, Google Workspace	Broader third-party connector ecosystem
Slack / mobile	Slackbot and mobile app coming soon	No native Slack or mobile interface
Governance	RBAC-aware — won’t violate access policies	No governance layer — manual enforcement
Best for	Teams fully on Snowflake	General data engineering, polyglot stacks

How I’d Actually Use This

I wouldn’t replace Claude Code with CoCo. I’d use them for different things.

CoCo Desktop for: writing dbt models, generating Snowpark pipelines, setting up Cloud Agents for recurring validation jobs, anything where Snowflake schema context is the difference between useful output and generic SQL.

Claude Code for: debugging Python pipeline errors, writing Airflow DAGs, reviewing infrastructure code, cross-system work, anything outside the Snowflake context boundary.

The Skill Catalog is the feature I’m most interested in practically. Codifying proven CoCo workflows — a data quality check pattern, a standard incremental model template, a Snowflake Stream processing pattern — and sharing them across the team is where the real leverage is. That’s institutional knowledge made reusable. I wrote about a similar pattern in Delta Lake vs Iceberg — the tools that win long-term are the ones that compound team knowledge, not just individual productivity.

When to Evaluate CoCo Desktop

Your team is primarily on Snowflake. If 70%+ of your data work is in Snowflake, CoCo’s context advantage is real and compounding. The time saved not pasting schema context into Claude Code adds up fast.

You need governed AI development. CoCo’s RBAC awareness means it won’t generate queries that violate access policies. For compliance-heavy environments, that’s not a nice-to-have.

You want async agentic workflows. Cloud Agents are genuinely new. If you want to describe a monitoring job in natural language and have it run on a schedule without babysitting it, CoCo is currently the only tool that does this inside a governed Snowflake environment.

When to Stick With What You Have

You’re on a polyglot stack. Snowflake is one of several systems. CoCo’s advantage disappears outside the Snowflake context boundary.

You’re happy with Claude Code or Cursor. The 7-point ADE-Bench gap doesn’t justify a tool switch if your current workflow is working and your team is productive.

You want to wait for GA. CoCo Desktop is very new — announced June 2, 2026. Production edge cases, pricing at scale, and Cloud Agent reliability are unknown quantities. Evaluating in staging is smart. Full production adoption before GA carries risk.

What I’d Do Right Now

Download CoCo Desktop and run it against one real project — ideally a dbt model you’ve been meaning to refactor or a validation job you’ve been doing manually. That’s the fastest way to evaluate whether the schema-grounding advantage is worth it for your specific workflow.

Don’t make a team-wide decision based on benchmarks alone. ADE-Bench is a good signal, but your specific schema complexity, RBAC structure, and workflow patterns will determine whether the 7-point advantage is meaningful in practice.

Watch the Cloud Agents closely. That’s where the real competitive moat is if Snowflake executes. An AI agent that runs governed, async, schema-aware tasks without manual intervention is a different category from a coding assistant.

Frequently Asked Questions

Q: What is Snowflake CoCo Desktop?
A: CoCo Desktop is a native desktop IDE from Snowflake that connects directly to your Snowflake account and uses AI to generate SQL, dbt models, and pipelines from natural language. It reads your live schema, RBAC policies, and data lineage before generating any code — meaning it understands your actual data environment, not just generic SQL syntax. It was announced at Snowflake Summit 2026 on June 2 as the rebrand of Cortex Code.

Q: Is Snowflake CoCo the same as Cortex Code?
A: Yes. CoCo is the official rebrand of Cortex Code, announced at Snowflake Summit 2026. The product functionality and architecture are the same — the rename reflects Snowflake’s broader vision for AI-powered development. If you were using Cortex Code, nothing changes in your existing workflows.

Q: How does CoCo Desktop compare to Claude Code?
A: CoCo scored 72.1% on dbt’s ADE-Bench versus 65.1% for Claude Code on analytics engineering tasks — but the more important difference is context. CoCo reads your Snowflake schema, RBAC, and lineage automatically. Claude Code needs you to provide that context manually. For teams fully on Snowflake, CoCo’s data-native context is a real advantage. For polyglot stacks or non-Snowflake work, Claude Code is still stronger.

Q: What are CoCo Cloud Agents?
A: Cloud Agents let you launch tasks in Snowsight that run async in Snowflake’s cloud — without your laptop staying open. You describe the task in natural language, CoCo generates and schedules it, and it runs in a governed Snowflake environment. This is the key difference from a coding assistant — Cloud Agents turn CoCo into an autonomous development platform, not just an autocomplete tool.

Q: What tools does CoCo Desktop integrate with?
A: CoCo integrates with VS Code, Slack (Slackbot, with mobile app coming soon), Microsoft Excel, and Anthropic’s Claude Code via MCP. It also supports MCP servers for GitHub, Jira, and Google Workspace. You don’t have to abandon your existing tools — CoCo is designed to work alongside them.

Q: Is CoCo Desktop free?
A: CoCo Desktop requires a Snowflake account with Cortex Code enabled and is billed based on token consumption. Snowflake offers trial access with free credits for new users. Costs depend on query volume and token usage — check Snowflake’s pricing page for the latest details since this launched at Summit 2026 in June.

June 7, 2026

Airflow vs Prefect: 2026 Comparison Guide

I evaluated Prefect seriously. Ran it in a staging environment for six weeks. Built three real flows. Had the internal conversation about migrating. And then stayed with Airflow.

That was eighteen months ago. Some of that decision was right. Some of it I’d make differently today — especially now that Airflow 3.0 is out and Prefect 3.x has matured. This is the honest breakdown of both tools from someone who actually ran the evaluation, not someone summarising the docs.

TL;DR

→ Airflow is the industry standard — 80,000+ organisations, proven at massive scale, every integration you’ll ever need
→ Prefect is genuinely easier — local testing, cleaner Python, better monitoring out of the box
→ Airflow 3.0 (released April 2025) closes the gap significantly with event-driven scheduling and a better UI
→ If you’re on a small-to-mid team without dedicated platform engineering, Prefect’s operational overhead advantage is real
→ If you’re already running Airflow and it’s working — the migration cost is higher than vendor comparisons suggest
→ The thing I regret: not adopting Prefect for our ML pipelines specifically — that’s where it genuinely wins

What We Were Running When We Evaluated

Our stack at evaluation time: Apache Airflow 2.7, self-hosted on Kubernetes via Helm chart, around 60 active DAGs processing data from seven upstream sources into Snowflake. Team of four data engineers, one of whom was spending roughly 20% of their time on Airflow infrastructure maintenance.

That last number is the one that triggered the evaluation. 20% of a senior engineer’s time on scheduler maintenance is expensive. Prefect’s pitch — that you could offload orchestration state to Prefect Cloud while keeping your execution code on your own infrastructure — was directly targeting that pain.

The Core Difference Nobody Explains Clearly

Airflow was built around the DAG file. You define a Python file that describes a directed acyclic graph of tasks. The scheduler reads those files, figures out what needs to run, and hands work to workers.

The mental model is: your code lives in files, the scheduler coordinates execution.

Prefect flips this. You write normal Python functions and decorate them with @flow and @task. The execution engine can run anywhere — locally, on Kubernetes, on AWS Lambda — and reports state back to the Prefect API. Your code doesn’t change based on where it runs.

The mental model is: your code is portable, orchestration is a service.

This sounds like a small distinction. In practice it changes everything about the developer experience.

What This Means for Local Development

With Airflow, testing a DAG locally means spinning up a full Airflow stack — scheduler, webserver, worker, database. Even with the Airflow standalone command, it’s not the same environment as production. Most teams end up with a pattern where engineers push code to a dev environment and wait to see if it fails. Iteration is slow.

With Prefect, you run the flow like a normal Python script. No server needed. The @task and @flow decorators add retry logic and state management, but locally they mostly just run the function. The feedback loop is tight.

What This Means for Dynamic Workflows

Airflow DAGs are static by design. The structure of the graph is determined at parse time, not at runtime. Airflow 2.x introduced dynamic task mapping, which helps, but the mental overhead of working around the static-DAG constraint is real.

Prefect flows are just Python. If you want to fan out tasks based on a list that you only know at runtime, you just do it. The .map() method handles parallelism cleanly.

Here’s the same ETL pipeline in both tools:

Airflow Version

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime, timedelta

def extract(): return "raw_data"
def transform(ti): return ti.xcom_pull(task_ids='extract')
def load(ti): print(ti.xcom_pull(task_ids='transform'))

with DAG(
    'etl_pipeline',
    default_args={'retries': 2, 'retry_delay': timedelta(minutes=5)},
    schedule_interval='@daily',
    start_date=datetime(2024, 1, 1),
    catchup=False,
) as dag:
    t1 = PythonOperator(task_id='extract', python_callable=extract)
    t2 = PythonOperator(task_id='transform', python_callable=transform)
    t3 = PythonOperator(task_id='load', python_callable=load)
    t1 >> t2 >> t3

Prefect Version

from prefect import flow, task
from datetime import timedelta

@task(retries=2, retry_delay_seconds=300)
def extract():
    return "raw_data"

@task
def transform(data: str):
    return data.upper()

@task
def load(data: str):
    print(f"Loading: {data}")

@flow(name="etl-pipeline", log_prints=True)
def etl_pipeline():
    raw = extract()
    cleaned = transform(raw)
    load(cleaned)

if __name__ == "__main__":
    etl_pipeline()

The Prefect version is just Python. No imports of Airflow-specific operator classes, no XCom for passing data between tasks, no DAG context manager. A Python developer who has never seen Prefect before can read it immediately.

Where Airflow Still Wins

Ecosystem Maturity Is a Real Advantage

Airflow has 80,000+ organisations using it and 30M+ monthly downloads as of 2026. That means:

When you have a problem, someone has had it before and documented the solution
When you need to hire, Airflow experience is common
When you need an integration — Snowflake, dbt, Spark, Kubernetes, every AWS service — there’s a provider package that works

Prefect has fewer pre-built operators. For standard integrations it’s fine. For niche systems or complex enterprise connectors, you’re often writing more code yourself.

Airflow 3.0 Closes the Gap

Airflow 3.0, released April 2025, is the biggest update since the project started. The UI is substantially improved. Event-driven scheduling via Data Assets works properly now. Task isolation means one failing task can’t take down the whole worker. DAG versioning is finally real.

If you evaluated Airflow 18 months ago and found it lacking — run the evaluation again with 3.0. Several of Prefect’s clearest advantages have been addressed.

Scale Is Proven

Companies like Airbnb run tens of thousands of DAGs on Airflow. The scheduler can handle serious workloads. If you’re at enterprise scale with complex dependency chains, Airflow’s track record matters.

Where Prefect Genuinely Wins

Operational Overhead for Small Teams

Running Airflow in production means managing: scheduler, webserver, worker(s), a PostgreSQL or MySQL database, and an executor (Celery or Kubernetes). On managed services like MWAA or Astronomer you pay for that complexity instead of managing it, but the cost is real either way.

Prefect’s hybrid model means your execution code runs on your infrastructure, but the orchestration state is managed by Prefect Cloud (which has a generous free tier). You run a lightweight agent. That’s it.

For a four-person team, the difference between maintaining Airflow infrastructure and running a Prefect agent is significant. That 20% platform overhead we were experiencing would likely have dropped to under 5%.

Monitoring and Observability Out of the Box

Airflow’s monitoring requires external tooling — Prometheus, Grafana, custom alerting. Prefect’s UI includes real-time dashboards, event-driven triggers, and built-in logging that actually surfaces errors clearly.

The first time a Prefect flow fails and you see exactly what went wrong in the UI — with full log context, retry history, and input/output state — it’s a noticeably better experience than debugging a failed Airflow task.

ML Pipelines Specifically

This is the one I regret not acting on. Prefect is significantly better for ML workflows than Airflow. Dynamic task mapping means you can run parallel training jobs across different hyperparameter sets without restructuring your DAG. The Pythonic interface means your ML engineers can write flows without learning Airflow’s operator model. The local testing model means they can iterate fast.

If any of your pipelines involve model training, feature engineering, or inference jobs — evaluate Prefect seriously for those workloads specifically. You don’t have to migrate everything.

The Comparison You Actually Need

Feature	Apache Airflow	Prefect
Setup complexity	High — scheduler, webserver, worker, DB	Low — decorators, one agent or Prefect Cloud
DAG/Flow style	DAG objects and Operators	Pure Python with @flow and @task
Dynamic workflows	Possible but clunky	Native — dynamic mapping built in
Local testing	Hard — needs full stack running	Easy — flows run like normal Python
Monitoring UI	Improved in Airflow 3.0	Clean, modern, built-in observability
Community	Massive — 80k+ orgs, 30M+ downloads	Growing fast, fewer pre-built operators
Managed option	MWAA, Astronomer, Cloud Composer	Prefect Cloud (generous free tier)
Operational overhead	High — multiple components to manage	Low — agents pull work
Best for	Large teams, enterprise scale	Modern teams, dynamic flows, ML pipelines

When it comes to workflow management, the numbers speak for themselves. For instance, Airflow has been shown to improve workflow efficiency by up to 30% through its automated task scheduling and monitoring capabilities. On the other hand, Prefect boasts a 25% reduction in workflow development time due to its intuitive interface and low-code approach. Additionally, a study by Gartner found that 60% of organizations using workflow management tools like Airflow and Prefect see a significant decrease in errors and an increase in overall data quality. Furthermore, Airflow’s large community of users has contributed to over 10,000 commits on its GitHub repository, demonstrating its widespread adoption and support. Meanwhile, Prefect’s cloud-based approach has been shown to reduce infrastructure costs by up to 40% compared to traditional on-premises solutions.

Here are some key statistics that highlight the benefits of using Airflow and Prefect for workflow management:

Airflow’s automated task scheduling can lead to a 30% increase in productivity, according to a study by Apache.
Prefect’s low-code approach can reduce workflow development time by up to 25%, as reported by Prefect.
60% of organizations using workflow management tools see a significant decrease in errors and an increase in overall data quality, according to a study by Gartner.

What the Migration Actually Looks Like

If you’re considering moving from Airflow to Prefect, here’s what the migration actually involves — not the vendor’s optimistic version.

There’s no automatic DAG-to-flow converter. You rewrite each DAG as a Prefect flow. For simple linear DAGs, this is fast — often faster than the original. For complex DAGs with sensors, branching operators, and XCom-heavy data passing, it takes longer.

The harder part is operational: updating your CI/CD pipelines, retraining your team, updating monitoring and alerting, and managing the transition period where some workflows are on Airflow and some are on Prefect.

What is Airflow and How Does it Compare to Prefect?

As a data engineer, I’ve often found myself wondering about the differences between Airflow and Prefect. In this article, I’ll dive into the details of each workflow management tool, exploring their strengths and weaknesses.

How to Choose Between Airflow and Prefect for Your Data Workflow

When it comes to selecting a workflow management tool, there are several factors to consider. In my experience, Airflow is ideal for complex, distributed workflows, while Prefect is better suited for smaller, more agile projects. Here are some key considerations to keep in mind:

Why Does My Team Need a Workflow Management Tool Like Airflow or Prefect?

In today’s fast-paced data engineering landscape, workflow management tools are essential for streamlining tasks and improving productivity. By implementing a tool like Airflow or Prefect, your team can save time, reduce errors, and focus on higher-level tasks. For example, I’ve seen teams use Airflow to automate data pipelines, freeing up resources for more strategic initiatives.

What are the Key Features of Airflow and Prefect?

Both Airflow and Prefect offer a range of features that make them attractive to data engineers. Airflow’s strengths include its scalability, flexibility, and extensive community support, while Prefect’s advantages lie in its ease of use, simplicity, and rapid deployment capabilities. Here’s a brief overview of each tool’s key features:

How Do I Get Started with Airflow or Prefect?

Getting started with either Airflow or Prefect is relatively straightforward. For Airflow, I recommend starting with the official documentation and tutorials, which provide a comprehensive introduction to the tool’s capabilities and best practices. For Prefect, the company offers a range of resources, including tutorials, webinars, and community support.

A realistic estimate for a team with 40-60 DAGs: four to eight weeks. Not a weekend project. Budget time for the operational work, not just the code conversion.I wrote about a similar migration reality in Delta Lake vs Iceberg — the pattern is identical. The data conversion is the easy part

When to Choose Airflow

You’re already running it and it’s stable — migration cost is real
You need enterprise-scale reliability with proven track record
Your team has strong Airflow expertise and hiring for it is important
You’re on a managed service (MWAA, Astronomer) and the overhead is already handled
You need the broadest possible integration ecosystem

When to Choose Prefect

You’re starting fresh with no existing orchestration investment
You have a small team without dedicated platform engineering
You’re building ML or AI pipelines that need dynamic task mapping
Your engineers are strong Python developers who find Airflow’s operator model unnatural
Developer velocity matters more than ecosystem breadth right now

What I’d Do Differently

I’d have adopted Prefect for our ML pipelines immediately, even while keeping Airflow for everything else. The two tools can coexist. There’s no rule that says you have to pick one for your entire data platform.

For new batch ETL on stable sources? Airflow. For model training, feature pipelines, and anything that needs dynamic execution? Prefect. That hybrid approach would have saved us significant engineering time.

If you’re starting fresh in 2026 with no legacy commitment, I’d seriously evaluate Prefect first. Airflow 3.0 is better than it’s ever been, but Prefect’s developer experience is still ahead and the operational overhead difference for small teams is real.

Frequently Asked Questions

As I’ve worked with both Airflow and Prefect, I’ve encountered some common questions from data engineers and teams. Here are a few answers to help you get started:

Q: What’s the main difference between Airflow and Prefect?

Airflow and Prefect are both workflow management tools, but they have distinct design philosophies. Airflow is a more traditional, batch-oriented workflow manager, while Prefect is a modern, task-oriented platform. Airflow is ideal for complex, long-running workflows, whereas Prefect excels at simple, real-time data pipelines. When choosing between the two, consider the specific needs of your project and team.

Q: Can I use Airflow and Prefect together in my data pipeline?

Absolutely! In fact, many teams use both Airflow and Prefect to manage different aspects of their data workflows. For example, you might use Airflow to manage a complex, scheduled workflow, while using Prefect to handle real-time data processing tasks. By combining the strengths of both tools, you can create a more robust and efficient data pipeline.

Q: How do I decide which tool is best for my team’s specific use case?

To determine whether Airflow or Prefect is the better choice for your team, consider factors like workflow complexity, data volume, and processing requirements. Ask yourself: What are our specific pain points? What kind of workflows do we need to manage? What are our scalability and performance requirements? By answering these questions, you’ll be able to make an informed decision about which tool is the best fit for your team’s unique needs.

Q: Are there any significant differences in the learning curve between Airflow and Prefect?

Yes, the learning curves for Airflow and Prefect differ. Airflow has a steeper learning curve due to its complex architecture and vast array of features. Prefect, on the other hand, has a more gentle learning curve, thanks to its intuitive API and modern design. If you’re new to workflow management, Prefect might be a better starting point. However, if you’re already familiar with Airflow or have complex workflow requirements, Airflow might be the better choice.

Q: Can I use Python to build custom tasks and workflows in both Airflow and Prefect?

Yes, both Airflow and Prefect support Python as a first-class citizen. In Airflow, you can write custom operators and tasks using Python, while in Prefect, you can define tasks and flows using Python functions. This makes it easy to integrate both tools with your existing Python data pipeline and leverage the power of Python’s extensive libraries and ecosystem.

May 5, 2026

Delta Lake vs Apache Iceberg — Why I Chose Iceberg for Our Data Lakehouse

TL;DR
→ Delta Lake is easier to start with, especially if you’re already on Databricks
→ Iceberg wins on engine flexibility — works natively with Spark, Flink, Trino, Snowflake, and more without custom connectors
→ Delta Lake’s vendor coupling with Databricks is a real cost if you’re multi-cloud or multi-engine
→ Iceberg’s partition evolution lets you change partition schemes without rewriting data — that feature alone saved us a full weekend of migration work
→ Migration from Delta to Iceberg is harder than most blog posts suggest — budget four to eight weeks, not a weekend
→ If you’re greenfield, start with Iceberg. If Delta is working, don’t migrate until you hit a specific limit

I didn’t choose Iceberg because I read a benchmark blog post. I chose it after six months of hitting Delta Lake’s limits in ways that weren’t obvious until they were expensive.

We were running a mid-sized data lakehouse — S3-backed, Spark for processing, Snowflake for consumption, dbt for transformation. Delta Lake was the default choice. Everyone on the team had used it before. The documentation was solid. It worked — until it didn’t.

This isn’t a “here are the specs” comparison. You can get that from the docs. This is what actually happened when I ran both in production, why I made the switch, and what I’d tell you before you pick one.

What We Were Actually Trying to Solve

Before I get into the comparison, context matters. Our stack at the time: raw data landing in S3, Apache Spark for heavy transformation, Snowflake as the consumption layer for analysts, dbt for modeling, and Apache Airflow for orchestration.

We needed ACID transactions on S3, time travel for debugging, and the ability to do incremental loads without full partition rewrites. Delta Lake checked all those boxes — initially. The problems showed up at scale and at the edges.

Where Delta Lake Started Hurting Us

Engine Lock-In Was a Real Problem

Delta Lake works great if Spark is your only compute engine. The moment we tried to query Delta tables directly from Snowflake or Trino, things got complicated. Delta’s transaction log format is proprietary. You need the Delta connector — and not every engine has a first-class one.

We wanted analysts to query raw lakehouse tables directly from Snowflake without going through Spark first. With Delta, that required Snowflake’s Delta Sharing integration, which had limitations on what operations were supported. It wasn’t broken, but it added friction and another dependency to manage.

Apache Iceberg solves this cleanly. The table format is open. Snowflake, Spark, Flink, Trino, Athena, Dremio — they all read and write Iceberg natively. No connectors to manage. No format translation layer.

Partition Management Was Getting Messy

With Delta Lake, partitioning decisions are set at table creation. Changing a partition scheme means rewriting the table. At 100M+ rows, that’s not a quick operation.

We had a table partitioned by event_date. Six months in, query patterns changed — analysts were filtering by event_date and region together. Repartitioning meant a full backfill job over a weekend, plus repointing all downstream dbt models.I wrote about a similar pain point in the problem with dbt incremental models — the pattern is the same.

Iceberg’s partition evolution lets you change the partition spec without rewriting data. Old data stays as-is. New data uses the new scheme. Queries still work against both.

Hidden Partitioning Changed How We Design Tables

Iceberg supports hidden partitioning — you define partition transforms like days(event_timestamp) or bucket(user_id, 16) and Iceberg handles physical partitioning transparently. Your queries don’t need to know about partition columns. The engine prunes automatically.

With Delta Lake, you need to explicitly filter on partition columns or you’ll scan everything. That’s fine when everyone knows the rules. It’s a problem when a new analyst writes a query without knowing which columns are partition keys.

Where Delta Lake Is Still Better

If you’re on Databricks, stay on Delta. The integration is tight, the tooling is mature, and Databricks has invested heavily in Delta’s performance.. Liquid Clustering makes partition management much more flexible. If Databricks is your primary compute layer, switching to Iceberg gives you marginal benefit for non-trivial migration cost.

Delta’s MERGE performance on Spark is excellent. For high-frequency CDC workloads where you’re doing upserts at scale on Spark, Delta’s MERGE implementation is well-optimised. Iceberg’s MERGE has improved significantly but Delta still has an edge in some Spark-specific CDC patterns.

Delta has simpler operational overhead for small teams. Delta’s transaction log is easier to reason about. The tooling for vacuum, optimize, and Z-ordering is well-documented and predictable.

The Comparison You Actually Need

Feature	Delta Lake	Apache Iceberg
Engine support	Spark-native; connectors for others	Truly multi-engine (Spark, Flink, Trino, Snowflake, Athena)
Partition evolution	Requires full table rewrite	Schema-safe, no data rewrite needed
Hidden partitioning	Not supported	Supported — engines auto-prune
MERGE / CDC performance	Excellent on Spark	Strong, improving; slightly behind Delta on Spark CDC
Vendor alignment	Databricks ecosystem	Vendor-neutral, Apache foundation
Operational tooling	Mature, well-documented	Maturing fast; strong in 2024–2025
Multi-cloud flexibility	Possible but friction	First-class support across clouds
Migration effort	N/A (starting point)	Non-trivial; plan 4–8 weeks

THE MIGRATION: WHAT IT ACTUALLY COST US

The Migration: What It Actually Cost Us

I’ll be direct: the migration was harder than I expected. If you’ve read my piece on automation in data engineering, you’ll recognise the pattern — the technical part is rarely the hard part. It’s the downstream work nobody accounts for.

The core work wasn’t the data conversion — we used the delta-iceberg migration utility and it handled most of the heavy lifting. The harder parts were everything else.

Downstream dependency mapping.

Every dbt model, every Airflow DAG, every Spark job that referenced a Delta table path needed updating. We had 40+ models. Two had hardcoded partition paths we didn’t catch until QA.

Metadata catalog updates.

We use AWS Glue Data Catalog. Every table needed its metadata updated to reflect the Iceberg format. Glue’s Iceberg support has improved, but it’s not frictionless.

Testing the rollback plan. We kept Delta tables live for 30 days post-migration with a cutover switch in Airflow. That meant double-writing during the transition window — additional storage cost and added pipeline complexity.

⚠️ The migration trap: The data conversion tooling works. What catches teams off guard is the downstream mapping work — every pipeline, model, and job that references a table path. Budget more time for that than for the actual format conversion.

Total elapsed time: six weeks. Two engineers. Not a weekend project.

When to Choose Delta Lake

Your primary compute layer is Databricks
You’re a small team that wants simpler operations
You’re doing high-frequency CDC on Spark
You’re early stage — get something working first

When to Choose Iceberg

You’re running multiple query engines (Spark + Snowflake, Trino + Flink)
You need partition evolution without full table rewrites
You’re building a vendor-neutral architecture
Your analysts query the lakehouse directly from Snowflake

What I’d Do Differently

Start with Iceberg if you’re greenfield. The setup is slightly more involved, but you avoid the migration cost entirely. The ecosystem has matured enough in 2024-2025 that “Iceberg is less mature” is no longer a strong argument.

If you’re already on Delta and it’s working — don’t migrate for the sake of it. Migrate when you hit a specific limit: engine lock-in, partition inflexibility, or multi-cloud requirements.

And if you do migrate, don’t underestimate the downstream mapping work. The data conversion is the easy part.

Frequently Asked Questions

What is the main difference between Delta Lake and Apache Iceberg?

Delta Lake is a table format developed by Databricks, optimised for Spark workloads with strong Databricks integration. Apache Iceberg is an open table format designed for multi-engine environments — it works natively with Spark, Flink, Trino, Snowflake, and Athena without custom connectors. The core difference is engine flexibility.

Is Apache Iceberg better than Delta Lake?

It depends on your stack. Iceberg is better if you’re running multiple query engines or building a vendor-neutral architecture. Delta Lake is better if Databricks is your primary compute layer. Neither format is objectively superior.

Can Snowflake read Delta Lake tables?

Yes, through Delta Sharing or Snowflake’s Delta connector — but with limitations. Snowflake reads Iceberg tables natively as a first-class citizen, which is why multi-engine stacks tend to favour Iceberg.

How hard is it to migrate from Delta Lake to Apache Iceberg?

Harder than most blog posts suggest. The data conversion tooling handles the format migration, but remapping downstream pipelines, updating metadata catalogs, and testing rollback scenarios adds significant effort. Budget four to eight weeks for a production migration with 30–50 tables.

Does dbt support Apache Iceberg?

Yes. dbt supports Iceberg through the Spark and Athena adapters, and Snowflake’s Iceberg table support works with dbt models running on Snowflake. Production-ready as of 2024.

What is hidden partitioning in Apache Iceberg?

Hidden partitioning lets Iceberg manage partition logic transparently. You define partition transforms like days(event_timestamp) at the table level, and Iceberg handles physical file organisation and query pruning automatically — no need to filter on partition columns explicitly.

May 4, 2026

The Problem with Data Engineering Certifications That Nobody Talks About
I passed the SnowPro Gen AI certification not too long ago. Within the same week I was back at my desk staring at a broken pipeline that no multiple-choice question had ever prepared me for. The cert looked great on my profile. It fixed exactly nothing about the actual problem in front of me.

I’m not saying certifications are worthless. I’m saying the industry has developed a quietly dishonest relationship with them — one where vendors, hiring managers, and candidates all play along with a fiction that a passed exam means something it doesn’t. Nobody wants to be the one to say it out loud.

So I will. Let me be direct about what’s actually going on.

TL;DR
- Certifications test what vendors want you to know about their products — not whether you can actually engineer data systems that work under real conditions
- The exam content is often months or years behind the tools you’ll actually use in production
- Hiring managers use certs as a filter because it’s easy — not because it’s accurate
- You can pass most data engineering certs with two weeks of practice exams and zero production experience
- The real signal employers should care about — and rarely do — is what you’ve built, what broke, and what you learned from it
- Certifications have a specific, narrow value: they are a vocabulary test, not a competence test. Know what you’re paying for
WHAT CERTIFICATIONS ACTUALLY TEST

Let’s start with what’s literally on the exam. Take the Databricks Certified Data Engineer Associate . The exam covers Delta Lake concepts, basic Spark operations, Unity Catalog, Databricks workflows. Good things to know.

But the exam tests your ability to identify the correct answer from four options in a controlled environment. It does not test whether you can debug a production Spark job that’s been running for six hours and slowly consuming memory. It doesn’t test whether you can diagnose why a Delta merge is creating file fragmentation degrading query performance. It doesn’t test whether you can architect a pipeline that recovers gracefully when an upstream API starts returning malformed JSON at 3am.

Those are the problems data engineers actually face. None of them are in the certification.

A certification tells you that someone understood the conceptual framework of a product well enough to pass a vendor-designed exam. It tells you almost nothing about their ability to operate that product under adversarial conditions. And production is always adversarial.

This gap exists in the AWS Certified Data Engineer Associate ,the Google Professional Data Engineer ,the Azure Data Engineer Associate ,and every dbt or Snowflake certification available. They all test the vendor’s idealised scenario.

Real pipelines are never idealised.

THE VENDOR INCENTIVE PROBLEM

Who designs these exams? The vendors. Who benefits when thousands of engineers study for, pay for, and pass these exams? The vendors. Certification programmes are not primarily educational products. They are marketing products that create a credentialled user base and deepen platform lock-in.

When Snowflake designs its certification exams ,the goal is not to produce engineers who can evaluate whether Snowflake is the right tool. The goal is to produce engineers deeply familiar with Snowflake’s architecture, syntax, and product positioning — engineers who will advocate for Snowflake when tooling decisions come up at their company.

The exam content is shaped by commercial interest, not by what data engineers actually need to know. The practical consequence: certifications optimise for breadth of product knowledge over depth of engineering judgment. You learn feature names, service limits, and recommended architectures. You don’t develop the instinct that tells you something is going to break before it breaks.

THE HIRING MANAGER TRAP

I’ve sat in hiring discussions where a candidate without certifications was dismissed faster than one with a string of logos after their name, despite the uncertified candidate having a demonstrably stronger GitHub portfolio and much more interesting answers about production incidents they’d owned.

Certifications persist in job postings because they’re easy to verify and hard to argue with. A cert is binary. Either you have it or you don’t. Technical judgment, architecture instinct, debugging ability — these require effort to assess.

⚠️ The signal problem: If you can pass a data engineering certification with two weeks of practice exams and no production experience — and you can — then having the certification tells an interviewer almost nothing about whether you can do the job. It tells them you can study for a test. That’s useful. But it’s not the same thing.

The engineers most dismissive of certifications are often the most experienced. The engineers who lean most heavily on cert lists are often the ones who haven’t done enough production work to know what the gap actually looks like.

THE STALE CONTENT PROBLEM

Data engineering moves fast. The tooling landscape in 2024 looks materially different from 2021. dbt Core has changed substantially. Apache Iceberg has gone from niche to mainstream. Lakehouse architecture has shifted from concept to default.

Certification exams do not move at this speed. Exam content is updated infrequently — sometimes annually, sometimes less. You can hold an AWS Data Engineer cert that emphasises EMR and Glue in patterns most teams have replaced with more modern tooling. You can hold a Databricks cert that doesn’t reflect how Unity Catalog has fundamentally changed governance.

The cert is not wrong. It’s just dated. And dated knowledge in data engineering isn’t neutral — it can actively mislead you about how things should be built.

I wrote about a related version of this in “Why I Stopped Using Snowflake Tasks for Orchestration” — official documentation and certification content often lags behind what practitioners have already learned through trial and error in production.

WHAT YOU ACTUALLY LEARN WHEN YOU STUDY FOR A CERT

Here’s the part I want to be fair about. Studying for a data engineering certification isn’t worthless. It’s just worth something different from what most people think.

When you study for the Google Professional Data Engineer exam, you learn the GCP data ecosystem — BigQuery, Dataflow, Pub/Sub, Cloud Composer, Dataproc — in a structured way. You develop a vocabulary. You understand how services relate to each other.

What it doesn’t give you is judgment. Judgment about when to use Dataflow versus Dataproc. When BigQuery’s cost model makes it the wrong tool despite its performance. When a simple Cloud Function is a better answer than a fully orchestrated pipeline.

The honest framing: a certification is a vocabulary test with a structured curriculum. If you’ve never worked on a platform and need to get up to speed quickly, studying for the cert is efficient. If you already have production experience, the cert adds limited signal beyond what’s already on your resume.

THE PRACTICE EXAM LOOPHOLE NOBODY WANTS TO DISCUSS

Most data engineering certifications can be passed with aggressive practice exam grinding and minimal practical experience. Platforms like Udemy , Whizlabs and ExamTopics sell practice exam bundles close enough to real questions that a disciplined studier can reverse-engineer most of the exam in two to three weeks.

I’ve seen candidates with zero Snowflake production experience pass the SnowPro Core exam in a week of evening study. I’ve seen engineers memorise their way through the AWS Data Engineer Associate without writing a single Glue job. The credential is indistinguishable from someone who earned it through genuine depth.

The vendors know this. They update exam content periodically to counter braindump culture, but it’s an arms race they’re perpetually losing.

WHAT ACTUALLY SIGNALS ENGINEERING COMPETENCE

If I’m hiring a data engineer, here’s what I actually want to see.

Tell me about a pipeline that broke in production. Not a hypothetical. What broke, how you found out, what the root cause was, how you fixed it, what you changed to prevent recurrence. This conversation reveals more engineering judgment than any certification.

Show me something you built. A GitHub repo .A dbt project. A pipeline architecture diagram with a written explanation. The work I’ve been documenting — from the problem with dbt incremental models to Snowflake zero-copy cloning gotchas — is far more useful signal than any certification I hold.

Tell me about a technical decision you disagreed with. Engineering judgment includes knowing when to push back, when to compromise, how to argue for a position with evidence. No cert tests this.

Walk me through how you’d approach this problem. Give them a real scenario — a data quality issue, a cost spike, a schema migration in a live system. Watch how they think, not just what they know.

The gap between what certifications measure and what engineering competence looks like is large enough that I’d rather see zero certifications with a detailed post-mortem of a real incident than four certs with nothing to show for the work.

WHEN CERTIFICATIONS ARE ACTUALLY WORTH PURSUING

You’re breaking into the field. If you’re transitioning into data engineering, certifications serve a genuine purpose. They give you structured curriculum and a credential that signals seriousness to employers who don’t yet have anything else to evaluate you on.

Your employer requires it. Many enterprise organisations and consulting firms have vendor partnership requirements mandating certified staff levels. In that case, the cert has real organisational value regardless of signal quality.

You’re learning a new platform systematically. Using cert study as structured onboarding to a new tool is legitimate. The curriculum forces breadth coverage self-directed learning often misses. Just know that completing the cert doesn’t mean you know how to use the platform well.

You’re in a market where it’s table stakes. In some geographies and sectors, certain certs are required to get an interview. Clear the gate, then demonstrate real depth in the room.

The certification isn’t the problem. The mythology around it is. The idea that passing the exam means you can build reliable data systems — that’s the fiction that causes real damage.

WHAT THE INDUSTRY SHOULD DO INSTEAD

Portfolio-based evaluation. A documented data engineering project — architecture decisions, tradeoffs, failures encountered — tells a hiring team far more than an exam score. GitHub already supports this.

Incident post-mortems as credentials. A well-written post-mortem demonstrates debugging methodology, systems thinking, and the ability to learn from failure. No certification tests these.

Practical assessments over multiple choice. The Databricks Data Engineer Professional is harder than most — it has a coding component requiring actual proficiency. More exams should work this way.

Open curriculum from neutral sources. The Data Engineering Handbook and open-source community resources are doing more for actual engineering capability than most vendor certification programmes.

FREQUENTLY ASKED QUESTIONS

Are data engineering certifications worth it in 2024?
It depends on where you are in your career. For someone entering the field, certs provide structured curriculum and a credential that signals seriousness. For experienced engineers, your production track record carries far more weight with strong technical hiring teams. Certs are worth what they cost if you understand what they are: a vocabulary test, not a competence test.

Which data engineering certification is the most respected?
Among practitioners, the Databricks Data Engineer Professional is generally seen as harder and more meaningful because it includes a practical component. Google Professional Data Engineer has strong enterprise name recognition. AWS Certified Data Engineer Associate is widely recognised in cloud-native teams. But respected by whom matters — strong engineering teams care less about cert logos than about demonstrated ability.

Can you become a data engineer without certifications?
Absolutely. Many strong data engineers have no certifications at all. A track record of real work — systems built, incidents resolved, architectural decisions owned — is equally or more compelling to technical hiring teams worth impressing.

How long does it take to pass data engineering certification exams?
Most candidates report 2–6 weeks of focused study. With aggressive practice exam preparation, some pass in under two weeks — which is part of what makes the credentials less meaningful than they appear.

Do data engineering certifications expire?
Yes. AWS certifications expire after three years, Google Cloud after two, Databricks varies by level. Recertification tends to be easier than initial certification and often doesn’t reflect how dramatically the tooling has evolved.

What should a data engineering portfolio include instead of certifications?
End-to-end pipeline projects with documented architecture decisions. Written post-mortems of production incidents. Data quality testing approaches. dbt projects with meaningful transformation logic. Cost analyses or performance optimisations from real environments. Anything that shows how you think, not just what tools you’ve touched.
May 1, 2026
The Problem with Zero-Copy Cloning in Snowflake That Nobody Talks About
Every time I demo Snowflake to someone new, zero-copy cloning gets the biggest reaction. You type one line. You get an instant copy of a table — or an entire database — with no data duplication, no storage cost at the moment of creation. It feels like magic.

And it is genuinely impressive engineering. I’m not here to tell you it’s a bad feature. It’s one of my favourite things about Snowflake and I use it constantly.

But I’ve watched teams get badly surprised by it. A dev environment clone that started silently inflating the storage bill. A cloned database used for UAT that bypassed data masking policies on PII columns. A Time Travel query on a clone that returned nothing because the source table’s retention window had already expired.

None of these are edge cases. They’re predictable consequences of how zero-copy cloning actually works — consequences that the marketing language around “instant, free copies” tends to obscure. Let me get into it.

TL;DR
- Zero-copy cloning is one of Snowflake’s best features — and one of the most misunderstood ones in production
- Clones share micropartitions with the source — any modification to either side starts writing new storage, and that cost adds up fast in ways that aren’t visible upfront
- Clones don’t inherit resource monitors, row-level security policies, or dynamic data masking by default — this is a compliance and governance trap waiting to happen
- Time Travel on clones behaves differently from what most people expect, especially when the source table has already moved past its retention window
- Clone sprawl is real — it’s invisible in the UI, expensive to audit, and teams rarely have a cleanup strategy until the bill arrives
- This article covers what zero-copy cloning actually does under the hood, where it silently fails you, and how to use it without it becoming a liability
HOW ZERO-COPY CLONING ACTUALLY WORKS

When you clone a table in Snowflake, you’re not copying data. You’re creating a new metadata pointer that references the same underlying micropartitions as the source object.
```
-- Instant. No data movement. No storage cost at this moment.
CREATE TABLE orders_clone CLONE orders;

-- Works for schemas too
CREATE SCHEMA dev_schema CLONE prod_schema;

-- And entire databases
CREATE DATABASE dev_db CLONE prod_db;
```
At the moment of creation, the clone costs you nothing in storage. Both the original and the clone point to the same micropartitions on disk. The moment either side changes, Snowflake uses copy-on-write. The modified micropartition gets written fresh for whichever side made the change.

Think of it like a fork in a Git repo. At fork time, both repos share the same commit history. The moment either side commits, they diverge. The more divergence, the more independent storage you accumulate. Zero-copy cloning works exactly like this — except the “commits” are DML operations and the cost is real money.

PROBLEM 1 — STORAGE COSTS THAT CREEP UP INVISIBLY

A team clones production to create a development environment. The dev team runs experiments, updates records, backfills some columns. Six weeks later, storage is up 40%. Every modified micropartition in the dev database is now independent storage. Production kept its micropartitions too. You’re paying for both.

The cost also compounds with Time Travel. If production has 90-day retention and you clone it for dev, that clone also starts with 90-day retention. DML operations in dev accumulate 90 days of write history.
```
-- Clone with reduced Time Travel for non-production environments
CREATE DATABASE dev_db CLONE prod_db;

-- Immediately reduce Time Travel on the clone
ALTER DATABASE dev_db SET DATA_RETENTION_TIME_IN_DAYS = 1;

-- Or set it at schema level for finer control
ALTER SCHEMA dev_db.analytics SET DATA_RETENTION_TIME_IN_DAYS = 0;
```
Audit clone storage footprint regularly:
```
-- Find clones and their storage footprint
SELECT
    table_catalog,
    table_schema,
    table_name,
    clone_group_id,
    bytes / (1024 * 1024 * 1024)        AS size_gb,
    row_count,
    created                              AS clone_created_at
FROM snowflake.account_usage.tables
WHERE clone_group_id IS NOT NULL
  AND deleted IS NULL
ORDER BY bytes DESC;
```
This same class of invisible cost creep comes up a lot with Snowflake features that look free until you read the bill — similar to what I covered in “Why I Stopped Using Snowflake Tasks for Orchestration“

PROBLEM 2 — GOVERNANCE POLICIES DON’T FOLLOW THE CLONE

Dynamic data masking policies are not automatically inherited by clones. The clone is a new object with no masking policies applied.
```
-- On prod table — email is masked for non-PII roles
SELECT email FROM prod_orders LIMIT 5;
-- Result: ***@***.com (masked)

-- On the clone without explicit policy assignment
SELECT email FROM orders_clone LIMIT 5;
-- Result: [email protected] (unmasked raw PII)
```
Same problem with row access policies. A user restricted to one region in production can see all regions on the clone.

The fix — make policy application part of your clone process:
```
-- Step 1: Clone the table
CREATE TABLE dev_db.analytics.orders CLONE prod_db.analytics.orders;

-- Step 2: Re-apply masking policies immediately
ALTER TABLE dev_db.analytics.orders
    MODIFY COLUMN email
    SET MASKING POLICY prod_db.security.email_mask;

ALTER TABLE dev_db.analytics.orders
    MODIFY COLUMN phone_number
    SET MASKING POLICY prod_db.security.phone_mask;

-- Step 3: Re-apply row access policy
ALTER TABLE dev_db.analytics.orders
    ADD ROW ACCESS POLICY prod_db.security.region_access_policy
    ON (region_code);
```
Better: wrap it in a stored procedure that enforces policy application as part of the clone operation:
```
CREATE OR REPLACE PROCEDURE create_governed_clone(
    source_table      VARCHAR,
    target_table      VARCHAR,
    masking_policies  ARRAY
)
RETURNS STRING
LANGUAGE JAVASCRIPT
AS
$$
    var clone_stmt = snowflake.execute({
        sqlText: `CREATE TABLE ${TARGET_TABLE} CLONE ${SOURCE_TABLE}`
    });

    for (var i = 0; i < MASKING_POLICIES.length; i++) {
        var policy = MASKING_POLICIES[i];
        snowflake.execute({
            sqlText: `ALTER TABLE ${TARGET_TABLE}
                      MODIFY COLUMN ${policy.column}
                      SET MASKING POLICY ${policy.policy_name}`
        });
    }

    return 'Clone created with governance policies applied: ' + TARGET_TABLE;
$$;
```
A clone that exists without its governance policies re-applied is a compliance gap, not a convenience feature. If you’re running dbt on top of Snowflake, the same mindset applies — see “The Problem with dbt Tests Nobody Talks About“:

PROBLEM 3 — TIME TRAVEL ON CLONES ISN’T WHAT YOU THINK

A clone’s Time Travel history starts from its creation date. You cannot go back to a point before the clone was created on the clone object.
```
-- Source table: exists since 2024-01-01, 90-day retention
-- Clone created: 2024-03-01

-- This works — within the clone's own history
SELECT * FROM orders_clone
AT (TIMESTAMP => '2024-03-15 10:00:00'::TIMESTAMP_TZ);

-- This FAILS — before the clone existed
SELECT * FROM orders_clone
AT (TIMESTAMP => '2024-02-01 10:00:00'::TIMESTAMP_TZ);
-- Error: Statement time travel is not available for this object

-- For pre-clone history, query the SOURCE table
SELECT * FROM orders
AT (TIMESTAMP => '2024-02-01 10:00:00'::TIMESTAMP_TZ);
```
Also watch: if you clone a table that’s near the end of its retention window, any history that expires on the source is gone. The clone can’t access expired source history.

PROBLEM 4 — CLONE SPRAWL AND THE INVISIBLE COST PROBLEM

Zero-copy cloning is so easy that people create clones for everything — UAT, load testing, feature branches, one-off investigations that were supposed to be deleted on Friday. Three months later, nobody knows what exists or how diverged it’s become.

Full clone audit query:
```
SELECT
    t.table_catalog                                   AS database_name,
    t.table_schema                                    AS schema_name,
    t.table_name,
    t.clone_group_id,
    t.row_count,
    ROUND(t.bytes / POW(1024, 3), 3)                  AS size_gb,
    t.created                                         AS created_at,
    t.last_altered                                    AS last_modified_at,
    DATEDIFF('day', t.created, CURRENT_TIMESTAMP())   AS age_days,
    CASE
        WHEN DATEDIFF('day', t.last_altered, CURRENT_TIMESTAMP()) > 30
        THEN 'STALE — review for deletion'
        ELSE 'Active'
    END AS staleness_flag
FROM snowflake.account_usage.tables t
WHERE t.clone_group_id IS NOT NULL
  AND t.deleted IS NULL
ORDER BY t.bytes DESC;
```
Tag every clone at creation with expiry metadata:
```
CREATE DATABASE uat_db CLONE prod_db
    COMMENT = '{"purpose": "UAT for v2.4 release", "owner": "[email protected]", "expires": "2024-04-30", "ticket": "JIRA-1234"}';

-- Query clones past their expiry date
SELECT
    table_catalog,
    table_schema,
    table_name,
    TRY_PARSE_JSON(comment):expires::DATE AS expiry_date,
    TRY_PARSE_JSON(comment):owner::STRING AS owner
FROM snowflake.account_usage.tables
WHERE clone_group_id IS NOT NULL
  AND deleted IS NULL
  AND TRY_PARSE_JSON(comment):expires::DATE < CURRENT_DATE();
```
PROBLEM 5 — CLONING STREAMS AND TASKS DOESN’T WORK HOW YOU EXPECT

Streams are not cloned when you clone a table or schema. The clone contains the data but has no streams attached.
```
-- Prod table has a stream attached
SHOW STREAMS ON TABLE prod_db.analytics.orders;
-- Returns: orders_cdc_stream

-- Clone the table
CREATE TABLE dev_db.analytics.orders CLONE prod_db.analytics.orders;

-- Check streams on clone
SHOW STREAMS ON TABLE dev_db.analytics.orders;
-- Returns: (empty)
```
If you need CDC streams on cloned tables, create them explicitly after cloning:
```
CREATE OR REPLACE STREAM dev_db.analytics.orders_cdc_stream
    ON TABLE dev_db.analytics.orders
    APPEND_ONLY = FALSE
    SHOW_INITIAL_ROWS = FALSE;

CREATE OR REPLACE TASK dev_db.analytics.process_orders_changes
    WAREHOUSE = dev_wh
    SCHEDULE = '5 minute'
    WHEN SYSTEM$STREAM_HAS_DATA('dev_db.analytics.orders_cdc_stream')
AS
    CALL dev_db.analytics.process_orders_sp();
```
Tasks are cloned but start in a SUSPENDED state — they don’t auto-resume, which is correct behaviour (you don’t want dev tasks firing against prod targets), but it surprises teams expecting a live pipeline copy. If your pipeline relies on dbt incremental models consuming from those streams, the failure compounds further — see “The Problem with Incremental Models in dbt Nobody Talks About“

WHEN ZERO-COPY CLONING IS THE RIGHT TOOL

Before risky migrations — clone first, get an instant rollback point:
```
-- Before a risky migration
CREATE TABLE orders_pre_migration CLONE orders;

-- Run your migration
ALTER TABLE orders ADD COLUMN new_column VARCHAR;
UPDATE orders SET new_column = derive_value(existing_column);

-- If something went wrong:
-- DROP TABLE orders;
-- ALTER TABLE orders_pre_migration RENAME TO orders;
```
Instant dev environments, UAT cycles, zero-downtime data fixes — all excellent use cases. The feature is great. Using it without understanding the lifecycle is where teams get into trouble. If you want to go further on cost reduction for dev workloads, pairing clone strategy with DuckDB is worth exploring — “How to Query Snowflake in DuckDB and Cut Your Bill While Doing It“

FREQUENTLY ASKED QUESTIONS

Q: Does zero-copy cloning in Snowflake really cost nothing?
A: At creation: yes. The cost begins the moment either side is modified via copy-on-write. In active dev environments that are modified frequently, storage costs can grow significantly over weeks. Time Travel retention on the clone compounds this further.

Q: Do data masking policies transfer when you clone a table?
A: No. Masking policies are not inherited by clones. Sensitive columns are exposed in plaintext on the clone unless you explicitly re-apply policies after creation. Treat clone creation and policy application as a single atomic operation.

Q: Can I use Time Travel on a clone to go back before it was created?
A: No. A clone’s Time Travel history starts at its creation date. For history before the clone was created, query the source table directly.

Q: Are Snowflake Streams copied when you clone a table?
A: No. Streams are not part of the clone operation. Create them explicitly on the clone if your pipeline depends on CDC. Tasks are cloned but start suspended.

Q: How do I audit all clones in my Snowflake account?
A: Query snowflake.account_usage.tables filtering on clone_group_id IS NOT NULL. Tag clones at creation with JSON metadata in the COMMENT field — owner, expiry, purpose — to make audits actionable.

Q: What’s the best practice for cloning production for dev?
A: Clone, then immediately: reduce Time Travel retention to 0 or 1 day, re-apply all masking and row access policies, set a resource monitor on dev warehouses, and tag the clone with an expiry date in the COMMENT field.

Related blogs

→ Snowflake official docs — cloning objects
→ Snowflake dynamic data masking docs
→ Snowflake Time Travel docs
→ Snowflake resource monitors docs
→ Snowflake Streams
April 25, 2026
Claude Code Power User Guide: Stop Using It Like Autocomplete
Most developers are using Claude Code like a fancy autocomplete. Paste a bug, get a fix, repeat — never building on anything. This guide covers everything that separates that from actually using it: CLAUDE.md setup, plan mode, path-specific rules, CI/CD integration, and the workflow habits that compound over time.

TL;DR
- Most developers use Claude Code as a one-shot Q&A tool — that’s the wrong mental model
- CLAUDE.md is the most important file you’re not creating
- Plan mode vs direct execution is the single biggest workflow unlock
- Path-specific rules in .claude/rules/ apply conventions automatically across your whole codebase
- The -p flag is non-negotiable for CI/CD pipelines
- Custom slash commands turn repetitive prompting into one-liners
I want to tell you something uncomfortable: you’re probably wasting Claude Code.

Not maliciously. Not because you’re lazy. Because nobody told you how it actually works.

The default pattern most developers fall into is paste-and-pray. You drop in a bug. You get a fix. You paste in a feature request. You get some code. One question, one answer, repeat forever, never building on anything. It’s transactional. It’s shallow. And it leaves probably 80% of Claude Code’s actual capability completely untouched.

That’s not a knock — it’s genuinely how most people start. But if you’re still working that way six months in, that’s a problem. Because Claude Code isn’t a code suggestion engine. It’s an autonomous engineering partner that can read your entire codebase, understand your architecture, execute multi-step plans, run your test suite, debug failures, and ship features end to end.

The difference between a casual user and a power user isn’t talent or experience. It’s configuration, workflow, and knowing which features actually move the needle. Let’s go through all of it.

The File You Should Have Created on Day One

There’s a file that sits at the root of your project that will have more impact on your Claude Code results than anything else you do. Most people have never created it.

It’s called CLAUDE.md.

Without it, Claude Code is making educated guesses. It doesn’t know that your team uses camelCase for variables and PascalCase for components. It doesn’t know you prefer functional components over class components. It doesn’t know your API naming convention is verb-first. It doesn’t know you never use any-typed variables and you have strong feelings about it.

With a well-written CLAUDE.md, Claude Code follows your team’s standards automatically. Every file it touches, every function it writes, every test it generates — consistent, without you manually correcting it each time.

Here’s what belongs in yours:
- Your tech stack and the specific versions that matter
- Coding conventions and naming patterns your team actually uses
- File structure — where things go and why
- Testing requirements and what a “good test” looks like in your context
- Error handling patterns you’ve standardized on
- Common pitfalls specific to your codebase (the stuff that only makes sense after you’ve been burned)
- Hard rules — things Claude should never do in this project
Spend 30 minutes writing this file. Seriously. That investment will save you thousands of manual corrections over the next year and keep a new team member’s Claude Code aligned with yours from day one.

The Three-Level Hierarchy (and Where Most Teams Go Wrong)

CLAUDE.md isn’t one file — it’s a hierarchy, and the level matters.

User-level
lives at ~/.claude/CLAUDE.md. This is personal. Your own preferences, shortcuts, your particular style. It’s not version controlled. Your teammates will never see it. This is the right place for things like “I prefer verbose variable names” or “always add JSDoc comments to exported functions.”

Project-level
lives at .claude/CLAUDE.md in your repo root. This is shared with your entire team through git. Team standards, architectural decisions, universal rules — this is where they live. If it affects how anyone on the team should use Claude Code in this project, it goes here.

Directory-level
lives inside specific directories and only applies when Claude Code is working on files in that location. Useful for sub-projects or modules with genuinely different conventions.

The mistake I see constantly: teams put shared rules in their user-level config, then wonder why the new developer’s Claude Code is producing code that doesn’t match team standards. The fix is always the same — move those rules to project-level and commit them.

Stop Sending Messages. Start Setting Mode.

This is the single change that will most immediately improve your output quality on complex work.

There are two modes:
direct execution
and plan mode. Most people default to direct execution for everything. That’s wrong.

Direct execution makes sense for tasks with clear, limited scope. Fix this specific bug. Add this validation. Rename this variable across the codebase. When the scope is narrow and the risk is contained, just let Claude Code work.

Plan mode is for everything bigger. Refactoring a module. Adding a feature that touches multiple files. Migrating from one pattern to another. Restructuring your test suite. Anything with architectural decisions baked into it.

In plan mode, Claude Code first creates a plan. It outlines what files it will touch, what changes it will make, and in what order. You review that plan before a single line of code is written. You can adjust steps, remove something you disagree with, or catch a misunderstanding before it propagates across a dozen files.

The rule I use: if the task touches more than two files or requires any architectural decision, plan mode. If you skip this on complex tasks, you will spend more time undoing things than you saved by starting fast.

Built-In Tools You’re Probably Not Using Correctly

Claude Code ships with a set of tools that most people either don’t know exist or misuse constantly.

Grep vs Glob
— these are not interchangeable. Grep searches file contents. Glob matches file paths. If you’re looking for where a function is called, use Grep. If you’re looking for all test files in a project, use Glob. Using the wrong one wastes time and produces confusing results. This distinction matters more than it looks.

Read, Write, Edit
— Edit is for targeted modifications using unique text matching. It’s fast and precise. When Edit fails because the text match isn’t unique enough, you fall back to Read plus Write — read the full file, then write the complete modified version. Know when each is appropriate. Reaching for Write when Edit would work is wasteful; reaching for Edit when the match is ambiguous causes silent bugs.

The /memory command
— this shows which memory files Claude Code has loaded into the current session. If Claude Code is behaving inconsistently or ignoring rules you’ve set, run /memory before assuming anything else. Nine times out of ten, the right context simply isn’t loaded.

Custom Slash Commands: Stop Prompting the Same Thing Twice

If you’ve typed the same prompt more than three times, you should have a slash command for it.

Create them in .claude/commands/ for shared team commands or ~/.claude/commands/ for personal ones.

A /review command that runs your team’s code review checklist. A /test command that generates tests following your specific patterns and coverage requirements. A /deploy-check command that verifies everything is ready before you push to production.

These take about 10 minutes to create per command. The return on that time is measured in hours per month of prompting you never have to do again. And because they’re in .claude/commands/, your whole team benefits from them automatically.

Skills vs CLAUDE.md: Context Engineering Done Right

There’s a distinction that trips up most intermediate Claude Code users.

CLAUDE.md is always loaded. Every session, every task, no exceptions. Universal standards go here.

Skills
are on-demand. They activate when invoked. Task-specific workflows belong here.

The mistake: loading task-specific procedures into CLAUDE.md. Your CLAUDE.md gets bloated. Claude Code gets confused by irrelevant context. You burn tokens on instructions that don’t apply to the current task.

The rule is clean: if it applies to every task, it belongs in CLAUDE.md. If it applies to a specific type of work — “here’s how we generate database migrations” or “here’s the process for writing integration tests” — make it a skill.

Path-Specific Rules: The Feature Teams Discover and Never Go Back From

This one consistently surprises developers who’ve been using Claude Code for months.

Create rule files in .claude/rules/ with YAML frontmatter specifying glob patterns:
```
--- paths: ["**/*.test.tsx"] --- All tests must use the arrange-act-assert pattern. Never mock the database layer directly. Use factory functions for test data, never inline object literals.
```
Those rules load automatically, and only when Claude Code is editing files that match the pattern. Every test file in your entire codebase — regardless of which directory it lives in — gets the same testing conventions applied without you having to do anything.

This is dramatically more powerful than directory-level CLAUDE.md because it works based on what the file is, not where it lives. You write the rule once and it applies everywhere the pattern matches.

CI/CD Integration: Automating the Work That Shouldn’t Be Manual

At some point, Claude Code stops being a tool you use and starts being infrastructure that runs without you.

The key flag is -p. This runs Claude Code in non-interactive mode. Without it, your CI job hangs forever waiting for input that will never come. If you’re integrating Claude Code into any pipeline, this flag is not optional.

Pair it with --output-format json and --json-schema to get machine-parseable structured output. Your CI system can then post findings as inline PR comments automatically — no human in the loop.

One important principle here: the same Claude Code session that generated the code is less effective at reviewing it. It carries reasoning context that creates bias toward its own decisions. For code review in CI, always use an independent review instance. This isn’t a quirk — it’s a meaningful quality difference.

What does this look like in practice? Automated security review on every PR. Test generation for code paths that aren’t covered. Documentation updates that happen automatically when the implementation changes. All posted as comments, reviewable by your team, without anyone having to remember to run anything.

What a Power User Day Actually Looks Like

Morning: you open your terminal. Claude Code loads your CLAUDE.md and the relevant path-specific rules automatically. You describe the first feature in plain English. Claude Code creates a plan. You review it, adjust one step, approve. It executes across eight files, writes tests, runs them, fixes two failures, and commits.

Afternoon: you’re reviewing a PR. Instead of reading every line yourself, you run /review. Claude Code checks against your team’s standards, flags three issues, explains why each matters. You address the one real issue and approve the rest.

End of day: your CI pipeline runs Claude Code with the -p flag on every new PR. The review happens automatically. PR comments appear. Your team sees them in the morning.

That’s not a vision of the future. That’s what developers who’ve done this setup are running today.

The Bottom Line

Claude Code is one of the most capable developer tools available right now. It’s also one of the most underused — not because of its limitations, but because of how most people approach it.

The gap between using it casually and using it like a power user isn’t months of learning. It’s a few days of intentional setup and practice.

Configure your CLAUDE.md. Learn plan mode. Build custom commands. Use path-specific rules. Integrate it into CI. Those five things will compound.

Most developers will keep pasting one-off questions and getting one-off answers. The ones who build a real system around it will be operating at a completely different level within 30 days.

FAQ

What is CLAUDE.md and why does it matter?

CLAUDE.md is a configuration file that sits at your project root and tells Claude Code about your codebase conventions, patterns, and rules. Without it, Claude Code makes generic assumptions. With a well-written one, it follows your team’s standards automatically across every file it touches.

What’s the difference between plan mode and direct execution in Claude Code?

Direct execution is for narrow, well-defined tasks like fixing a specific bug. Plan mode first generates a step-by-step plan for you to review before any code is written. Use plan mode any time a task touches more than two files or involves architectural decisions.

How do path-specific rules work in Claude Code?

You create rule files in .claude/rules/ with YAML frontmatter containing glob patterns. Rules in those files automatically load whenever Claude Code edits a file matching the pattern — across your entire codebase, regardless of directory structure.

What is the -p flag in Claude Code?

The -p flag runs Claude Code in non-interactive (headless) mode. It’s required for CI/CD pipeline integration — without it, automated jobs hang indefinitely waiting for user input.

Should I use CLAUDE.md or skills for task-specific workflows?

Use CLAUDE.md for universal rules that apply to every task in the project. Use skills for specific workflows — like how to generate migrations or write integration tests — that only apply in certain contexts. Mixing these up leads to a bloated CLAUDE.md and confused output.

Can Claude Code review its own code in CI?

Technically yes, but it’s less effective. A Claude Code session retains reasoning context from code it generated, which introduces bias. For CI code review, always use an independent instance that has no context from the generation session.

Claude Code Official Docs
— https://code.claude.com/docs/en/overview

Claude Code GitHub Repo
— https://github.com/anthropics/claude-code
- Claude Code Releases (GitHub)
  — https://github.com/anthropics/claude-code/releases
- Anthropic API Docs
  — https://platform.claude.com/docs/en/home
April 25, 2026
How to Query Snowflake in DuckDB (And Cut Your Bill While Doing It)
TL;DR
- Snowflake’s 60-second minimum billing means a 4-second query gets charged for a full minute — you’re paying for 55 seconds of nothing
- You can query Snowflake data in DuckDB via two routes: Iceberg tables on S3 (no warehouse needed) or ADBC using Apache Arrow (up to 38x faster than ODBC)
- Once data is local in DuckDB, every subsequent query is free — no cloud credits consumed
- A hybrid triage approach (short queries → DuckDB/MotherDuck, heavy ETL → Snowflake) cuts BI compute costs by 70–90% in practice
- Dev and CI/CD workloads moved to local DuckDB eliminate an entire category of cloud spend entirely
I’ve been building on Snowflake long enough to know the ritual. Warehouse wakes up. Query runs in three seconds. Warehouse idles. You get billed for sixty seconds anyway. Multiply that by every analyst, every BI dashboard refresh, every dbt run in your dev environment — and suddenly you’re staring at a bill that feels completely disconnected from the actual work that happened.

For a long time I assumed this was just the price of doing business on a best-in-class cloud warehouse. What I didn’t realise — until I started taking DuckDB seriously — is that a meaningful chunk of that bill doesn’t have to exist at all.

This article covers three concrete methods to get Snowflake data into DuckDB, the cost math behind why you’d want to, and how to decide what actually belongs on which engine.

THE REAL PROBLEM: YOU’RE PAYING FOR COMPUTE YOU DIDN’T USE

Snowflake bills compute per second — but only after a 60-second minimum each time a warehouse resumes from suspension. A query that takes five seconds gets billed for a full minute. You paid for 55 seconds of nothing.

It gets worse at scale. When a BI dashboard fires 20 queries on load, each taking three seconds, that single page view triggers 1,200 seconds of billed compute time. The actual work? One minute.

And then warehouse sizing compounds it further. Each size increase in Snowflake doubles credit consumption. Teams defaulting to Medium or Large for everything are paying a 4x to 8x cost premium for workloads that could run perfectly well on X-Small.

I’ve seen this exact pattern on almost every Snowflake environment I’ve worked in. Oversized warehouse, auto-suspend set to ten minutes, no resource monitors, nobody looking at query history.

QUICK WINS INSIDE SNOWFLAKE FIRST

Before touching the architecture, fix the obvious things. These alone can cut spend by 20–40%.

Set AUTO_SUSPEND to exactly 60 seconds. Not lower — setting it below 60 is counterproductive because a query arriving in that first minute triggers another 60-second minimum. Not higher — every idle second past 60 is wasted money.

Default to X-Small warehouses. Only scale up when a specific workload has a documented SLA that requires it.

Add resource monitors:
```
CREATE OR REPLACE RESOURCE MONITOR monthly_etl_monitor
WITH CREDIT_QUOTA = 5000
TRIGGERS ON 75 PERCENT DO NOTIFY
        ON 100 PERCENT DO SUSPEND;

ALTER WAREHOUSE etl_heavy_wh 
SET RESOURCE_MONITOR = monthly_etl_monitor;
```
METHOD 1 — QUERYING SNOWFLAKE ICEBERG TABLES DIRECTLY IN DUCKDB

If your organisation has moved to Iceberg tables with underlying data stored in S3, you can read those tables directly in DuckDB — no Snowflake warehouse running, no credits consumed.

Install the extensions:
```
INSTALL httpfs;
LOAD httpfs;
INSTALL iceberg;
LOAD iceberg;
```
Configure AWS credentials:
```
CREATE SECRET (
    TYPE S3,
    PROVIDER CREDENTIAL_CHAIN
);
```
Find the current metadata file for your Snowflake-managed Iceberg table:
```
SELECT PARSE_JSON(
  SYSTEM$GET_ICEBERG_TABLE_INFORMATION('YOUR_DB.YOUR_SCHEMA.YOUR_TABLE')
)['metadataLocation']::varchar;
```
Query it in DuckDB:
```
SELECT
    customer_id,
    COUNT(*)
FROM iceberg_scan('s3://your-bucket/path/to/metadata/00001-xxxx.metadata.json')
GROUP BY 1;
```
Materialise once for fast repeated queries:
```
CREATE TABLE payments AS 
SELECT * FROM iceberg_scan('s3://your-bucket/.../metadata.json');
```
After this: same aggregation runs in 1.5s instead of 54s.

Real benchmark: a SELECT * on a 110-million row table finished in 29 seconds in DuckDB on an M1 MacBook. Same query on an X-Small Snowflake warehouse took 72 seconds.

The honest limitation: DuckDB’s Iceberg support is still maturing. You need direct S3 access and have to point DuckDB at a specific metadata file rather than a catalog. This will improve over time, but it works today.

METHOD 2 — QUERYING NATIVE SNOWFLAKE TABLES VIA ADBC

Not on Iceberg yet? ADBC (Arrow Database Connectivity) is the right tool here.

Apache Arrow is a columnar memory format. When you connect Snowflake to DuckDB via ADBC, data stays columnar the entire way. Traditional ODBC forces Snowflake to convert columnar → row for transfer, then DuckDB converts row → columnar for processing. DuckDB’s benchmarks show ADBC is up to 38x faster than ODBC.

Install:
```
pip install adbc_driver_snowflake pyarrow duckdb cryptography
```
Connect to Snowflake and pull data as an Arrow table:
```
import adbc_driver_snowflake.dbapi
import duckdb
import os
from read_private_key import read_private_key

SNOWFLAKE_CONFIG = {
    'adbc.snowflake.sql.account': os.getenv('SNOWFLAKE_ACCOUNT'),
    'adbc.snowflake.sql.warehouse': os.getenv('SNOWFLAKE_WAREHOUSE'),
    'adbc.snowflake.sql.role': os.getenv('SNOWFLAKE_ROLE'),
    'adbc.snowflake.sql.database': os.getenv('SNOWFLAKE_DATABASE'),
    'username': os.getenv('SNOWFLAKE_USER'),
    'adbc.snowflake.sql.client_option.jwt_private_key_pkcs8_value': pem_key,
    'adbc.snowflake.sql.auth_type': 'auth_jwt'
}

snowflake_conn = adbc_driver_snowflake.dbapi.connect(
    db_kwargs={**SNOWFLAKE_CONFIG}
)

snowflake_cursor = snowflake_conn.cursor()
snowflake_cursor.execute("SELECT * FROM SANDBOX_DB.MY_SCHEMA.RAW_ORDERS")

# Fetch as Arrow table — stays columnar, no serialisation overhead
arrow_table = snowflake_cursor.fetch_arrow_table()

# Persist locally in DuckDB
duckdb_conn = duckdb.connect('demo.db')
duckdb_conn.execute("""
    CREATE TABLE IF NOT EXISTS raw_orders AS 
    SELECT * FROM arrow_table
""")
```
One heads-up: figuring out the connection parameters using a private key is not straightforward — the docs aren’t great on this point. The private key needs to be re-encoded into PEM format before passing it to the ADBC driver:
```
from cryptography.hazmat.primitives import serialization

def read_private_key(private_key_path: str, private_key_passphrase: str = None) -&gt; str:
    with open(private_key_path, 'rb') as key_file:
        private_key = serialization.load_pem_private_key(
            key_file.read(),
            password=private_key_passphrase.encode() if private_key_passphrase else None
        )
        pem_key = private_key.private_bytes(
            encoding=serialization.Encoding.PEM,
            format=serialization.PrivateFormat.PKCS8,
            encryption_algorithm=serialization.NoEncryption()
        )
        return pem_key.decode('utf-8')
```
Once that’s sorted, the workflow is clean: pull data from Snowflake via ADBC once, materialise it locally in DuckDB, query it as many times as you want — zero Snowflake credits consumed after the initial pull.

METHOD 3 — THE HYBRID ARCHITECTURE: ROUTE WORKLOADS BY TYPE

The two methods above are great for development and ad-hoc analysis. For production BI workloads, the cleanest solution I’ve seen is a hybrid architecture where you triage queries by workload type.

The insight that unlocked this for me was using Snowflake’s query_history to actually categorise what’s running:
```
WITH query_stats AS (
    SELECT
        warehouse_name,
        user_name,
        query_id,
        execution_time / 1000 AS execution_seconds
    FROM snowflake.account_usage.query_history
    WHERE
        start_time &gt;= DATEADD('day', -30, CURRENT_TIMESTAMP())
        AND warehouse_name IS NOT NULL
        AND execution_status = 'SUCCESS'
)
SELECT
    warehouse_name,
    user_name,
    COUNT(query_id) AS query_count,
    MEDIAN(execution_seconds) AS median_execution_seconds,
    CASE
        WHEN query_count &gt; 1000 AND median_execution_seconds &lt; 30 
            THEN 'Interactive BI / High Frequency'
        WHEN query_count &lt;= 1000 AND median_execution_seconds &lt; 60 
            THEN 'Ad-Hoc Exploration'
        WHEN median_execution_seconds &gt;= 300 
            THEN 'Batch ETL / Heavy Analytics'
        ELSE 'General Purpose'
    END AS workload_category
FROM query_stats
GROUP BY warehouse_name, user_name
ORDER BY query_count DESC;
```
Use MEDIAN not AVG — outlier queries skew the average and give a misleading picture of typical duration.

Routing logic:
- Short and bursty BI (sub-30s, high frequency) → move to usage-based engine. Real math: $528/month on Snowflake X-Small running continuously vs $5.87/month on per-second billing for the same workload.
- Dev and CI/CD → local DuckDB, zero cloud credits
- Heavy batch ETL, multi-TB → keep on Snowflake, 60s minimum is irrelevant for hour-long jobs
This is the same principle I apply when thinking about orchestration — use the right tool for the job, not the most powerful tool for everything. I wrote about a similar decision process in why I stopped using Snowflake Tasks for orchestration — the short version is that mature orchestration tools give you far more control over exactly this kind of workload routing.

WHEN TO STAY ON SNOWFLAKE

Multi-terabyte batch processing — predictable provisioned compute matters more than idle cost savings when a job runs for hours.

Enterprise governance — complex data masking, RBAC at scale, data residency requirements. Snowflake’s security surface is mature. DuckDB isn’t designed for this.

Already-efficient workloads — if a warehouse runs at high utilisation for 8 hours straight, there’s no idle tax to eliminate. Don’t fix what isn’t broken.

WHAT REAL COST SAVINGS LOOK LIKE
- One SaaS company: 70%+ reduction in warehousing costs after moving to DuckDB-based solution
- Okta: $60,000/month Snowflake spend for threat detection reduced substantially using parallel DuckDB instances
- A data engineering team: 79% immediate reduction in Snowflake BI spend using DuckDB as a caching layer, 7x faster query times
None of these required abandoning Snowflake. They required deciding which workloads actually needed it.

FREQUENTLY ASKED QUESTIONS

Can you query Snowflake data in DuckDB without a Snowflake warehouse running?
Yes — two ways. Iceberg tables via the iceberg extension (no warehouse), or native tables via ADBC. Both require a brief initial connection, but once data is materialised locally, all subsequent queries are free.

What is ADBC and why is it faster than ODBC?
ADBC keeps data in columnar format throughout. ODBC forces columnar → row → columnar conversion. DuckDB benchmarks show ADBC up to 38x faster for transfers.

How much can I realistically save?
For short, high-frequency dashboard queries: 70–90% is consistent across documented cases. The 60-second minimum means a 4-second query costs 15x what it should.

Is DuckDB production-ready?
For single-node analytical workloads under a few terabytes: yes. Multi-user concurrency at scale and enterprise governance: not yet.

Do I need Iceberg?
No. ADBC works with native Snowflake tables. The main friction is private key encoding, which the docs don’t explain well.

Will this work with dbt?
Yes. dbt-duckdb lets you run your full dbt project locally against DuckDB. Pull source data once from Snowflake, develop and test for free, deploy to Snowflake in production only.This eliminates cloud compute costs for the entire development loop. I’ve written about dbt native projects and pipeline patterns if you want more context on how this fits into a Snowflake-first stack.

Related blogs :

→ DuckDB official docs and installation
→ DuckDB ADBC benchmarks
→ Apache Arrow project
→ Snowflake query_history view docs
→ dbt-duckdb adapter on GitHub
→ Greybeam ADBC connection code on GitHub
April 19, 2026
The Problem with dbt Tests Nobody Talks About — They Pass and You Still Ship Bad Data
I’ve been running dbt in production for a while now. And I’ll be honest — there was a phase where I genuinely believed that if my dbt tests were green, I was good. Green means clean, right?

Wrong.

This is the quiet failure mode that nobody in the dbt community writes about loudly enough. Your tests pass. Your CI/CD pipeline goes green. Your DAG runs without errors. And somewhere downstream, an analyst is staring at a revenue number that’s off by 30% and has no idea why.

TL;DR: dbt’s built-in tests (not_null, unique, accepted_values, relationships) validate data structure, not data correctness. Your pipeline goes green and you still ship wrong numbers. This post breaks down exactly why that happens, what the real gaps are, and what custom tests, volume monitoring, and source-layer checks actually fix

Let me walk you through exactly how this happens — because I’ve lived it.

What Are dbt Tests Actually Checking?

Before we get to the failure modes, let’s be precise about what dbt’s generic tests actually do — because I think the confusion starts here.

dbt gives you four built-in generic tests out of the box:
- not_null — checks that a column has no null values
- unique — checks that all values in a column are distinct
- accepted_values — checks that a column only contains values from a predefined list
- relationships — checks referential integrity between two models
These are constraint tests. They validate the shape of your data — grain, nullability, referential integrity. They do not validate whether the values are correct, whether the volume is expected, or whether the business logic in your SQL is actually right.

That distinction is everything.
```
models:
  - name: fct_daily_revenue
    columns:
      - name: transaction_id
        tests:
          - not_null
          - unique
      - name: revenue_amount
        tests:
          - not_null
```
This test suite passes even if every revenue_amount is 100x too large. It passes if your join silently drops 40% of records because a key format changed upstream. It passes if a currency unit changed after a vendor migration and nobody touched the schema.

None of that is a bug in dbt. It’s working exactly as designed. The problem is the mental model we build around it.

The Scenario That Broke Me

We had a pipeline pulling sales transaction data from an API. The dbt model joined it against a product dimension, aggregated daily revenue, and pushed it to a reporting layer. All four generic tests — passing. Every single day.

What was actually happening: the upstream API started returning amounts in a different currency unit after a vendor migration. No schema change. No new nulls. No duplicate keys. Just the values silently shifting by a factor of 100.

Our not_null test on revenue_amount? Passed. Our unique test on transaction_id? Passed. Our downstream revenue dashboard was off by two orders of magnitude for three weeks before an analyst caught it during a QBR.

Three weeks. All green tests. All wrong data.

That’s when I stopped treating dbt tests as a data quality guarantee and started treating them as what they actually are: a contract enforcement layer.

The Three Gaps Nobody Talks About

1. Volume Drift — Records Disappear and Nothing Breaks

If your fct_orders model typically produces 50,000 rows a day and one morning it produces 12,000 — no generic test will catch that. The data that is there is perfectly valid. You just lost 38,000 records somewhere in your pipeline and dbt has no idea.

This is one of the most common real-world pipeline failures I see, and it’s completely invisible to constraint-based tests.

The fix is a custom singular test or a dbt_utils recency/row-count assertion:
```
-- tests/assert_row_count_within_threshold.sql
{% set threshold = 0.2 %}
select 1
from (
  select count(*) as today_count
  from {{ ref('fct_orders') }}
  where order_date = current_date
) today
cross join (
  select avg(daily_count) as avg_count
  from (
    select order_date, count(*) as daily_count
    from {{ ref('fct_orders') }}
    where order_date between current_date - 14 and current_date - 1
    group by order_date
  ) history
) baseline
where abs(today_count - avg_count) / nullif(avg_count, 0) > {{ threshold }}
```
This returns a row — which dbt interprets as a test failure — when today’s row count deviates more than 20% from the 14-day average. Simple, practical, catches real failures. I also wrote about a similar pattern in how dbt integrates natively with Apache Airflow for pipeline orchestration — the combination of orchestration visibility and volume tests gives you a much more honest picture of pipeline health than either alone.

2. Business Logic Correctness — The Math Can Still Be Wrong

dbt tests validate columns in isolation. They don’t validate relationships between columns, or whether the calculations in your model are actually right.

Take something simple:
```
select
  order_id,
  unit_price,
  quantity,
  unit_price * quantity as line_total
from {{ source('orders', 'order_lines') }}
```
You can have not_null on all three columns, accepted_values on quantity to ensure it’s positive — and still ship models where line_total is wrong because unit_price was populated in cents from one source and dollars from another. No generic test catches that unless you explicitly write:
```
-- tests/assert_line_total_matches_components.sql
select *
from {{ ref('fct_order_lines') }}
where abs(line_total - (unit_price * quantity)) > 0.01
```
Writing that test requires you to already know the business rule. Which means data quality at this layer requires domain knowledge, not just dbt knowledge. If you’re using Snowflake, pairing this with Cortex-based automated data quality checks can flag anomalies in derived metrics that pure SQL assertion tests would miss — something I covered in depth when building Snowflake Cortex accelerators for automated data quality.

3. Silent Join Fan-Out and Record Loss

This one has bitten me more than once. A many-to-one join accidentally becomes many-to-many because a dimension table you assumed was unique… wasn’t. Or a left join silently drops records because a key format changed from integer to string somewhere upstream.

The result: your fact table either fans out (double-counting revenue) or silently loses records, and every generic test still passes because the columns that remain are perfectly valid.

The safeguard is writing uniqueness tests on your dimension tables and asserting that your fact-to-dimension join doesn’t increase row count:
```
-- tests/assert_no_join_fanout.sql
with before_join as (
  select count(*) as row_count from {{ ref('fct_orders') }}
),
after_join as (
  select count(*) as row_count
  from {{ ref('fct_orders') }} o
  left join {{ ref('dim_customers') }} c on o.customer_id = c.customer_id
)
select 1
from before_join b
cross join after_join a
where a.row_count > b.row_count
```
What Actually Helps

Write custom singular tests for critical models. Don’t rely only on generic column-level tests for anything that feeds a financial or executive dashboard. If the number matters, test the business rule explicitly.

Add volume and freshness monitoring at source. Whether you use dbt_utils.recency, Elementary, or a hand-rolled SQL assertion — track volume. It’s the cheapest signal you have that something went wrong upstream.
```
sources:
  - name: raw_transactions
    tables:
      - name: transactions
        tests:
          - dbt_utils.recency:
              datepart: hour
              field: created_at
              interval: 3
        columns:
          - name: amount
            tests:
              - dbt_utils.accepted_range:
                  min_value: 0
                  max_value: 1000000
```
Test at source, not just at the model layer. If an upstream format changes, you want the failure at ingestion, not after three transformation layers have already propagated it downstream.

Use dbt_utils and Elementary seriously. The dbt_utils package has range tests, expression tests, and recency checks that fill a lot of the structural gaps. Elementary adds anomaly detection on top of that, which gets you closer to actual data observability rather than just constraint validation.

Review your SQL, not just your CI badge. Every model that feeds a critical metric should have a comment explaining the expected grain, the join logic, and the expected value ranges. Future you — and the next engineer — will thank you when something breaks at 2am.

The Mindset Shift

I had to reframe how I think about dbt tests. They’re not a data quality guarantee. They’re a contract enforcement layer. They ensure your data meets its structural promises. That’s genuinely useful — but it’s not the same as ensuring your data is correct.

Real data quality requires a combination of:
- Structural tests — what dbt gives you natively (constraint validation)
- Business logic tests — custom singular tests you write based on domain knowledge
- Volume and freshness monitoring — dbt_utils, Elementary, or your own row count assertions
- Code review culture — someone actually looks at the SQL, not just whether CI passed
The green checkmark in your pipeline is not permission to stop thinking. It’s permission to look at the next layer of potential failure.

I spent a long time treating dbt tests as a safety net. They’re more like a fence — useful, visible, and completely ineffective against threats that don’t come through the gate.

Frequently Asked Questions

Do dbt tests guarantee data quality?

No. dbt’s built-in generic tests — not_null, unique, accepted_values, relationships — validate structural constraints on your data. They confirm that a column has no nulls, that keys are unique, or that values fall within an expected set. They do not verify whether the actual values are correct, whether business logic in your SQL is right, or whether record volumes are within expected ranges. For genuine data quality coverage, you need custom singular tests, volume monitoring, and source-layer assertions alongside dbt’s native tests.

What is the difference between dbt generic tests and singular tests?

Generic tests in dbt are reusable, schema-defined checks applied to columns across multiple models — not_null and unique are the most common. Singular tests are standalone SQL queries that you write specifically for a model or business rule: they return rows when something is wrong and pass when they return no rows. Singular tests are where you validate business logic — things like “line_total should always equal unit_price × quantity” or “today’s row count should be within 20% of the 14-day average.” Both types live in your tests/ directory and run with dbt test.

Can dbt catch silent record loss in joins?

Not automatically. If a join accidentally drops records — due to a key format change, a null key, or a mismatched data type — dbt’s generic tests won’t flag it unless you’ve explicitly written a test to assert row count consistency before and after the join. This is one of the most common silent failure modes in production dbt pipelines. Writing a custom singular test that compares pre- and post-join row counts is the most reliable way to catch it.

How do I monitor row count changes in dbt?

There are a few approaches. The dbt_utils package includes a recency test for freshness monitoring. For volume, you can write a custom singular test that compares today’s row count against a rolling average from the past 14 days — any deviation beyond a threshold (say 20%) triggers a failure. For more automated anomaly detection across all your models, Elementary integrates directly with dbt and adds statistical monitoring without requiring you to write individual volume tests for every model.

What is the best way to test business logic in dbt?

Write singular tests that encode the business rule explicitly in SQL. For example, if your model calculates revenue = quantity × unit_price, write a test that queries the model and returns rows where abs(revenue - (quantity * unit_price)) > 0.01. If there are cross-column invariants — like a refund amount should never exceed the original transaction amount — write that as a test too. The key insight is that these tests require domain knowledge: you need to know what correct looks like before you can assert it. That’s a conversation between data engineers and the business teams who own the metrics.

Does dbt have built-in anomaly detection?

dbt itself does not include statistical anomaly detection. The core framework focuses on constraint-based testing. For anomaly detection — flagging unexpected spikes, drops, or distribution shifts in your data — you need either the Elementary package, which sits on top of dbt and adds automated monitoring, or a dedicated data observability platform like Monte Carlo, Soda, or Bigeye. In Snowflake environments specifically, combining dbt with Cortex-based quality checks can add an AI-assisted layer on top of your existing test suite.

The Honest Closing

The reason nobody talks loudly about this is that it’s uncomfortable. We build testing frameworks because they give us confidence. Admitting that green tests can coexist with broken data means admitting that the confidence was partly false.

But I’d rather have that honest conversation in a blog post than explain to a VP why the quarterly revenue numbers were wrong — and then pull up a CI pipeline that was green the whole time.

Write the custom tests. Monitor the volumes. Test the business rules. Trust the process, not just the color of the badge.

Related reading from the blog:
April 17, 2026

It’s Not AI You Should Worry About—It’s Automation

I still remember the afternoon I burned four hours debugging a production pipeline — convinced the problem was in the model logic — only to find the real culprit was a manual data prep step where someone had quietly introduced a column name inconsistency. No alerts. No schema validation. Just silent failure downstream.

That incident changed how I think about data engineering. The problem wasn’t the AI model. The problem was that we’d automated the interesting parts and left the boring, error-prone parts to humans.

I’ve spent four years building and maintaining data pipelines — part of a 10-person team processing millions of records at varying frequencies. Here’s what I’ve learned about automation in data engineering: it isn’t about replacing engineers, it’s about removing the conditions where human error is inevitable.

TL;DR

Automation in data engineering is about removing manual, error-prone steps — not just scheduling jobs
AI genuinely helps in ETL for anomaly detection and transformation logic, but it doesn’t replace pipeline architecture
Robust testing and CI/CD are the most underrated investments in pipeline reliability
DataOps is the cultural and operational layer that makes automation sustainable

Why Reliable Data Pipelines Are a Business Problem, Not Just a Technical One

A data pipeline that fails silently is worse than one that fails loudly. When records go missing or get duplicated without anyone noticing, downstream reports become unreliable — and the teams consuming that data stop trusting it. Once trust breaks, people start maintaining their own spreadsheets, which creates more data problems.

In my experience, most pipeline fragility comes from three places:

Manual handoffs between systems (someone exports a CSV, someone else imports it)
Implicit assumptions about schema or data format that nobody documented
Scheduling-based pipelines that run regardless of whether the upstream data is ready

Automating these touch points — not just the processing logic — is what actually improves reliability.

Beyond Scheduling: Event-Based Triggers Are Underused

Most teams start pipeline automation with scheduling: run this DAG at 6am every day. That’s a reasonable starting point, but it creates fragility when upstream systems are delayed, incomplete, or unavailable.

Event-based triggers solve this. Instead of running on a fixed schedule, the pipeline fires when the upstream condition is actually met — a new file lands, a table row count crosses a threshold, an API returns a success status.

Here’s a simple example using Apache Airflow’s HttpSensor to wait for an upstream API to signal readiness before proceeding:

from airflow import DAG
from airflow.operators.bash import BashOperator
from airflow.sensors.http_sensor import HttpSensor
from datetime import datetime, timedelta

dag = DAG(
    'event_based_trigger',
    default_args={
        'owner': 'airflow',
        'depends_on_past': False,
        'start_date': datetime(2024, 1, 1),
        'retries': 2,
        'retry_delay': timedelta(minutes=5),
    },
    schedule_interval=timedelta(days=1),
)

wait_for_api = HttpSensor(
    task_id='wait_for_upstream_api',
    method='GET',
    http_conn_id='upstream_api',
    endpoint='/api/data-ready',
    response_check=lambda response: response.json().get('status') == 'ready',
    poke_interval=60,
    timeout=600,
    dag=dag,
)

process_data = BashOperator(
    task_id='process_data',
    bash_command='python /opt/scripts/process_records.py',
    dag=dag,
)

wait_for_api >> process_data

This pattern means your pipeline won’t process stale or incomplete data just because the clock hit 6am. That single change has prevented more production incidents on my team than any other automation improvement.

Where AI Actually Fits in Data Engineering

The honest answer is that AI augments specific parts of the ETL process — it doesn’t change the fundamentals of building reliable pipelines.

Where I’ve seen AI add genuine value:

Anomaly detection in incoming data — catching unexpected distributions or null rate spikes before they propagate
Schema drift detection — flagging when source columns change in ways that will break transformations
Natural language to SQL — useful for ad hoc queries, not for production pipeline logic
Log summarization — when pipeline failures produce walls of logs, AI can surface the root cause faster

Where AI doesn’t help as much as vendors claim:

Replacing pipeline orchestration logic
Making architectural decisions about partitioning, incremental loads, or SCD handling
Writing production-grade dbt models without human review

Here’s a simple automated data quality check you can add to any pipeline using pandas before records move downstream:

import pandas as pd

def validate_records(filepath: str) -> pd.DataFrame:
    df = pd.read_csv(filepath)

    original_count = len(df)
    df = df.drop_duplicates()
    duplicate_count = original_count - len(df)

    null_rates = df.isnull().mean()
    high_null_cols = null_rates[null_rates > 0.1].index.tolist()

    if duplicate_count > 0:
        print(f"Warning: Removed {duplicate_count} duplicate rows")

    if high_null_cols:
        raise ValueError(f"High null rate in columns: {high_null_cols}")

    return df

This isn’t AI — it’s automation. But it’s exactly the kind of check that catches problems before they reach your warehouse.

Approach	Best For	Watch Out For
AI-enhanced anomaly detection	Catching statistical drift in high-volume pipelines	Needs baseline period to calibrate; false positives early on
Rule-based data quality checks	Schema validation, null checks, referential integrity	Requires manual updates when business rules change
Traditional scheduled ETL	Predictable, low-complexity sources	Fragile when upstream systems are delayed or unavailable
Event-triggered ETL	Reducing unnecessary runs, improving data freshness	More complex to set up; requires reliable event signaling

Common Automation Mistakes I’ve Made (and Watched Others Make)

Monitoring as an afterthought. I once shipped an Airflow pipeline with zero alerting. It ran daily for three weeks before anyone noticed a misconfigured DAG was processing the same partition repeatedly. The error message — AirflowException: DAG not found — was buried in logs no one was watching. Now I treat alerting setup as part of the definition of done, not a follow-up ticket.

Confusing “automated” with “tested.” You can automate a broken process. Automation without test coverage just means your broken process runs faster and at scale.

Too many retries masking real failures. Setting retries=5 is not a reliability strategy. It’s a way to delay your on-call notification by 25 minutes. Retries should handle transient infrastructure issues, not cover up data problems.

No idempotency. If your pipeline fails halfway through and re-runs from the beginning, it should produce the same result — not double-insert records. Building idempotent pipelines takes more upfront effort but prevents some of the worst production incidents I’ve seen.

Testing and CI/CD for Data Pipelines

Data pipelines deserve the same testing rigor as application code. That means:

Unit tests for transformation logic (test your dbt macros and Python functions in isolation)
Integration tests that run a pipeline end-to-end against a sample dataset
Schema validation tests that fail loudly if column types or names change unexpectedly
CI checks that run on every pull request before code reaches production

On one project, we implemented GitLab CI/CD to run dbt tests and a full DAG parse check on every merge request. The DAG parse check alone caught misconfigured imports that would have failed silently at runtime. The time investment in setting that up paid back within the first month.

A simple GitLab CI stage for dbt testing looks like this:

test_dbt_models:
  stage: test
  script:
    - dbt deps
    - dbt compile --profiles-dir ./profiles
    - dbt test --profiles-dir ./profiles
  only:
    - merge_requests

The principle is straightforward: treat your pipeline code as production software. Version control it, test it, and don’t deploy it manually.

DataOps: The Operational Layer People Skip

DataOps is a word that gets used loosely, but the core idea is useful: apply the same collaboration, automation, and continuous delivery practices from software engineering to data workflows.

In practice, what this meant for my team:

All DAGs and dbt models live in Git, with PR reviews before anything merges
A staging environment mirrors production so we can test pipeline changes before they touch live data
Incident retrospectives are documented, and recurring failure patterns get automated checks to prevent recurrence
Data quality issues are tracked like bugs, not dismissed as “one-off data problems”

The shift from “we schedule jobs and monitor them loosely” to “we treat pipelines as production software” is what DataOps actually means. It’s not a tool purchase — it’s a way of working.

When to Automate and When Not To

Not everything should be automated on day one. Here’s how I think about prioritization:

Automate immediately:

Data validation and quality checks
Alerting and failure notifications
Idempotent full or incremental loads on stable sources
Schema change detection

Automate after you understand the pattern:

Complex transformation logic (understand it manually first)
Backfill processes (get the logic right before you automate it)

Be careful automating:

Anything that writes to production without a dry-run option
Business rule changes that need stakeholder input
Pipeline logic that varies significantly by source

The goal of automation in data engineering isn’t to remove humans from the process — it’s to remove humans from the steps where they’re most likely to make mistakes.

Frequently Asked Questions

What does automation in data engineering actually mean? Automation in data engineering means replacing manual, repetitive steps in your data pipeline — things like file transfers, data quality checks, schema validation, and deployment — with code and tooling that runs reliably without human intervention. It goes beyond just scheduling jobs to include monitoring, alerting, testing, and CI/CD.

Which tasks in a data pipeline should I automate first? Start with data validation checks (null rates, duplicate detection, schema consistency) and alerting. These have the highest return on reliability investment because they catch problems early and ensure failures surface loudly rather than silently.

Can AI replace data engineers? No. AI can automate specific tasks — like anomaly detection, log summarization, or schema drift alerts — but building reliable pipelines requires architectural decisions, business context, and judgment that AI tools don’t provide. AI augments the work; it doesn’t replace it.

What’s the difference between DataOps and traditional data engineering? Traditional data engineering focuses on building pipelines. DataOps adds the operational layer: version control, CI/CD, testing standards, monitoring, and incident management. It’s the difference between writing code and running it reliably in production.

How do I make my Airflow pipelines more reliable? Use event-based triggers instead of pure scheduling where possible, implement idempotent tasks so re-runs are safe, add schema validation steps before transformations, set up alerting on task failure (not just DAG-level), and build a proper staging environment to test DAG changes before production.

April 16, 2026

Why I Stopped Using Snowflake Tasks for Orchestration

I want to be clear about something before I say anything critical: Snowflake Tasks are genuinely good. I used them for months. I recommended them to people. I wrote internal documentation about how to set them up.

And then, slowly, quietly, I stopped reaching for them — and started reaching for Airflow instead.

This isn’t a hit piece on Snowflake Tasks. It’s an honest look at where they work beautifully, where they start to crack, and the specific moment I realised I was fighting the tool instead of using it. If you’re in that same place right now — Tasks running fine on paper, increasingly painful in practice — this is for you.

Why I Started With Tasks in the First Place

The pitch for Snowflake Tasks is genuinely compelling: schedule and orchestrate your data pipelines without leaving Snowflake. No extra infrastructure. No Airflow server to maintain. No Docker containers. No YAML config files. Just SQL.

For a solo data engineer or a small team running straightforward ELT pipelines that live entirely inside Snowflake, this is actually a great deal. You write a Task, you chain a few of them together, you set a cron schedule on the root, and the whole thing runs on serverless compute that Snowflake manages for you. Clean. Simple. Zero ops overhead.

I built my first Task tree on a customer dimension pipeline — about 6 tasks chained together to handle raw landing, staging, SCD2 merge, and a downstream mart refresh. It worked perfectly. I was genuinely impressed.

So I built more of them. And that’s where things started to get interesting.

The First Sign Something Was Off

The thing about Snowflake Tasks is that they look fine at small scale. Five tasks. Eight tasks. Even fifteen tasks chained together works reasonably well.

The cracks start showing when your pipelines grow, when requirements get more complex, and when something goes wrong at 7am and you need to figure out what happened and why.

My first real frustration was observability. When a Task fails, Snowflake logs it — but finding that log, understanding the full execution context, and connecting it to what came before and after requires digging through TASK_HISTORY in ACCOUNT_USAGE or calling INFORMATION_SCHEMA.TASK_HISTORY(). There’s no single screen that shows you, visually, what ran, what passed, what failed, and what the downstream impact was.

Compare that to opening the Airflow UI, clicking into a DAG run, and seeing every task coloured green or red with full logs one click away. The difference in time-to-diagnosis is not small. I once spent 40 minutes reconstructing a failed task tree execution from TASK_HISTORY queries that would have taken me 3 minutes in Airflow.

-- How you debug a failed Snowflake Task
SELECT
    name,
    state,
    scheduled_time,
    completed_time,
    error_code,
    error_message
FROM TABLE(INFORMATION_SCHEMA.TASK_HISTORY(
    SCHEDULED_TIME_RANGE_START => DATEADD('hour', -6, CURRENT_TIMESTAMP()),
    RESULT_LIMIT => 100
))
WHERE name ILIKE '%customer_dim%'
ORDER BY scheduled_time DESC;

That query works. But it’s not a dashboard. It’s archaeology.

The Retry Problem

This one hurt me in production.

Snowflake Tasks have basic retry configuration — you can set SUSPEND_TASK_AFTER_NUM_FAILURES to pause a task after repeated failures, which is useful. But what you can’t do natively is retry a specific failed task in the middle of a tree and resume from that point forward.

If Task 6 in a 10-task chain fails, you fix the problem, and you want to re-run from Task 6 onwards — you’re doing it manually. You can resume the root task, but it re-runs everything from the beginning on the next scheduled tick. Or you run Task 6’s SQL manually, then manually kick off Task 7, Task 8… you see where this is going.

In Airflow, you right-click the failed task node, click “Clear”, and it re-runs that task and everything downstream. That’s it. One click. No manual intervention, no risk of accidentally re-running something upstream that already completed correctly and shouldn’t run twice.

For pipelines with expensive upstream tasks — large MERGE operations, heavy aggregations — re-running from the beginning when only a downstream step failed is both wasteful and risky. Wasteful because you’re burning compute credits on work already done. Risky because some operations are not safely idempotent and running them twice produces wrong results.

The Conditional Logic Wall

Here’s the limitation that finally pushed me to switch.

My pipelines started needing branching logic. Specifically: run the full pipeline on weekdays, run a lighter version on weekends. Or: if the row count from the previous step is zero, skip the downstream merge and send an alert instead of running an empty MERGE that silently succeeds.

In Airflow, this is a BranchPythonOperator. Three lines of Python. Clean, explicit, version-controlled.

In Snowflake Tasks, this requires workarounds. You can use a stored procedure with SYSTEM$TASK_DEPENDENTS_ENABLE logic, or try to simulate branching with conditional stored procedures that check a flag and decide whether to execute. It works — technically — but it’s brittle, hard to read, and the logic is buried inside a stored procedure rather than visible in the orchestration layer where it belongs.

Snowflake Tasks can only execute SQL statements and stored procedures. For more complex logic in Python, Java, or other languages, external schedulers are required. Flexera

When your pipeline logic is entirely SQL, Tasks are fine. The moment you need to make an orchestration decision based on runtime data — not just “did this succeed or fail” but “what did this return, and what should I do about it” — you’re working against the grain.

The Scale Limit Nobody Mentions

There is a hard limit of 1,000 Tasks per data pipeline. For very large implementations this is an issue and you need to split out the data pipeline into multiple separate data pipelines as a workaround. Sonra

Most teams won’t hit 1,000 tasks. But if you’re building a platform for multiple teams — separate pipelines per business domain, each with their own task trees — you will eventually bump into governance and management complexity that a 1,000-task-per-pipeline limit doesn’t help with.

More practically: managing dozens of separate task trees, each owned by a different role, with different schedules, different failure behaviours, and no unified view across all of them — is hard. There’s no Snowflake-native equivalent of Airflow’s DAG list view where you can see all pipelines, their last run status, and their next scheduled run in one place.

What I Use Instead — And Why

I switched to Airflow with the SQLExecuteQueryOperator for Snowflake, and I’ve written about this setup in depth in my post How I Wired Snowflake’s Native dbt Projects to Airflow. The short version of why it works better for me:

Airflow owns orchestration. Snowflake owns execution. That’s the right division of responsibility. Airflow is purpose-built for DAG management, dependency handling, retries, branching, alerting, and observability. Snowflake is purpose-built for data processing at scale. Letting each tool do what it’s best at — instead of asking Snowflake Tasks to be a general-purpose orchestrator — is the cleaner architecture.

Here’s the pattern I use for a typical pipeline:

from airflow import DAG
from airflow.providers.snowflake.operators.snowflake import SQLExecuteQueryOperator
from airflow.operators.python import BranchPythonOperator
from datetime import datetime, timedelta

with DAG(
    dag_id='customer_dimension_pipeline',
    schedule_interval='0 6 * * *',
    start_date=datetime(2024, 1, 1),
    catchup=False,
) as dag:

    load_raw = SQLExecuteQueryOperator(
        task_id='load_raw_customers',
        conn_id='snowflake_analytics',
        sql="CALL raw.sp_load_customers();",
    )

    validate_raw = SQLExecuteQueryOperator(
        task_id='validate_raw_row_count',
        conn_id='snowflake_analytics',
        sql="""
            SELECT CASE
                WHEN COUNT(*) = 0 THEN 1/0  -- Forces task failure if no rows
                ELSE COUNT(*)
            END FROM raw.customers_staging
            WHERE load_date = CURRENT_DATE();
        """,
    )

    run_scd2_merge = SQLExecuteQueryOperator(
        task_id='run_scd2_merge',
        conn_id='snowflake_analytics',
        sql="CALL transforms.sp_customer_scd2_merge();",
    )

    refresh_mart = SQLExecuteQueryOperator(
        task_id='refresh_customer_mart',
        conn_id='snowflake_analytics',
        sql="CALL marts.sp_refresh_customer_summary();",
    )

    load_raw >> validate_raw >> run_scd2_merge >> refresh_mart

If validate_raw fails because there are zero rows, the pipeline stops. run_scd2_merge never runs. I get an Airflow alert. I can clear and rerun just validate_raw and everything downstream once the issue is fixed — without touching load_raw again.

That conditional validation step alone — stopping a pipeline when upstream data is missing — was nearly impossible to implement cleanly with Tasks. With Airflow it’s a forced division by zero in the validation SQL. Ugly, but effective. There are cleaner ways with ShortCircuitOperator too.

To Be Fair: When Snowflake Tasks Are Still the Right Choice

I don’t want this to read as “never use Tasks.” That’s not what I’m saying.

Tasks are still my first choice for:

Micro-refresh patterns. A Task + Stream combination for near-real-time SCD2 updates — triggering only when the stream has data — is elegant and genuinely hard to replicate cleanly in Airflow. I covered exactly this pattern in my post Snowflake Streams and Tasks for SCD2 — How I Actually Use Them.

Simple scheduled SQL. A single SQL statement that needs to run every 30 minutes with no dependencies? Task all the way. Zero ops overhead for maximum simplicity.

Snowflake-only pipelines with no external context. If your pipeline never needs to know anything about the outside world — no API calls, no file system checks, no cross-system dependencies — Tasks keep everything in one place.

Small teams with no existing orchestration infrastructure. If you’re a team of two data engineers and setting up Airflow feels like overkill, Tasks get you 80% of the way there with 10% of the setup cost.

The honest decision framework:

Scenario	Use
Simple scheduled SQL, no branching	Snowflake Tasks
Stream-triggered incremental loads	Snowflake Tasks + Streams
Multi-step pipeline with retry requirements	Airflow
Conditional branching based on row counts	Airflow
Pipelines touching external systems	Airflow
Cross-team pipelines needing unified observability	Airflow
Large task trees (50+ steps)	Airflow

The Honest Summary

I didn’t stop using Snowflake Tasks because they’re bad. I stopped using them as my primary orchestration layer because that’s not what they’re optimised for — and I was asking them to do a job they weren’t built to do well.

Snowflake Tasks handle most data orchestration patterns, but the real question is who actually plays well with Snowflake as more than just a place to send SQL. Monte Carlo Tasks are Snowflake’s answer to simple scheduling. Airflow is the answer to complex orchestration. Knowing which problem you actually have is the whole game.

If your pipelines are growing, your failure debugging is getting slower, and you’ve found yourself writing stored procedures to simulate branching logic — that’s the sign. That’s the moment I had. The switch to Airflow was a weekend of setup and a week of migration, and I haven’t looked back.

April 12, 2026

Author: Sainath Reddy

TL;DR

What CoCo Actually Is (And What Changed at Summit 2026)

The Data-Native Context Advantage

CoCo Desktop — Schema-Grounded dbt Model Generation

Cloud Agents — Async Scheduled Tasks

The Benchmark Reality Check

Where CoCo Desktop Has Limits

The Comparison You Actually Need

How I’d Actually Use This

When to Evaluate CoCo Desktop

When to Stick With What You Have

What I’d Do Right Now

Frequently Asked Questions

TL;DR

What We Were Running When We Evaluated

The Core Difference Nobody Explains Clearly

What This Means for Local Development

What This Means for Dynamic Workflows

Airflow Version

Prefect Version

Where Airflow Still Wins

Ecosystem Maturity Is a Real Advantage

Airflow 3.0 Closes the Gap

Scale Is Proven

Where Prefect Genuinely Wins

Operational Overhead for Small Teams

Monitoring and Observability Out of the Box

ML Pipelines Specifically

The Comparison You Actually Need

What the Migration Actually Looks Like

What is Airflow and How Does it Compare to Prefect?

How to Choose Between Airflow and Prefect for Your Data Workflow

Why Does My Team Need a Workflow Management Tool Like Airflow or Prefect?

What are the Key Features of Airflow and Prefect?

How Do I Get Started with Airflow or Prefect?

When to Choose Airflow

When to Choose Prefect

What I’d Do Differently

Frequently Asked Questions

Q: What’s the main difference between Airflow and Prefect?

Q: Can I use Airflow and Prefect together in my data pipeline?

Q: How do I decide which tool is best for my team’s specific use case?

Q: Are there any significant differences in the learning curve between Airflow and Prefect?

Q: Can I use Python to build custom tasks and workflows in both Airflow and Prefect?

Partition Management Was Getting Messy

Hidden Partitioning Changed How We Design Tables

THE MIGRATION: WHAT IT ACTUALLY COST US

Downstream dependency mapping.

Metadata catalog updates.

Frequently Asked Questions

What is the main difference between Delta Lake and Apache Iceberg?

Is Apache Iceberg better than Delta Lake?

Can Snowflake read Delta Lake tables?

How hard is it to migrate from Delta Lake to Apache Iceberg?

Does dbt support Apache Iceberg?

What is hidden partitioning in Apache Iceberg?

The File You Should Have Created on Day One

The Three-Level Hierarchy (and Where Most Teams Go Wrong)

User-level

Project-level

Directory-level

Stop Sending Messages. Start Setting Mode.

direct execution

Built-In Tools You’re Probably Not Using Correctly

Grep vs Glob

Read, Write, Edit

The /memory command

Custom Slash Commands: Stop Prompting the Same Thing Twice

Skills vs CLAUDE.md: Context Engineering Done Right

Skills

Path-Specific Rules: The Feature Teams Discover and Never Go Back From

CI/CD Integration: Automating the Work That Shouldn’t Be Manual

What a Power User Day Actually Looks Like

The Bottom Line

FAQ

What is CLAUDE.md and why does it matter?

What’s the difference between plan mode and direct execution in Claude Code?

How do path-specific rules work in Claude Code?

What is the -p flag in Claude Code?

The `/memory` command