Author: Sainath Reddy

  • Stop Spinning Up Spark clusters for 50GB Datasets

    Stop Spinning Up Spark clusters for 50GB Datasets

    Your team has a 200GB Parquet file on S3. Someone suggests running the analysis in Spark. You spin up a four-node cluster, configure executors, tune shuffle partitions, wait three minutes for the cluster to initialize, wait fourteen minutes for the job to run, tear the cluster down, and get back a number.

    The same query in DuckDB runs on a single VM in four minutes, costs one-twentieth as much, and requires zero cluster management. You didn’t need distributed computing. You needed a fast query engine — and you reached for a freight train when a Ferrari would have done the job in a quarter of the time.

    This is the most expensive habit in modern data engineering, and it’s happening in thousands of production pipelines right now. Not because engineers are incompetent. Because the “big data playbook” — spin up Spark, process everything, shut down cluster — was written when cloud VMs had 8GB of RAM. A $300/month VM in 2026 has 128GB of RAM and NVMe SSDs that can sustain 3GB/s reads. The old rule — “data doesn’t fit in memory, use a cluster” — is eroding fast. And DuckDB is the reason.

    TL;DR

    → DuckDB is an embedded, in-process, columnar OLAP database. No server. No cluster. No JVM. Install in one `pip install duckdb`. Query CSV, Parquet, JSON on S3 with standard SQL.

    → For 50GB–1TB OLAP workloads on Parquet, DuckDB is typically 3–10x faster than Spark and 10–20x cheaper because it eliminates network shuffle, JVM overhead, and cluster management overhead.

    → Real benchmark: 500GB Parquet (stock trades, time-series aggregation + groupby). Spark on a 4-node cluster: 14 minutes. DuckDB on a single 16-core, 128GB VM: ~4 minutes. Cost ratio: 1:20.

    → DuckDB wins: SQL-first OLAP on Parquet/CSV/JSON, data that fits on one machine (up to ~1TB), CI/testing pipelines, local development, cost-sensitive workloads.

    → Spark still wins: petabyte-scale distributed ETL, Structured Streaming for real-time pipelines, MLlib integration, cross-node joins on truly massive datasets, fault tolerance across hundreds of nodes.

    → The practical hybrid: DuckDB for local dev and CI (zero startup time vs Spark’s 3-minute init); Spark for production TB+ workloads. Most teams using Spark everywhere could do 80% of their work on DuckDB.

    → Polars is in this conversation too: Rust-based DataFrame API, great for Python-first teams who don’t want SQL. DuckDB for SQL, Polars for code. They’re complementary, not competitive.

    → MotherDuck extends DuckDB to a managed cloud warehouse — multi-user, persistent storage, connectors — for teams that outgrow single-node but don’t want Spark’s complexity.

    What DuckDB actually is (and isn’t)

    DuckDB is an OLAP (analytical) database engine that runs inside your process. Not a server. Not a service. An embedded library, like SQLite, except built from scratch for analytical queries instead of transactional ones. You pip install duckdb and start querying. No cluster to manage. No JVM. No configuration files. No driver program. No shuffle partitions to tune.

    Under the hood, DuckDB uses vectorized execution: it processes data in columnar chunks, exploiting CPU SIMD instructions to handle hundreds of rows per clock cycle. It reads Parquet files with column pruning and predicate pushdown — it doesn’t load the whole file into memory, it skips the pages and row groups it doesn’t need. The result is query performance that competes with Spark on single-machine workloads at a fraction of the infrastructure cost.

    What DuckDB is not: a distributed system. It runs on one machine. If your data genuinely cannot fit on one machine or you need streaming, DuckDB is not your answer. But here’s the part of the conversation that’s rarely said clearly: most data engineering workloads in production are not distributed workloads. They’re workloads that teams are running on distributed infrastructure out of habit, convention, or because that’s what the senior engineer learned in 2019.

    The benchmark that changes how you think about this

    Real benchmark numbers: DuckDB eliminates cluster overhead, JVM serialization, and network shuffle. Wins by 3–10x on OLAP queries up to ~1TB. Cost difference is even larger than time difference.

    The numbers that matter come from a controlled test on a 500GB Parquet dataset of stock trade records: time-series aggregation with a multi-column groupby, the kind of query that sits at the core of most analytical pipelines.

    Spark on a 4-node cluster: 14 minutes end-to-end (including cluster init and tear-down overhead), at cluster-runtime node pricing. DuckDB on a single 16-core, 128GB RAM VM: ~4 minutes, no init overhead, running as a single process. Cost ratio: roughly 20:1 in DuckDB’s favor.

    Why does DuckDB win? Spark pays for distributed resilience even when you don’t need it. It shuffles data across network to prepare for cross-node joins that will never happen because the data fits on one machine. It serializes and deserializes through JVM objects. It manages a driver program and executor lifecycle. All of that overhead is real cost — not just money but latency. DuckDB simply reads columnar Parquet from local NVMe, pushes predicates down to skip file sections, and runs vectorized aggregation in CPU cache. No network. No JVM. No shuffle.

    For smaller queries: a grouped aggregation benchmark (sales by region on 10M rows) took DuckDB 2.5 seconds, Spark in local mode 8 seconds. A join on two 20M-row tables: DuckDB under 5 seconds, Spark 15 seconds. Important caveat: these are single-machine comparisons. At true petabyte scale, Spark’s distributed architecture wins because DuckDB simply runs out of hardware. But most teams never get to petabyte scale, and the ones who believe they have are often running 200GB datasets on Spark clusters because nobody revisited the architecture decision from three years ago.

    The cost math most teams never do

    Assume you have a 300GB daily analytics pipeline running on Spark. A modest cluster: 4 worker nodes, each 8 cores, 32GB RAM. You run it twice a day. On AWS, that’s roughly $0.30/node-hour, four nodes, maybe 45 minutes per run. That’s $0.90/run, $1.80/day, $657/year. Sounds manageable.

    Now add: the 15 minutes of Spark startup overhead per run ($0.30 wasted per run), the 20% of engineer time spent debugging shuffle OOM errors and executor failures, the CI runs that take 12 minutes instead of 2 because you’re testing against a Spark local context instead of DuckDB.

    The DuckDB alternative: a single c6i.4xlarge instance (16 cores, 32GB RAM), on-demand at $0.68/hour. Run it twice a day, average 8 minutes per run. That’s $0.18/day, $66/year. Plus near-zero maintenance overhead. For a 300GB pipeline, you’re looking at $591/year saved, plus meaningfully less engineer time.

    For larger teams running many such pipelines, multiply accordingly. The savings aren’t theoretical.

    Where DuckDB actually fits in your stack

    The decision is simpler than it looks: does your data fit on one machine? If yes, DuckDB is almost always the right choice. If not, Spark. The hard part is being honest about your actual data size.

    The practical split is cleaner than most discussions make it sound:

    Use DuckDB when: your data fits on one machine (roughly up to 1TB with modern hardware), the workload is SQL-first analytical queries, you’re building CI/testing pipelines (DuckDB starts in milliseconds; Spark in minutes), you’re doing local development and iteration, or you’re running cost-sensitive batch workloads where cluster overhead is pure waste.

    Use Spark when: your data physically cannot fit on one machine or needs distributed partitioning, you’re building streaming pipelines with Structured Streaming and need exactly-once semantics, you need MLlib for distributed model training, you have genuinely petabyte-scale joins that require cross-node shuffles, or you need fault tolerance across hundreds of nodes where a single node failure would be catastrophic.

    The hybrid that most teams are converging on: DuckDB in local development and CI (the “inner loop”), Spark in production for workloads that actually need distribution. This is the pattern Zach Wilson has written about: DuckDB for fast local testing and EDA, Spark for the production pipelines processing billions of events per hour. The tools aren’t competing for the same role — they’re occupying different rungs of the same ladder.

    DuckDB’s SQL is genuinely better to write

    Benchmark numbers aside, the developer experience gap is significant. DuckDB has shipped SQL extensions that most engineers discover and then can’t go back from.

    EXCLUDE lets you select all columns except a few: SELECT * EXCLUDE (internal_id, created_at) FROM orders. No more writing out 40 column names. COLUMNS with regex lets you pattern-match columns: SELECT COLUMNS('amount.*') FROM ordersQUALIFY filters on window function results without a subquery. Function chaining — first_name.lower().trim() — reads like Python. These aren’t gimmicks; they’re hours of saved typing at scale.

    DuckDB also queries files directly without loading them: SELECT * FROM 's3://my-bucket/data/*.parquet' WHERE event_date = '2026-06-01'. No ETL to load into a table first. No Spark session to initialize. The file is the table.

    The gotchas nobody warns you about

    DuckDB’s concurrency model is not Postgres. DuckDB supports multiple readers, but only one writer at a time. If you’re building a production system where multiple processes need to write simultaneously, you’ll hit locking issues quickly. MotherDuck solves some of this, but the base DuckDB model is single-writer. Don’t architect a high-write-concurrency system on raw DuckDB without understanding this.

    Memory is managed, but you can still OOM. DuckDB’s query engine is smart about memory, using streaming execution to avoid materializing entire result sets. But complex multi-join queries with many intermediate results can still consume more RAM than your VM has. Size your VM with headroom — if your dataset is 100GB, don’t run it on a 128GB instance. Leave 30–40% for overhead.

    DuckDB is not a transactional database. It has ACID transactions, but it’s optimized for append-heavy analytical workloads, not OLTP update/delete patterns. Using it as a general-purpose application database is the wrong tool for the job.

    Distributed DuckDB exists but isn’t production-ready at Spark scale. There’s a distributed extension project, but it’s nowhere near Spark’s maturity or fault tolerance. If you’re planning to “scale DuckDB to Spark scale” — that’s not the right mental model. When you outgrow single-node DuckDB, the answer is MotherDuck (managed, serverless) or Spark (distributed, self-managed). Not “distributed DuckDB.”

    The Polars question. If your team writes Python-first data pipelines, Polars is a serious alternative to DuckDB for single-machine workloads. Polars is a Rust-based DataFrame library — think pandas but 10–30x faster with a proper lazy execution model. It doesn’t support SQL natively (though it has SQL-like expressions). The practical split: DuckDB for SQL-first analytical queries; Polars for code-first transformations. Many teams use both: DuckDB to query and load Parquet, Polars to transform the resulting DataFrame. They compose cleanly together.

    When to migrate existing Spark pipelines

    Migrating an existing Spark pipeline to DuckDB isn’t always worth the effort even if DuckDB would be faster. Before migrating, ask three questions:

    Is the Spark pipeline causing operational pain (OOM errors, long startup times, expensive debugging)? Is the dataset under 1TB and not expected to grow past single-node capacity? Does the pipeline use only Spark SQL or DataFrame operations, not Spark-specific features like Structured Streaming or MLlib?

    If all three are yes, the migration is usually a morning’s work: translate PySpark DataFrames to DuckDB SQL, replace S3 Spark readers with DuckDB S3 file queries, run both in parallel for one week, decommission the cluster. Most SQL-based Spark pipelines translate directly because DuckDB’s SQL is a superset of what most teams actually use in Spark SQL.

    If any answer is no, keep Spark for that pipeline and use DuckDB for new workloads below the threshold.

    The one principle

    Match the tool to the actual data size, not the data size you imagine you might have someday. Spark is the right answer for distributed workloads that genuinely cannot fit on one machine. It is not the right answer for a 200GB daily pipeline just because someone wrote the original architecture when “big data” was the thing to say. In 2026, a single cloud VM has enough RAM, CPU, and NVMe storage to handle most analytical pipelines that companies think require distributed computing. DuckDB is the proof of that claim.

    Related reading: DuckDB S3 extension docs · MotherDuck: Managed DuckDB in the cloud · Snowflake Iceberg v3: When to Migrate · dbt Fusion: 30x Faster Parsing · Snowflake Query Execution: What Really Happens

  • Someone renamed a column.Your Pipeline Died.Here’s the fix

    Someone renamed a column.Your Pipeline Died.Here’s the fix

    Someone on the backend team renamed order_total to order_amount. Clean name. Makes total sense for their domain model. They shipped it on a Thursday afternoon. By Friday morning, your revenue dashboard was showing zero. Not wrong numbers. Zero. Because your Snowflake pipeline was still selecting order_total from the events table, and the column simply wasn’t there anymore.

    You found out from a Slack message. From a director. At 9 AM.

    This is the most common production incident in data engineering in 2026, and it’s almost never caused by bad code. It’s caused by the absence of a formal agreement between the team producing data and the team consuming it. That agreement has a name: a data contract. And most data teams still don’t have one.

    The excuse is usually some version of “we move too fast.” The reality is that the teams who move fastest are the ones with contracts, because they stop discovering breaking changes from directors on Friday mornings and start catching them in CI on Thursday afternoons, before anything ships.

    TL;DR

    → A data contract is a formal specification — schema, semantics, SLAs, ownership — between a data producer and its consumers. Not documentation. Enforcement.

    → Most data incidents don’t start with missing data or broken code. They start with a well-intentioned upstream change that silently invalidated an assumption someone downstream was relying on.

    → Contracts have three parts: schema (structure and types), semantics (what fields actually mean), and SLAs (freshness, completeness, availability). Schema-only contracts miss most real breakages.

    → The dual-write pattern is the only safe migration path for breaking changes: keep old field + add new field → both populated during transition → deprecation notice with a hard date → removal at v2. Each phase takes at minimum 30 days. Skipping phases causes incidents.

    → 90 days minimum notice for breaking changes. Data pipelines have long release cycles; consumers need time to update downstream logic, tests, and dashboards.

    → A contract not enforced in CI is just documentation. The ODCS (Open Data Contract Standard) YAML spec plus `datacontract-cli` gives you executable, version-controlled contracts in about 30 minutes per dataset.

    → dbt integration: map contract checks to dbt tests. Require a version bump plus consumer sign-off on breaking changes before merge. After one month of this, most teams report significantly fewer schema surprises.

    → The worst gotcha: contracts that only cover schema, not semantics. A field that changes meaning without changing type is undetectable to automated checks — and it’s how revenue figures silently drift for weeks.

    Why schemas break and who owns the blame

    Schema evolution sits between two teams that don’t talk to each other on the same cadence. The producer team — usually a backend or platform engineering team — is shipping product features, often weekly, and treats every field they emit as their own. The consumer team — your data engineering team — is running pipelines that depend on those fields staying stable, and finds out about breaking changes the same way archaeologists find ruins: by digging through wreckage.

    The producer isn’t wrong for evolving their schema. The consumer isn’t wrong for depending on it. The incident happens because there was no shared definition of what “a safe change” means, no process for communicating it, and no tooling to enforce the agreement. The blame falls on the process, not the person. Which means the fix is a process change, not a person change.

    Schema evolution is the load-bearing problem in data engineering in 2026, and it’s the problem most teams handle the worst. The good teams treat upstream schemas as contracts and run checks against those contracts on every pipeline run. The teams that lose stakeholder trust treat upstream schemas as suggestions and find out about every breaking change from a Slack message that starts “hey, the dashboard looks weird.”

    That Slack message is always sent on a Friday. It is always sent to a director.

    What a data contract actually contains

    The mistake most teams make when they start with data contracts is writing schema-only contracts. Field names, data types, nullability. It feels rigorous. It catches a specific class of errors — column removed, type changed — but misses most real incidents.

    Real breakages happen at the semantics layer. The producer changes order_total from gross to net revenue. Same field name. Same FLOAT type. No schema violation. But your revenue dashboard is now off by 23%, silently, because the number means something different than it did last week. A schema validator cannot catch this. Only a semantic contract can — one that documents what a field means, how it should be used, and what constitutes a valid business interpretation of its values.

    A complete data contract has three layers. Schema: field names, data types, nullability, constraints (no negative values in a price field, for example). Semantics: what each field means in business terms, how it maps to domain concepts, what transformations are applied before it reaches the consumer. SLAs: freshness guarantees (this dataset is refreshed within 15 minutes of source update), completeness thresholds (at least 99.5% of expected rows must be present), availability targets, and a named owner with actual contact information — not “data team.”

    The Open Data Contract Standard and the YAML spec

    The good news for teams starting in 2026 is that there’s a growing standard: ODCS (Open Data Contract Standard), a YAML-based specification that defines schema, quality rules, SLAs, and ownership in a single document. It’s human-readable, version-controllable in git, and machine-parseable by tools like `datacontract-cli`, which can validate contracts, run compatibility checks, and generate reports.

    A minimal ODCS contract for an orders dataset looks like:

    dataContractSpecification: 0.9.3
    id: orders-v1
    info:
    title: Orders
    version: 1.0.0
    owner: [email protected]
    servers:
    production:
    type: snowflake
    database: PROD_DB
    schema: PUBLIC
    table: orders
    models:
    orders:
    fields:
    order_id:
    type: string
    required: true
    description: Unique identifier for the order
    order_amount:
    type: number
    required: true
    description: Net revenue after discounts and returns, in USD
    minimum: 0
    created_at:
    type: timestamp
    required: true
    servicelevels:
    freshness:
    description: Data refreshed within 15 minutes of source update
    threshold: PT15M
    completeness:
    description: At least 99.5% of expected rows present
    threshold: "99.5%"

    This is not documentation theater. This YAML file is executable. `datacontract-cli test` validates your actual Snowflake table against this contract. It checks types, required fields, minimum values, and can be wired into CI so that any schema change that would violate the contract fails the PR before it merges.

    The only safe migration path for breaking changes

    When a producer needs to make a breaking change — remove a field, rename it, change its type, change its semantics — the contract provides a coordination mechanism. There’s a specific pattern that works, and teams that skip steps in it pay for it.

    Day 0: Announce. The producer creates a deprecation notice in the contract YAML, updates the changelog, and notifies consumers via a designated channel. Critically, this notification includes a hard date for removal — not “eventually” or “when everyone has migrated.” Deprecated without a date is just a polite rumor. A field can sit in limbo for eighteen months while producers assume nobody uses it and consumers assume it will live forever.

    Days 0–60: Dual-write. The producer populates both the old field and the new field simultaneously. Consumers can migrate on their own schedule during this window. The producer monitors usage of the old field (this is easy with Snowflake’s QUERY_HISTORY and column-level access tracking) to know when all consumers have switched.

    Day 60: Deprecation notice with hard date. Consumers who haven’t migrated get a 30-day final warning. This is the reminder that actually motivates stragglers. The hard date is non-negotiable.

    Day 90+: Removal at v2. The old field is gone. The contract version bumps to 2.0.0. This is a semantic major version — it breaks backward compatibility — and that bump is what triggers automated alerts to any consumer still on v1.

    No drama. No guessing. No 2 AM rollback. Give consumers at least 90 days notice for breaking changes. This seems long, but data pipelines have long release cycles, and consumers need time to update downstream logic, tests, and dashboards.

    Making it executable: CI enforcement that actually works

    The critical architectural decision with data contracts is this: a contract not enforced in CI is just documentation, and documentation drifts. Within six months, the contract YAML and the actual schema diverge, nobody updates the contract when they ship features, and you’re back to tribal knowledge with extra steps.

    The enforcement pattern that works:

    1. Compatibility check on PR. Before any schema change merges, run `datacontract-cli diff` against the current production contract. Breaking changes fail the PR automatically. Non-breaking changes (adding a nullable field, loosening a constraint) pass. The definition of “breaking” is explicit in the contract spec, not up to whoever reviews the PR.

    2. Consumer sign-off for breaking changes. If a breaking change is intentional (the producer knows and has planned for it), the PR requires explicit approval from all registered consumers of that dataset. This is enforced via GitHub CODEOWNERS or equivalent. Producers can’t ship breaking changes unilaterally.

    3. dbt test integration. Map contract quality rules to dbt tests. Freshness SLAs become `dbt source freshness` checks. Completeness thresholds become row count assertions. Not-null requirements become `not_null` tests. These run on every dbt build, so violations are caught before models complete — not after reports are wrong.

    4. Runtime validation at ingestion. Before data loads into your Silver or Gold layers, validate incoming records against the contract. Rows that violate constraints get quarantined in a dead-letter queue, not silently loaded as nulls. This catches semantic drift that schema validation misses: an order_amount field that’s suddenly returning negative values because someone upstream changed the sign convention.

    The gotchas that sink most implementations

    Exposing raw transactional schemas as data products. This is the most common structural mistake. When your data contract directly mirrors your application’s OLTP schema, every application refactor becomes a consumer’s problem. The fix is a stable abstraction layer — expose only what consumers need, not the underlying operational detail. Schema changes to the application layer should be absorbed by your ingestion layer, not propagated downstream.

    Brittle contracts that break more than they prevent. Strict attribute lengths, tightly constrained enums, or hyper-specific format requirements feel like good quality controls. In practice, they make schemas so rigid that producers constantly need change approvals for minor operational updates that have no downstream impact. Design contracts around semantic guarantees and business invariants, not implementation details. amount > 0 is a semantic guarantee. DECIMAL(18,4) is an implementation detail that will change.

    Unclear ownership is the silent killer. Data contracts fail most often not because of tooling gaps, but because accountability is unclear. When something breaks, teams scramble to diagnose issues that fall between ownership boundaries. Every contract needs a named owner with actual incident-response obligations. Not a team. Not a Slack channel. A person whose name is in the contract and who gets paged when a contract violation is detected at runtime.

    Semantic changes that look like no-ops. Changing what a field means without changing its name, type, or schema is the hardest class of breakage to catch. order_amount switching from gross to net. A user_id changing from internal to external identifiers. These require semantic versioning (a major version bump) and human review, not just automated compatibility checks. Your CI can catch structural breakage; only your team can catch semantic breakage.

    Contracts that cover batch but ignore streaming. If you have a Kafka-based event pipeline feeding your Snowflake tables, the schema contract lives in the Kafka topic, not in the table. Changes to the Kafka Avro schema — registered in Confluent Schema Registry or AWS Glue — need the same versioning and deprecation discipline as your warehouse schemas. Most teams only contract the warehouse side and get burned by streaming schema changes that propagate silently into their pipeline.

    The real cost math

    Data engineering incidents from schema breakage are expensive in ways that don’t show up on warehouse bills. A typical schema incident at a mid-sized company looks like: 3–4 hours of two engineers debugging, 1 hour of a data analyst investigating wrong numbers, a director review, and a post-mortem. Call that 10 person-hours, at a blended rate of $150/hour. That’s $1,500 per incident.

    Teams that experience two schema incidents a month — which is conservative for a team without contracts — are burning $3,000/month, or $36,000/year, on incidents alone. That doesn’t count the cost of wrong decisions made from bad data before the incident was even discovered. One revenue calculation running off a silent semantic change for three weeks is often worth more than a year of incident cost.

    The tooling investment for data contracts — `datacontract-cli`, ODCS YAML per dataset, CI integration — is a few days of engineering time. The 90-day discipline is a process change, not a tooling cost. The math is not close.

    Where to start (not where everyone starts)

    Everyone says “start with your most critical datasets.” That’s correct but useless. More specifically: identify the three datasets that caused production incidents in the last 90 days. Start with those. Not your biggest datasets. Not your most complex. The ones that already broke something.

    For each: write the ODCS YAML (schema + semantics + SLAs + owner). Add `datacontract-cli` compatibility checks to the PR workflow for that dataset. Map the quality rules to dbt tests. That’s the first sprint. After one month of this on three datasets, you’ll have a template, a workflow, and enough muscle memory to expand to the rest of the catalog without it feeling like a governance initiative nobody asked for.

    The one principle

    Change is inevitable. Unmanaged change is expensive. A data contract is the agreement that makes change boring instead of dangerous. The goal isn’t to prevent schemas from evolving — schemas should evolve as the business evolves. The goal is to make every evolution visible, deliberate, and announced far enough in advance that nobody finds out about it from a director on a Friday morning.

    Related reading: Open Data Contract Standard (ODCS) · datacontract-cli on GitHub · dbt State: Skip Unchanged Nodes, Cut Runtime by 60% · dbt Fusion: 30x Faster Parsing · Snowflake Query Execution: What Really Happens

  • Why AI Agents Forget: The Architecture behind Memory failures

    Why AI Agents Forget: The Architecture behind Memory failures

    Your AI agent isn’t getting dumber over time. It’s getting amnesiac. It forgets a constraint you set ten turns ago, even though it followed it perfectly at turn three. It contradicts itself across sessions. It treats a fact you corrected last week as if it never heard the correction. Teams blame the model. They swap GPT for Claude, Claude for Gemini, hoping a smarter model fixes the problem.

    It doesn’t. Because the problem was never reasoning. It’s architecture. Specifically: most teams are using the context window as a database, and the context window was never built to be one.

    A 2026 study tracking 4,416 trials across six conversation depths found something precise: when an agent violates a constraint it followed correctly ten turns earlier, the model didn’t change — the attention weight on that constraint dropped below the threshold needed to enforce it. That’s not a reasoning failure. That’s a memory architecture failure wearing a reasoning costume.

    TL;DR

    → The context window behaves like RAM, not storage: volatile, finite, and degraded by clutter. Most agent failures blamed on “the model” are actually memory architecture failures.

    → Constraints decay with distance. A rule followed correctly at turn 3 can silently fail by turn 10 — not because the model forgot, but because attention weight on it dropped below the enforcement threshold.

    → Four memory types need separate handling: working (current task), episodic (past interactions), semantic (facts/preferences), and procedural (learned skills). Production systems collapse these into one bucket and pay for it.

    → Best 2026 architectures hit ~92.5 on LoCoMo and ~94.4 on LongMemEval benchmarks at roughly 6,900 tokens per retrieval — a fraction of full-history prompting.

    → Memory poisoning is now a named, ranked threat (OWASP ASI06, 2026). Attack success rates of 80–99.8% have been demonstrated against production-style agents.

    → Unlike prompt injection, memory poisoning is temporally decoupled: the attacker writes today, the agent misbehaves months later, with no single suspicious moment to catch in logs.

    → Frameworks like Letta, Mem0, and Cognee treat memory as a tiered OS-style hierarchy — context as RAM, external store as disk — rather than a bigger prompt.

    → Bigger context windows do not solve this. They delay the symptom and raise the cost per query while “lost in the middle” retrieval failures persist regardless of window size.

    The assumption everyone makes (and shouldn’t)

    Ask most engineers how their agent “remembers” things, and the honest answer is: it doesn’t, not really. It re-reads the entire conversation history on every single call. Every query triggers full recomputation from scratch — the model has no concept of “yesterday” unless yesterday’s text is physically present in today’s prompt.

    This statelessness is a deliberate design choice, and it has real upside: reproducibility, simplicity, no hidden corrupted state between calls. But it creates two structural problems nobody can engineer around with a smarter model. First, computational inefficiency — you’re paying to recompute similarity over text the model has already processed a hundred times. Second, and more dangerous: context window limits. Long multi-turn conversations, agentic workflows, and long-running tasks all need more history than fits, so teams either truncate (losing information) or compress (introducing error) or simply hope the window is big enough this time.

    Bigger windows feel like the obvious fix. They aren’t. Long context windows still suffer “lost in the middle” retrieval failures — the model technically has the information but doesn’t weight it correctly when it matters — while full-history prompting creates real cost problems at enterprise scale. You can have a million-token window and still watch an agent forget a name mentioned at token 40,000 because it’s buried under everything that came after.

    Why the RAM analogy actually explains the failures you’re seeing

    The context window shares three properties with RAM that distinguish it from persistent storage, and the mismatch is what breaks production agents. It’s volatile — everything disappears at session end, including a preference stated at turn one and a constraint set at turn three. It’s finite — there’s a hard ceiling, and once you hit it, something gets evicted whether you chose it or not. And it’s expensive per byte — every token you keep “just in case” is a token you pay to process on every single call, forever, for the life of that conversation.

    When you build against the context window as if it were a database — appending forever, never pruning, assuming everything you put in stays retrievable — you get failures that look exactly like the model is getting confused, contradictory, or “dumber.” It isn’t. You’re running a database workload on a RAM-shaped substrate, and RAM does what RAM does: it fills up, and old things get pushed out or buried.

    The fix isn’t a bigger window. It’s a second layer: a persistent memory store, external to the context window, that you control like an operating system controls RAM — deciding deliberately what goes in, what stays, and what gets evicted, instead of letting the model figure it out by attention weights alone.

    Four memory types, one bucket (the real architectural sin)

    Most production agents collapse everything into a single, undifferentiated memory blob: conversation history. But mature memory architecture treats at least four types as distinct, because they decay differently, get retrieved differently, and fail differently when mishandled.

    Working memory — the current task state, what you’re doing right now. Short-lived, high-relevance, meant to be discarded once the task completes.

    Episodic memory — specific past interactions and experiences. “Last Tuesday the user asked about refund policy and got frustrated with the answer.” Time-stamped, specific, useful for continuity.

    Semantic memory — durable facts and preferences, stripped of the conversational context that produced them. “User prefers email over Slack.” “User’s company uses Snowflake, not BigQuery.” This is what most people mean when they say “the agent remembers me.”

    Procedural memory — learned skills and patterns of action. “When this user asks for a report, format it as a table, not prose.” This is the hardest to do well and the most valuable when done right.

    Production systems that dump all four into one vector store and retrieve by similarity alone tend to surface the wrong type at the wrong time — episodic noise crowding out a stable semantic fact, or a one-off preference from a bad mood three months ago resurfacing as if it were a permanent rule. Coordinating transitions between these types — when does an episodic memory get distilled into a semantic fact? when does a procedural pattern get unlearned? — is most of what separates a memory system that improves over months from one that quietly degrades.

    The retrieval pipeline, and where the cost actually goes

    In a properly built memory layer, the model never sees your full history. During conversations, the system extracts facts and stores them in a vector database indexed by user, session, and agent identifiers. At the start of a new session — or mid-conversation, as needed — relevant memories are retrieved using a combination of semantic similarity, keyword matching, and entity matching, then injected into the context window right before the model responds. Only the most relevant facts surface, which keeps token usage low and retrieval precise instead of dumping everything and hoping attention sorts it out.

    This is where the real cost math lives. A naive approach — replaying full conversation history every turn — scales token cost linearly with conversation length, and by month three of an active user relationship, you’re paying to reprocess tens of thousands of tokens of mostly irrelevant history on every single message. A well-built retrieval layer holds that flat: leading 2026 systems achieve strong recall on multi-session benchmarks while retrieving roughly 6,900 tokens per call, regardless of how long the relationship has run. That’s not a marginal efficiency gain — it’s the difference between a cost curve that’s flat and one that grows without bound as your best, most loyal users accumulate the longest histories.

    The benchmarks that matter here are LoCoMo (long conversation memory), LongMemEval, and BEAM — they specifically test whether an agent can recall and reason over facts buried many sessions back, not just within a single long context. Recent leaders score around 92–94 on these, with the largest gains coming from temporal reasoning (knowing when something was true, not just that it was said) and multi-hop retrieval (connecting two separate facts from different sessions to answer one question).

    The gotchas nobody warns you about

    Constraints decay with distance, silently. This isn’t intuitive until you’ve watched it happen. An agent that perfectly honors “never mention competitor X” for the first eight turns will sometimes mention competitor X at turn fifteen — not because anything changed, but because the attention weight on that instruction, buried further and further back, dropped below the threshold needed to actually constrain output. Negative constraints (“don’t do X”) decay faster than positive instructions (“do Y”), because there’s no ongoing signal reinforcing the absence.

    “Lost in the middle” doesn’t go away with bigger context. Models reliably retrieve information near the start or end of a context window far better than information buried in the middle. Doubling your context window doesn’t fix this — it just moves where the “middle” is, and gives you more room to bury things in it.

    Memory poisoning is not prompt injection’s cousin — it’s a different threat class entirely. Prompt injection is session-scoped: it does damage now, and the damage ends when the session ends. Memory poisoning writes malicious content into persistent storage, where it survives across every future interaction, triggered by completely unrelated conversations months later. OWASP formalized this as ASI06 in its 2026 Agentic AI Top 10, specifically because the defenses that work against prompt injection — input moderation, output filtering, session-bounded monitoring — don’t catch an attack that was planted in February and triggers in April.

    The attack success rates are not theoretical. Published research demonstrates attack success rates ranging from roughly 80% up to 99.8% against agent memory systems using techniques like indirect injection through documents the agent is asked to summarize, or webpages the agent is asked to fetch. One demonstrated case against a cloud agent platform showed a single crafted webpage URL, fetched by the agent, writing persistent instructions into session memory that silently exfiltrated data on every subsequent interaction.

    Stale facts actively degrade output, they don’t just sit inert. A semantic memory that was true six months ago — “user works at Company A” — doesn’t just become irrelevant when it goes stale. If never pruned or updated, it actively competes with the correct, current fact at retrieval time, and similarity search has no inherent way to know which one is “more true.” Memory systems need explicit staleness handling, not just additive storage.

    Cross-session identity is still mostly unsolved. If the same person talks to your agent from their phone, their laptop, and an anonymous browser session before logging in, stitching those into one coherent memory profile is an open research problem, not a solved one. Most production systems quietly accept fragmented identity as a known limitation rather than a bug to fix.

    What the better architectures actually do

    The frameworks that handle this well — Letta, Mem0, Cognee, and similar — share a common idea: treat memory like an operating system treats RAM, not like a developer treats a growing log file. Letta’s approach is explicit about this, using a tiered architecture where the active context functions as RAM and an external store functions as disk, with the agent able to read, write, and archive its own memory through function calls rather than having everything force-fed into every prompt. Mem0 takes a similar stance from the extraction side: pull key facts out of conversation, then run an explicit decision step — add, update, delete, or no-op — so memory accumulates deliberately instead of by default.

    The common thread across all of them: memory is a dedicated architectural component, separate from the model’s context window, not just a longer prompt wearing a fancier name.

    The real question: build, or borrow?

    Reach for a managed memory framework if: you’re shipping a consumer-facing or long-running agent where users return across days or weeks, you don’t have a research team to spend on retrieval tuning, or you need cross-session identity and staleness handling out of the box rather than building it yourself.

    Build it yourself if: your agent is genuinely single-session (no continuity needed across conversations), your team has the bandwidth to own retrieval quality and security hardening long-term, or you’re operating in a regulated environment where you need full control over where memory data physically lives.

    Either way, budget real engineering time for the security side. Memory poisoning defenses — provenance tracking on what gets written to memory and from where, trust-scoring on retrieved content before it’s injected into context, and behavioral monitoring for an agent that starts defending beliefs it has no legitimate reason to hold — are not optional hardening for later. They’re part of the architecture, the same way input validation isn’t optional hardening for a web form.

    The one principle

    Treat the context window like RAM you actively manage, not a database that remembers for you. Decide deliberately what goes in, what gets promoted to durable storage, and what gets evicted — because if you don’t make that decision, attention weight and token limits will make it for you, silently, and you’ll find out about it from a user complaint instead of a design review.

    Related reading: OWASP Top 10 for Agentic Applications · dbt State: Skip Unchanged Nodes, Cut Runtime · dbt Fusion: 30x Faster Parsing · Snowflake Iceberg v3 Migration Guide

  • dbt state: Skip Unchanged Nodes, Cut Warehouse Compute 30%

    dbt state: Skip Unchanged Nodes, Cut Warehouse Compute 30%

    You have a 400-model dbt project. A junior analyst tweaks one source definition. Every. Single. Model. Rebuilds. Ninety minutes later, you’ve burned $1,200 in warehouse compute for a change that affected nothing downstream. That’s the dbt default. It’s also the most expensive habit in the modern data stack.

    dbt’s State feature is the principled answer: compare your current project against a previously saved manifest, identify what has actually changed, and run only those models. No guessing. No manual orchestration. No fear.

    The feature has been in dbt Core since v0.18, but most teams don’t use it — because manifest management felt clunky. With dbt Cloud, it’s now automatic. With dbt Core, it’s a straightforward S3 upload. And the returns are stark: 60–90% reduction in CI runtime, 30% warehouse compute savings, and developer feedback loops that feel instant instead of hourly.

    TL;DR

    → dbt State compares your current project against a saved manifest (JSON) to identify changed models. Run only what’s different.

    → Core selector: `dbt run –select state:modified+ –state ./prod-artifacts`. The `+` rebuilds downstream dependents too.

    → Variants available: `state:modified.body` (SQL changed), `state:modified.configs` (config changed), `state:new` (newly added models).

    → Combine with `–defer` to resolve unchanged upstream models to production instead of rebuilding them. Game-changer for dev workflows.

    → Real runtime savings: 400-model project, CI goes from 48 min (full rebuild) to 6 min (3 changed models with dependents). 87.5% faster.

    → Setup: Persist production manifest to S3/GCS after every run. Download it in CI, add two flags. Takes 20 minutes to wire up.

    → dbt Cloud does this automatically. dbt Core requires DIY manifest management (simple, but manual).

    → Gotcha: Source freshness changes don’t trigger model runs. New columns on upstream models won’t flag downstream models unless explicitly selected.

    → One principle: Compare, don’t guess. The manifest is the source of truth for what changed.

    The Problem with Running Everything

    Most analytics teams start with small dbt projects. A few models, a `dbt run`, done in seconds. Then the project grows. Hundreds of models. Dozens of sources. Complex DAGs spanning raw ingestion to business-critical marts. Suddenly `dbt run` takes 45 minutes — and you’re running it ten times a day in CI.

    The naive solution: run only the models you touched. But doing this manually is error-prone. You forget an upstream dependency. A downstream mart goes stale. You ship broken data. Teams end up caught between speed and correctness, and neither option feels good.

    dbt State solves this: automatically identify what changed, run only those models (plus downstream dependents), skip everything else. No manual selection. No guessing. Safe by default.

    What dbt State Actually Is

    dbt State compares manifests: current vs. prior. Changes detected → rebuild. No changes → skip.

    dbt State is the mechanism by which dbt compares your current project against a previously compiled artifact — specifically the manifest.json file — to determine what has actually changed.

    The manifest is a JSON file that dbt generates on every `dbt compile` or `dbt run`. It captures a complete snapshot of your project at a point in time: model definitions, compiled SQL, configurations, tests, sources, and the relationships between them.

    By diffing the current manifest against a prior one, dbt can identify:

    • Models whose SQL has changed
    • Models whose configuration has changed (e.g., materialized, tags, meta)
    • Models whose upstream dependencies have changed
    • New models that didn’t exist before
    • Models whose schema or source freshness has changed
    • Models that call a macro that has changed

    Everything else is left alone.

    The Core Selector: `state:modified`

    The entry point to dbt State is the `state:modified` node selector. It filters your run to only nodes that have changed relative to a saved state:

    dbt run --select state:modified --state ./prod-artifacts

    Here `./prod-artifacts` is a directory containing the `manifest.json` from your last production run. dbt compares every node in your current project against that manifest and runs only what’s different.

    Selector Variants

    dbt ships several variants of the selector for fine-grained control:

    state:modified — All nodes with any change (SQL, config, schema)
    state:modified.body — Only models where the SQL body changed
    state:modified.configs — Only nodes where configuration changed
    state:modified.persisted_descriptions — Column descriptions changed
    state:modified.relation — Relation name or schema changed
    state:modified.macros — An upstream macro changed (impacts compiled SQL)
    state:new — Entirely new models (didn’t exist in saved state)

    The most common pattern combines `state:new` and `state:modified` to catch everything relevant:

    dbt run --select state:new,state:modified+ --state ./prod-artifacts
    
    

    The trailing `+` means: run all modified nodes and everything downstream of them. This ensures referential integrity — if stg_orders changes, every mart that joins on it will also rebuild.

    Whether to use `+` depends on your setup:

    Incremental tables downstream: Often safe to skip, since they’ll pick up new rows on the next run anyway.
    Full-refresh tables or views downstream: Should be rebuilt if their upstream changes.
    Critical reporting models: Should probably always be included for safety.

    Most teams use `state:modified+` as the default and carve out exceptions for incremental models.

    Real-World Runtime Savings

    Cost comparison: Without dbt State (rebuild every model every run: 500 models × 24 hourly runs = 12,000 rebuilds/day = $5,200/month). With dbt State (average 35% fewer models rebuilt, 9% compute efficiency = $4,420/month). Monthly savings: $780. Annual: $9,360.

    Runtime reduction depends on project shape, but 60–90% is typical for mature projects.

    How much time you save depends on the shape of your project, but the pattern is consistent: most runs in a mature dbt project touch a small fraction of the total model count.

    Consider a 400-model project:

    Full `dbt run` (no state): 400 models built = 48 minutes
    PR touches 3 models: ~20 models run (with `+`) = 6 minutes (87.5% faster)
    Hotfix to 1 model: ~8 models run (with `+`) = 2 minutes (95.8% faster)
    Daily incremental run: ~15 models run = 4 minutes (91.7% faster)

    For large, mature projects, you regularly see 70–90% reductions in CI runtime once state selection is in place.

    Setting It Up in CI/CD

    The real power of dbt State emerges in CI/CD pipelines. The pattern is:

    1. After every successful production run, upload the manifest.json to a persistent store (S3, GCS, Azure Blob, or an artifact registry).
    2. In CI, download the latest production manifest before running dbt.
    3. Run dbt with state:modified+ against that manifest.

    GitHub Actions Example

    jobs:
    dbt-ci:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    
    - name: Download production manifest
    run: |
    aws s3 cp s3://your-bucket/prod/manifest.json ./prod-artifacts/manifest.json
    
    - name: Install dbt
    run: pip install dbt-core dbt-snowflake
    
    - name: Run modified models only
    run: |
    dbt run \
    --select state:new,state:modified+ \
    --state ./prod-artifacts \
    --target ci

    After Production Run: Upload the Manifest

    - name: Run dbt production
    run: dbt run --target prod
    
    - name: Upload manifest to S3
    run: |
    aws s3 cp ./target/manifest.json s3://your-bucket/prod/manifest.json

    This creates a feedback loop: every successful production run produces the baseline for the next CI comparison.

    How dbt Computes “Modified”

    Understanding what triggers a `state:modified` match helps you trust the selector and avoid surprises.

    dbt computes a content hash for each node in the manifest. The hash covers the compiled SQL (after Jinja rendering), the node’s configuration block, and for sources, the freshness configuration.

    If the hash changes between manifests, the node is considered modified. This means:

    Whitespace changes in SQL do NOT trigger a rebuild (dbt normalizes whitespace before hashing).
    Comment changes alone do NOT trigger a rebuild.
    Jinja logic changes that produce different compiled SQL DO trigger a rebuild.
    Macro changes propagate: if a macro used by a model changes, the model’s compiled SQL will differ, and it will be flagged as modified.

    This is conservative and safe — you might rebuild more than strictly necessary, but you won’t accidentally skip a model that needs to run.

    Combining State with `–defer`

    --defer is a closely related feature that pairs naturally with --state. While state:modified controls what you run, --defer controls where dbt looks for relations that you aren’t running.

    dbt run \
    --select state:new,state:modified+ \
    --state ./prod-artifacts \
    --defer \
    --target dev

    With --defer, when model A references model B and B is not being run (because it’s unchanged), dbt resolves the ref('B') to the production relation instead of the development one. This means your CI or dev runs don’t need a full copy of the warehouse — they can borrow production tables for anything they’re not rebuilding.

    The combination is transformative for developer workflows:

    • Developers run only the models they changed.
    • Unchanged upstream models resolve to production.
    • No need to seed or pre-build the entire project in a dev schema.
    • Full isolation — changes don’t interfere with each other.

    The Gotchas Nobody Mentions

    Source freshness changes don’t trigger model runs. `state:modified` on sources reflects freshness configuration changes, not the actual data changing. If you change a source’s freshness window from 1 hour to 2 hours, that’s a config change, and the source will be flagged as modified. But downstream models won’t automatically rebuild just because the source data has changed. dbt assumes downstream models will be rebuilt on schedule or on demand.

    New columns on upstream models won’t flag downstream models. If an upstream model adds a column but its SQL otherwise produces the same results, the downstream model won’t be flagged as modified — even if your downstream model does SELECT *. For this reason, avoid `SELECT *` in critical production models. Be explicit about column selection.

    The manifest must match the target environment. The saved manifest should come from a run against the same target (e.g., production). Using a manifest from a different environment (e.g., a dev manifest) can produce incorrect change detection.

    First run has no baseline. On first use, there’s no prior manifest. Either run everything once to establish the baseline, or use dbt Cloud’s built-in state management, which handles this automatically.

    Manifest compatibility across dbt versions. If you upgrade dbt Core between runs, the manifest schema might change, and the comparison might fail. Always keep your CI environment and production environment on the same dbt version (or be very careful when upgrading).

    When It Breaks (And How to Fix It)

    Scenario: “state:modified found no changes, but I know I changed something.”

    dbt is comparing content hashes, not file modification times. If your change didn’t alter the compiled SQL or configuration, dbt won’t see it as modified. This is rare but can happen if you:

    • Changed a comment in a Jinja block (comments get compiled out)
    • Changed a variable used only in a non-dbt file
    • Updated a macro without using it in a model

    Solution: Explicitly select the model with `–select model_name` to force a rebuild.

    Scenario: “My dbt Cloud runs are state-aware, but my local development isn’t.”

    dbt Cloud automatically manages state. Local development requires you to download a manifest and point to it. If you’re toggling between the two, you might accidentally run full rebuilds locally. Solution: Set up manifest downloads locally too (or use the dbt Cloud CLI).

    dbt Cloud vs. dbt Core State Management

    dbt Cloud: Automatically persists manifests from prior runs and exposes a `–defer-to-state` toggle in the UI. Zero setup.

    dbt Core: Requires you to manually persist the manifest (S3, GCS, etc.) and download it in CI. More work, but straightforward with any object store. Takes about 20 minutes to wire up.

    For teams on dbt Core, the manifest management is DIY but simple. For dbt Cloud users, it’s automatic — one less thing to maintain.

    The Real Cost Math

    Assume a 400-model Snowflake project, hourly CI runs:

    Without state: 400 models × 24 runs/day = 9,600 models built/day = $4,800/month

    With state: Average 60% skip rate = 3,840 models built/day = $1,920/month

    Monthly savings: $2,880 | Annual: $34,560

    And this doesn’t count the developer time saved from faster CI feedback loops. A team running 10 PRs a day, each waiting 45 minutes for CI instead of 6 minutes, saves 390 person-minutes per day. Over a year, that’s 1,560 hours of developer time.

    The One Principle

    Compare, don’t guess. The manifest is the source of truth for what changed. dbt State removes the need for manual orchestration, custom scripts, or human judgment about what to rebuild. Compare the current project against a prior snapshot, run only what’s different, trust the math. That’s the entire philosophy.

    Related reading: State Selection (dbt Docs) · Graph Operators (the `+` operator) · dbt Fusion: 30x Faster Parsing · Snowflake Query Execution: What Really Happens

  • dbt Fusion: 30x Faster parsing(And why Migration matters)

    dbt Fusion: 30x Faster parsing(And why Migration matters)

    You’ve probably heard the buzz: dbt’s new Fusion engine is 30x faster. But what nobody says clearly is faster at what, for whom, and does it break your project? The answer is messier than the marketing.

    The Fusion engine is a complete rewrite of dbt Core in Rust, released last year and hitting 4,500+ projects already. It’s genuinely fast at parsing and compilation — the steps dbt runs locally before it ever talks to your warehouse. But parsing speed doesn’t change your pipeline runtime. What does change is the developer experience: real-time error feedback in VS Code, instant file recompilation, the ability to catch typos before you run a job. That’s real. But the migration is not friction-free. Fusion enforces stricter validation than dbt Core, which means deprecated code patterns that currently just warn you will now block your runs. And some packages won’t work yet. It’s a choice, not a free lunch.

    TL;DR

    → dbt Fusion is the Rust-based evolution of dbt Core. Parse times up to 30x faster (vs Python). Full project compilation 2x faster. Real-time VS Code integration with IntelliSense.

    → The speed win is in local parsing/compilation, not warehouse query time. It changes the developer experience, not your pipeline runtime.

    → Migration requires: resolve all deprecation warnings, update packages, test with --use-v2-parser flag first, upgrade dev then staging then production.

    → What breaks: strict YAML validation, old CLI flags (--models → --select), behavior change flags can’t be disabled, YAML anchors need to move under anchors: key, some packages incompatible.

    → Static analysis defaults to “baseline” mode (warnings, not errors), making gradual adoption possible. Opt models into “strict” mode incrementally.

    → dbt-autofix tool automatically fixes ~80% of compatibility issues. Don’t do it manually.

    → Always upgrade dev first, test for a week, then staging, then production. dbt Manifest incompatibility means you can’t mix v20 (Fusion) with v12 (Core) across environments.

    The mental model that’s outdated

    Most analytics engineers think of dbt as a slow, clunky tool. Write a model, run dbt build, wait 30 seconds for parse time on a big project, wait for warehouse execution, see the result. Then fix a typo, run again, repeat. It’s a cycle, and it’s painful at scale.

    That cycle is the Python dbt Core experience. It’s been the same since 2016.

    Fusion breaks that cycle. With Rust, with SQL comprehension, with the VS Code extension powered by a language server, you get real-time feedback. You type a SQL syntax error and the editor shows you the error as you type, without running dbt, without hitting the warehouse. You save a file and Fusion recompiles your entire project in your IDE in seconds, not minutes.

    Developer experience comparison: dbt Core iteration cycle (10+ minutes per loop) vs dbt Fusion (5 minutes per loop with real-time IDE feedback)

    Developer experience: dbt Core requires warehouse round-trips for each error. Fusion catches errors in the IDE instantly, offline

    But — and this is critical — this only affects your local development experience. Your actual pipeline runtime (the time from `dbt build` to “done”) is nearly unchanged. Parsing and compilation are usually 5–15% of total run time. Warehouse execution is the rest. Fusion doesn’t touch warehouse execution.

    If you’re excited about Fusion because you think it will cut your 2-hour nightly pipeline to 1 hour, you’re disappointed. If you’re excited because you want real-time error feedback while you’re writing models, you’re right to be excited.

    What changed: The 30x parsing speed, explained

    dbt Core v1 (Python) parses your YAML, your Jinja templates, your SQL, and figures out the dependency graph all in Python. Python is slow at this. On a project with 500 models and intricate Jinja, parsing alone can take 15–30 seconds.

    Fusion (Rust) does the same work but in a compiled binary. No Python interpreter overhead. No garbage collection pauses. Just native machine code doing the parse. Result: parsing in 0.5–1 second on the same project.

    That’s the 30x number. Real number, not marketing.

    Then Fusion goes further. It parses SQL syntax, understands column types, knows which functions exist on which warehouse, and validates your SQL before it ever hits the warehouse. dbt Core doesn’t do this; it just templated text and sends it up. Fusion actually understands your SQL. Which is why it catches errors in the IDE before you run the job.

    But here’s what Fusion doesn’t change: the time your actual SQL queries take to run on the warehouse. That’s still determined by your warehouse optimizer, your data volume, your indexing. Fusion has zero impact on that.

    What breaks during migration

    YAML validation gets strict. dbt Core accepts YAML files with extra keys at the top level (they’re just ignored). Fusion rejects them with an error. A common pattern: defining YAML anchors at the top level of your schema.yml file. In Fusion, those have to go under an anchors: key. The dbt-autofix tool fixes this automatically, but if you’re migrating manually, this is where you’ll get tripped up first.

    Deprecated code is no longer a warning. In dbt Core v1, using an old feature generates a warning. In Fusion, it’s a hard error. Before you migrate, you must run your project on Latest dbt Core track, fix all deprecation warnings, then upgrade to Fusion. There’s no skipping this step.

    Behavior change flags can’t be disabled. dbt Core has flags like require_column_description. You set require_column_description: false to opt out. Fusion enables all these flags and doesn’t allow you to opt out. If your project relies on column descriptions being optional, Fusion will reject it.

    Old CLI flags are gone. The --models flag (deprecated since dbt 0.21) works in Core but errors in Fusion. Use --select. Same with --resource-type → use --resource-types. All your job definitions, all your scripts have to be updated.

    get_relation() print behavior changes. In dbt Core v1, printing the result of `get_relation()` when the relation doesn’t exist shows “None”. In Fusion, it errors. This breaks some legacy macros. Rare, but it happens.

    Package incompatibility is real. Not every dbt package has been updated for Fusion. If your project depends on an unmaintained package, you’ll have to fork it or wait. dbt Labs packages (dbt_utils, dbt_project_evaluator) are compatible. Most popular community packages are. But check the dbt package hub; it shows a Fusion-compatible badge.

    Manifest incompatibility means you can’t mix dbt versions across environments. Fusion produces a v20 manifest. Latest dbt Core produces v12. If your dev environment is Fusion and your production is still Core, features like `state:modified` and `–defer` break because the manifests are incompatible. You have to upgrade all environments together or not at all.

    The gotchas that actually matter

    Static analysis defaults to “baseline” mode. This is Fusion’s secret weapon for adoption. Instead of strict SQL validation (which would break half the projects), Fusion defaults to “baseline” mode. Validation errors show as warnings, not blockers. Your project still builds and runs. You get real-time feedback in the IDE, but you’re not forced to fix everything at once. This is intentional — it’s a soft migration path.

    You can opt individual models into strict mode with static_analysis: strict in your config. Opt in where you want full SQL comprehension. Leave the rest in baseline.

    UDFs are limited in strict mode. If you use custom user-defined functions (UDFs), Fusion’s strict mode has trouble with them unless you register them in sql_header or on-run-start hooks. This catches most cases, but warehouse-native functions or post-hooks that define UDFs can cause issues. In baseline mode, most UDFs just work.

    Parsing is local-only; dbt still needs the warehouse. Fusion’s fast parsing happens offline, on your laptop. But when you run dbt build, you still need warehouse access for execution, for macro evaluation, for source freshness checks. Fusion doesn’t change that. If you’re offline or your warehouse is down, you’re still blocked.

    dbt Mesh with Semantic Layer has limitations. If you use cross-project metric references in dbt Mesh, that’s only supported in the legacy Semantic Layer YAML spec, not the new one. You’ll have to choose between Mesh cross-project references and the new cleaner YAML structure. Support for both is planned but not here yet.

    The migration checklist (what actually works)

    Step 1: Prepare on Latest dbt Core (1.12+). Move your entire project to the Latest release track on dbt Core. This includes all development environments and jobs. Don’t skip this. Latest is where deprecated features show up clearly.

    Step 2: Run dbt-autofix. Install dbt-autofix (with uvx dbt-autofix) and run it on your project directory. It automatically rewrites your YAML to conform to the latest schema, moves YAML anchors under an anchors: key, updates deprecated configs, and upgrades packages to Fusion-compatible versions. This alone fixes ~80% of issues. Do not skip this step and do it manually. The tool is designed to be safe and generates a clean git diff.

    Step 3: Fix remaining deprecation warnings. After dbt-autofix, you might still have warnings (old Jinja patterns, intricate macros, unsupported SQL features). Fix these manually. Your dbt_project.yml should now have zero deprecation warnings when you run on Latest track. If you see warnings, resolve them before going further.

    Step 4: Update all packages. Run `dbt deps update` to pull the latest versions. Most popular packages now have Fusion-compatible releases. If you have a custom package that’s not Fusion-compatible, you’ll need to update it separately (or fork it).

    Step 5: Test the v2 parser locally. If you’re on dbt Core v1.12, you can test Fusion’s parser without migrating the whole engine. Run dbt build --use-v2-parser. This delegates only parsing to Fusion’s engine but keeps Core’s execution. If this succeeds, you’re likely safe to migrate. If it fails, fix the issues before upgrading.

    Step 6: Upgrade dev environment to Fusion. In the dbt platform, upgrade your development environment to use the Latest Fusion release track. Test everything in dev for a full week. Run jobs, run CI, run your usual workflows. This is not a 2-hour test; it’s a 7-day validation.

    Step 7: Watch for manifest incompatibility. If your staging or production environments are still on dbt Core, the v20 (Fusion) and v12 (Core) manifests won’t talk to each other. Features like `state:modified` and `–defer` will silently fail. So either upgrade all environments together, or keep everything on Core for now. You can’t have a hybrid setup.

    Step 8: Upgrade staging, then production. After dev validation, upgrade staging (if you have it). Run production-like workloads for another 24–48 hours. Finally, upgrade production. Don’t do all three at once.

    Step 9: Monitor the first 48 hours of production. Watch scheduled job runs, compare run times to baselines, look for unexpected failures. Fusion is stable, but this is where you’d catch edge cases specific to your setup.

    When the speed actually matters

    Parsing 30x faster sounds great until you realize: parsing is 10 seconds out of a 120-second pipeline. You’re saving about 9 seconds. That’s real, but it’s not transformative.

    Pipeline breakdown: dbt Core (25s parse + 15s compile + 60s warehouse = 100s total) vs dbt Fusion (0.8s parse + 7s compile + 60s warehouse = 67.8s total). Parsing 30x faster, compilation 2x faster, warehouse execution unchanged. Total savings: 32 seconds.

    Where the real win is: iterative development. You’re writing a model, trying to get the SQL right, going back and forth. In dbt Core, every iteration is a round-trip to the warehouse. In Fusion, every save gives you instant feedback in the IDE. No warehouse call. No wait. That’s where 30x matters. It’s not about total runtime; it’s about the feel of development.

    If your team is not iterating heavily (you write once, test once, deploy), Fusion’s speed boost feels small. If your team is constantly tweaking, testing incrementally, exploring SQL, the developer experience jump is massive.

    The one principle

    Fusion trades migration friction for developer velocity. It requires you to clean up deprecated code, update packages, be intentional about configuration. That’s work upfront. But on the other side, you get a development experience that’s fundamentally faster. It’s not a free upgrade; it’s a real investment that pays off if you’re writing and iterating regularly.

    If your team writes dbt code once and deploys it to production untouched, the friction might not be worth it. If your team is constantly refining, testing, exploring — which is most analytics teams — it is.

    Related reading: Upgrading to dbt Fusion (official docs) · Migrate to the Latest YAML Spec · Snowflake Streams & Tasks: SCD2 Pipeline Guide · Snowflake Query Cost Estimator

  • Snowflake Iceberg V3: When to Actually Migrate(vs Native Tables)

    Snowflake Iceberg V3: When to Actually Migrate(vs Native Tables)

    Most data engineers I talk to still store everything in Snowflake native format. It’s simple: load data, query data, done. But here’s what nobody’s talking about: if you’re querying that data from anywhere else — Spark, Databricks, even just a local Python script — you’re paying a hidden “data tax.” Redundant storage, egress fees, ETL pipeline complexity. For Fortune 500 companies, that tax runs $2 million to $7 million a year. And Snowflake’s new Apache Iceberg v3 support (GA May 2026) actually changes the math. But migrating is a choice, not a reflex — and there are specific gotchas that’ll bite you if you don’t plan right.

    The honest decision tree: migrate if you’re paying egress fees or running multi-engine queries.

    TL;DR

    → Apache Iceberg v3 is GA on Snowflake (May 2026). New features: deletion vectors (10x faster DML), row lineage for CDC, VARIANT type for semi-structured data, nanosecond timestamps, default column values.

    → Month-to-month costs are roughly equal to native Snowflake tables (compute identical, storage ~$23/TB native vs ~$0.023/GB S3, negligible difference).

    → Migrate if: (a) you query from Spark/Databricks (egress fees kill you), (b) you’re paying >$500/month for Snowflake storage, (c) you want single copy of truth across multiple engines.

    → Don’t migrate if: you only query from Snowflake, storage bill is small, and you’re not building a multi-engine architecture.

    → New gotcha: You can’t upgrade v2 tables in-place to v3. No writing to v3 tables via external engines (Spark) yet. External engine compaction gets billed starting May 21, 2026.

    → Real win: Snowflake Storage for Iceberg (GA April 2026) means you don’t manage S3 buckets. Snowflake handles it, with Fail-safe recovery built in.

    → The “data tax” of $2M–$7M annually on Fortune 500 costs more than Iceberg migration ever will.

    The mental model that’s keeping you locked in

    Here’s the picture most teams hold: Snowflake stores data. We query it in Snowflake. Done. Native tables, simple syntax, life is easy. And if you only query in Snowflake, that model works fine. You get the speed, the simplicity, the integration with dbt, the Time Travel.

    But the moment you have data living in two systems — Snowflake for reporting, Spark for ML training, Databricks for a BI tool, even just a DuckDB instance on your laptop — you’ve broken the simple model. Now you have two copies of the data, or worse, a pipeline that’s constantly syncing between them. You’re paying Snowflake egress fees to get data out ($0.02 per GB across regions, $0.08 between clouds). You’re rebuilding the same transformation logic in both systems. You’re managing schema evolution in two places. The complexity compounds.

    Iceberg was built to solve exactly this. One copy of the data, on open cloud storage (S3, Azure Blob, GCS), readable by any engine that supports the Iceberg format. Snowflake, Spark, Databricks, Trino, DuckDB. All of them see the same table, the same schema, the same snapshot. No replication, no egress fees, no syncing.

    But Iceberg isn’t free. It trades simplicity for flexibility. And for teams that genuinely don’t need that flexibility, native tables are still the right call.

    The hidden cost of locking data into proprietary formats. For large teams, it’s massive.

    What changed in Iceberg v3, and why it matters

    Iceberg v2 shipped in 2023 and covered the basics: open format, ACID transactions, schema evolution, snapshots. v3 (released June 2025, GA on Snowflake May 7, 2026) added seven new capabilities. Only three actually change how you’d use it.

    Deletion vectors. In v2, if you deleted or updated a row, Iceberg had to rewrite the entire data file (copy-on-write). Slow and expensive. v3 adds deletion vectors — a separate, small metadata file that marks rows as deleted without touching the original data. Result: 10x faster DML operations on large tables. If you’re doing frequent small updates (common in streaming ingestion), v3 matters.

    Row lineage. v3 tracks which rows were inserted, updated, or deleted with metadata fields (_row_id, _last_updated_sequence_number). This is how Snowflake implements change data capture (CDC) without external tooling. A Dynamic Iceberg Table can now refresh incrementally on only the rows that changed, not the whole partition. Critical for SCD2 and CDC pipelines.

    VARIANT type. v2 forced you to choose: store JSON as a string (slow parsing at query time) or explode it into a wide schema (thousands of nullable columns, query disasters). v3 adds native VARIANT support, and Snowflake automatically shreds it (extracts nested fields and indexes them) at write time. Query performance on semi-structured data jumps dramatically. This alone is why observability platforms are betting on Iceberg.

    The other four (default column values, geometry/geography types, nanosecond timestamps, partition transform improvements) are niche. Don’t worry about them unless you hit them.

    The cost math: Native vs Iceberg in real dollars

    Let’s be honest: most articles skip the cost comparison and jump to “Iceberg is cheaper!” It usually isn’t, month-to-month. Here’s why.

    Two-column cost breakdown. Native Snowflake: 2,000 credits at $3 = $6,000 compute, $23/TB storage = $230, total $6,230/month. Iceberg (Snowflake managed): same $6,000 compute, S3 at $0.023/GB = $235 storage, bundled compaction = $0, total $6,235/month. Verdict: same cost, but Iceberg enables multi-engine and zero egress.

    The real numbers. On a month-to-month basis, they’re nearly identical. The wins come from elsewhere.

    For a typical 10 TB table with 1,000 queries per month (small-to-medium workload):

    Native Snowflake: Compute 2,000 credits ($6,000) + Snowflake storage 10TB at $23/TB ($230) = $6,230/month.

    Iceberg (Snowflake-managed storage, GA April 2026): Compute 2,000 credits ($6,000) + S3 storage 10TB (10,240 GB × $0.023/GB = $235) + compaction bundled ($0) = $6,235/month.

    Basically the same. Where Iceberg wins is not in monthly costs. It wins in:

    Egress fees. If you query that 10 TB table from a Databricks cluster once a month, native Snowflake costs 10,000 GB × $0.08/GB (cross-cloud egress) = $800. Iceberg: $0. Over a year, that’s $9,600. At any real-world scale (multi-engine queries), egress dominates.

    No data duplication. If you’re currently syncing data between Snowflake and Databricks (ETL pipeline, manual export, Fivetran), that pipeline costs money too. Shared Iceberg table means you stop paying to move the data. One table, multiple readers.

    Storage simplicity. With Snowflake Storage for Iceberg (new, April 2026), you don’t manage S3 buckets yourself. Snowflake handles encryption, replication, Fail-safe recovery. You save the operational tax of bucket management, lifecycle policies, and debugging storage issues.

    So here’s the honest scorecard:

    For Snowflake-only users: Native tables win. Simpler, no migration pain, costs are identical.

    For multi-engine shops (Snowflake + Spark + Databricks): Iceberg wins. Egress fees alone justify the migration, and you get single source of truth as a bonus.

    The gotchas that will hurt your migration

    You can’t upgrade v2 tables to v3 in-place. There’s no ALTER TABLE ... SET ICEBERG_VERSION = 3. To get v3, you have to CREATE a new table. That means copying data (compute cost, time), repointing your queries, and hoping nothing breaks downstream. On large tables, this is a multi-day operation.

    External engines can’t write v3 tables yet. You can read v3 tables from Spark, Trino, DuckDB, all day. But writing is blocked. Snowflake says it’s “planned,” but if you’re building a shared Iceberg table that Spark needs to update, you’re stuck on v2. This is a major limitation if you’re counting on true multi-engine write access.

    Compaction gets billed starting May 21, 2026. When an external engine writes to an Iceberg table (via Spark, Trino, etc.), it creates small data files. Snowflake’s compaction automatically consolidates them into bigger files for query performance. Until May 21, that was free. Now it costs credits. Budget for ongoing compaction maintenance if you have heavy external write workloads.

    ⚠️ Don’t convert cloned tables with vended credentials. If you clone a native Snowflake table and then convert it to Iceberg, you can’t write to it with vended credentials (external query engine creds). You’d have to connect the external engine directly to your S3 bucket, defeating the whole point. Create the Iceberg table fresh if you’re using vended creds.

    Schema changes are cheap but metadata bloat is real. Iceberg tracks every schema change as a separate metadata version. On tables with thousands of ALTER COLUMN operations, metadata can get unwieldy. Compact your metadata regularly with CALL SYSTEM$OPTIMIZE(...).

    The mistakes teams make when migrating

    1. Migrating for the wrong reason. “Everyone’s talking about Iceberg, so we should move.” Wrong. Migrate only if you have a concrete use case: egress fees, multi-engine queries, or storage cost >$500/month. Otherwise you’re trading simplicity for nothing.

    2. Not testing external engine read performance first. Iceberg’s query performance depends heavily on your cloud setup, partitioning strategy, and how many small files are sitting around. Test Spark/Databricks queries on a small Iceberg table before migrating your 100 TB production table. You might find that your workload is slower on Iceberg, not faster.

    3. Assuming v3 is backward-compatible with v2. It’s not. Engines that only understand v2 (like older Spark runtimes, Trino versions) will fail on v3 tables. Check that every tool in your stack supports v3 *before* upgrading. v2 → v3 is one-way; there’s no downgrade.

    4. Ignoring the partition evolution story. Iceberg lets you change your partitioning scheme without rewriting the whole table. It’s a huge feature, but it’s also easy to mess up. Bad partitioning (e.g., partitioning by a column with 10 million distinct values) creates a partition explosion. Get your partitioning right before you migrate, not after.

    5. Migrating everything at once. Pick one critical table, migrate it, test multi-engine queries for a month, then move the rest. Iceberg is mature enough for production, but it’s not old enough that every edge case is documented. Be intentional.

    When to actually migrate: The real decision

    Stop and ask yourself: Do you actually need Iceberg?

    Yes, if: You query the same data from Snowflake and Spark/Databricks. You’re paying egress fees. You have data warehouses in multiple clouds and want to query across them. You’re building a data lakehouse and want to ditch proprietary formats.

    No, if: You only query from Snowflake. Your storage bill is <$500/month. You’re using Snowflake’s Time Travel, zero-copy clones, and other native features heavily. You don’t need to share data with other engines.

    For most teams, the answer is no. And that’s okay. Native Snowflake tables are extremely good. Simple, fast, well-integrated with dbt. There’s no shame in staying native.

    But for teams hitting the “data tax” — redundant copies, egress fees, multi-engine complexity — Iceberg v3 actually delivers. The gotchas are real, but they’re manageable. The cost savings are modest month-to-month, but the flexibility is transformative.

    The one principle that matters

    Interoperability beats simplicity when you’re already paying for fragmentation. If your current architecture already costs you $800/month in egress, $300/month in ETL pipelines, and engineering time chasing sync issues, Iceberg’s “complexity” is actually a simplification. You’re not adding complexity; you’re replacing it with a standard.

    If you’re simple and integrated today, stay there. Don’t pay the cost of flexibility you don’t need. But if you’re paying the data tax, Iceberg’s math changes fast.

    Related reading: Snowflake Apache Iceberg tables (official docs) · Snowflake Time Travel: The Real Architecture · Snowflake Optima: 15x Faster Queries at Zero Cost · Query Snowflake in DuckDB and Cut Costs

  • Snowflake Query Execution: what really happens under the hood

    Snowflake Query Execution: what really happens under the hood

    Ask ten data engineers what happens when you run a query in Snowflake and most of them will tell you the same thing: the warehouse runs it. SQL goes in, the warehouse chews on it, results come out. Clean, simple, and just wrong enough to cost you money. The truth is that Snowflake query execution is a trip through three separate layers, and the one you actually pay for is the last to get involved — if it gets involved at all.

    Flowchart of the life of a Snowflake query: query submitted, cloud services layer parses and prunes, result cache check returns a cached result on a hit with zero credits, and on a miss the virtual warehouse runs it across MPP nodes before fetching micro-partitions from storage and returning results.

    The path every query takes — and the shortcut a cache hit gets to skip compute entirely.

    TL;DR

    → Running a query isn’t one step. It’s a trip through three layers that scale independently, and the compute you pay for is the last one to wake up.

    → The cloud services layer does the thinking first — parses, plans, and prunes — using per-partition metadata to throw out data that can’t match your filter before a single byte is read.

    → If the exact same query ran in the last 24 hours and the data hasn’t moved, you get a cached result for free. No warehouse. No credits. Milliseconds.

    → Only on a cache miss does a virtual warehouse actually spin up and crunch the query across its nodes in parallel.

    → Warehouses keep recently-read micro-partitions on local SSD, but that cache vanishes the second the warehouse suspends.

    → “Identical” is brutally literal. A stray table alias, lowercase keywords, or a RANDOM() call quietly knocks you off the cache and back onto billed compute.

    → When a query is slow, it’s usually scanning too much, not computing too much. A bigger warehouse fixes the second problem and never the first.

    The mental model that’s quietly costing you money

    Here’s the reframe that changes how you think about every query: the virtual warehouse is the last thing to get involved, and plenty of queries never touch it at all. Almost everything that decides whether your query is fast or slow, cheap or expensive, happens in a layer you probably weren’t even picturing.

    Get this wrong and you end up doing what I’ve watched teams do a dozen times: a dashboard feels sluggish, someone bumps the warehouse from Medium to Large, it gets a little faster, everyone moves on. The bill goes up. The query was never compute-bound in the first place. They paid more to run the same broken query faster.

    So let’s actually follow a query, from the moment you hit run to the moment rows come back.

    The three layers a query passes through

    Snowflake splits into three layers, and the thing to internalize is that they scale completely independently. That’s not a trivia fact — it’s the reason the whole platform behaves the way it does. Every query touches all three. Some only touch the first.

    Diagram of Snowflake’s three layers: the cloud services layer handling parse, optimize, prune and result cache at the top; the compute layer with virtual warehouse MPP nodes in the middle; and the storage layer holding micro-partitions in S3, Azure Blob or GCS at the bottom, with a query entering the top and a cached result returning from it

    The cloud services layer is the brain. The warehouse is just the muscle.

    At the top sits the cloud services layer — the brain. Logins, sessions, parsing, query planning, transaction coordination, all the metadata: it lives here. Underneath that is the compute layer, made up of virtual warehouses. These are the MPP clusters that do the actual SQL grunt work. And at the bottom is storage: your data sitting in cloud object storage (S3, Azure Blob, or GCS) as immutable, compressed, columnar micro-partitions.

    The magic is that none of these share resources with each other. You can resize compute without moving a single byte of storage. You can point ten warehouses at the same table and none of them slow the others down. Keep that in your back pocket, because it’s exactly why the “just make the warehouse bigger” instinct fails so often.

    Step 1: The cloud services layer does the thinking

    Your query lands, and before any table data gets read, the cloud services layer goes to work.

    It parses the SQL, and the cost-based optimizer builds a plan. Then it does the single most important thing for performance, and it does it for free: partition pruning. Every micro-partition carries metadata, including the min and max value for each column. So when you write WHERE order_date = '2026-06-01', the optimizer reads that metadata first and skips every partition whose range can’t possibly hold a matching row. Ten thousand partitions in the table, three that actually get scanned. No index. No tuning. You didn’t do anything.

    This is also why a query can come back in milliseconds with no warehouse at all. Run a SELECT COUNT(*) or a MAX() and Snowflake often answers straight from the metadata cache — you’ll see a lone METADATA-BASED RESULT step in the query profile and zero compute on the bill.

    How good the pruning is comes down to how well your data is naturally ordered, which is the whole point of clustering and Snowflake’s automatic optimization features.

    Step 2: The result cache check, before any compute

    Now, still before any warehouse gets involved, cloud services checks the result cache. Identical query in the last 24 hours, data hasn’t changed since? Snowflake just hands back the stored result from the cloud services layer. Nothing resumes. Nothing gets billed. The answer is basically instant.

    This is the cheapest query you’ll ever run, and it’s the reason refreshing a dashboard a second time costs nothing. Here’s the part that surprises people: every time that cached result gets reused, the 24-hour clock resets. Keep hitting it and a result can stay alive for up to 31 days. A popular dashboard query can effectively sit in cache for a month and never cost a credit.

    Which brings us to the catch. “Identical” is doing a lot of work in that sentence, and it’s the thing nobody reads about until they’re staring at a bill wondering where the money went.

    Step 3: The virtual warehouse finally runs it

    Cache miss. Now — and only now — the query goes to a virtual warehouse. If it was asleep, it resumes (a second or two, usually). The warehouse is a little cluster of compute nodes, and this is where massively parallel processing earns its name: the surviving micro-partitions get split across the nodes and cores, and the filters, joins, and aggregations all happen in parallel.

    This is the layer with a meter running. You’re billed by the second, with a 60-second minimum every time a warehouse starts or resumes. That minimum trips people up constantly — a warehouse that keeps flickering on and off for tiny one-off queries can quietly cost more than one you just leave warm, because every single resume restarts that 60-second clock. It’s also the only layer where resizing actually helps, and that matters less often than you’d think.

    Step 4: Storage, and the cache that disappears

    The warehouse goes and fetches the micro-partitions that survived pruning. Each micro-partition holds 50–500 MB of uncompressed data in a columnar format, so only the columns you actually asked for get read. Ask for two columns out of fifty and you pay to read two.

    And there’s a second cache down here. As the warehouse pulls partitions from remote storage, it stashes them on local SSD. Next query on that same warehouse that needs the same data? It reads from SSD instead of making the round trip to object storage, which is a lot faster. This is what people mean when they say a warehouse “warms up.”

    The catch — and there’s always a catch — is that this cache gets wiped the instant the warehouse suspends. That’s the real trade-off hiding inside your AUTO_SUSPEND setting. Suspend fast and you stop paying for idle time but you toss the warm cache. Leave it running and you keep the cache but pay for the idle. A lot of teams land on 60 seconds, but honestly the right answer depends entirely on how often your queries actually fire.

    Three caches, one comparison to bookmark

    There are three caches, they live in different layers, and mixing them up is behind half the “wait, why was that slow?” conversations I’ve ever had.

    CacheLayerWhat it storesSurvives suspend?Compute cost
    Result cacheCloud servicesFinal query result setsYes (24h, resets on reuse, 31-day max)None
    Metadata cacheCloud servicesRow counts, min/max, distinct countsYesNone
    Local disk (warehouse) cacheComputeMicro-partitions read from storageNo — purged on suspendWarehouse already running

    The two cloud-services caches are shared across the whole account — every warehouse, every user benefits. The local disk cache belongs to one warehouse and dies with it. If you want the gory details with real query-profile screenshots, Snowflake’s own community piece on caching is the place to go.

    The cache rules nobody warns you about

    The result cache only kicks in when Snowflake decides the new query is identical to the old one, and “identical” is far more literal than anyone expects. Snowflake’s own docs spell it out: add a table alias, or just retype your keywords in lowercase, and you miss the cache. The query goes to billed compute and you never get a warning.

    It also bails on the cache if the query contains a non-deterministic function — UUID_STRINGRANDOMRANDSTR are the usual suspects — or an external function, or if it reads from a hybrid table. And the role running it needs privileges on every table involved.

    ⚠️ What this means in practice: if you want your dashboards and scheduled jobs riding the cache, lock down the exact text your BI tool emits. Random aliasing, dynamic comments, an injected session variable, a stray current_timestamp() — any of it silently drops you off the cache, and you start paying to recompute answers you already had. The Query Cost Estimator is a quick way to put a number on what those misses are costing you.

    Why a bigger warehouse usually isn’t the answer

    This is the most expensive misunderstanding in all of Snowflake, so it’s worth being blunt about it. A query is slow for one of two reasons: it’s scanning too much data, or it’s doing too much actual computation. Resizing the warehouse only helps the second one.

    When the real problem is bad pruning — Snowflake had to scan most of the table because the data wasn’t laid out in a way it could skip — a bigger warehouse just reads the same mountain of data faster and charges you more per second for the privilege. The fix isn’t more nodes. It’s better pruning: clustering, a tighter filter, a smarter query shape. The query profile tells you which world you’re in. Look at partitions scanned versus partitions total. If you’re chewing through 95% of them, no warehouse size on earth is going to save you.

    And sometimes the honest answer to a giant, I/O-heavy scan isn’t a bigger Snowflake warehouse at all — it’s moving that work somewhere cheaper, which is the whole idea behind running the query in DuckDB instead. If you do decide a resize is warranted, the warehouse sizing guide will at least keep you from overshooting.

    What I actually check when a query misbehaves

    When something’s slow or weirdly expensive, here’s where I start. All of it comes out of the query profile and INFORMATION_SCHEMA — no guessing, no vibes.

    -- Did your last query hit the result cache?
    -- A cache hit shows bytes_scanned = 0 and a near-zero execution time.
    SELECT query_id,
           query_text,
           bytes_scanned,
           percentage_scanned_from_cache,
           execution_time / 1000 AS exec_seconds
    FROM TABLE(information_schema.query_history())
    ORDER BY start_time DESC
    LIMIT 10;
    
    -- Turn the result cache off for honest benchmarking (this session only)
    ALTER SESSION SET USE_CACHED_RESULT = FALSE;
    -- ...run your test queries...
    ALTER SESSION UNSET USE_CACHED_RESULT;
    
    -- Hunt down your worst pruning offenders
    SELECT query_id,
           partitions_scanned,
           partitions_total,
           ROUND(100 * partitions_scanned / NULLIF(partitions_total, 0), 1) AS pct_scanned,
           bytes_scanned / 1e9 AS gb_scanned
    FROM snowflake.account_usage.query_history
    WHERE partitions_total > 0
    ORDER BY bytes_scanned DESC
    LIMIT 20;
    
    -- Balance warm cache against idle cost
    ALTER WAREHOUSE analytics_wh SET AUTO_SUSPEND = 60;  -- seconds

    The number to stare at is pct_scanned in that third query. High on your slow queries? You’ve got a pruning problem, and that’s where your time should go — not the warehouse dropdown.

    What this actually costs you

    Architecture is interesting, but the bill is what makes it matter. So let’s put real numbers on it. Credit prices swing with your edition, cloud, and region, so treat these as illustrative — drop your own rate into the cost calculator for the exact figure.

    Picture one ordinary dashboard query. It runs on a Medium warehouse (4 credits an hour, call it roughly $3 a credit), and it fires 100 times a day because that’s how often people open the dashboard. Here’s how the same query plays out depending on whether the layers are doing their job:

    ScenarioWhat actually runs~Cost / day~Cost / month
    No cache, bad pruning (30s/run, scans ~95% of partitions)100 full compute runs~$10~$300
    Result cache hits 90% of the time10 compute runs, 90 free cached returns~$1~$30
    Cache + good pruning (3s/run, scans ~5%)10 runs at a tenth of the scan~$0.10~$3

    Same query. Same data. Roughly $300 a month versus $3, and the only difference is whether you let the result cache and pruning do what they’re built to do. Now multiply that across a dashboard with thirty queries on it, and you can see how a Snowflake bill quietly triples without anyone writing a single new query.

    The gotchas nobody warns you about

    The 60-second minimum punishes spiky workloads. A query that finishes in two seconds still bills a full minute if the warehouse had to resume to run it. Set AUTO_SUSPEND too aggressively on a workload that fires a query every couple of minutes and you’ll pay more in cold-start minimums than you ever saved on idle time. Aggressive suspend is not automatically cheaper.

    One write nukes the whole result cache. The cache is invalidated by any change to the underlying table, not just changes to the rows your query touched. A single late-arriving record at 2 a.m. quietly wipes the cached result for every dashboard query built on that table, and tomorrow morning they all run on compute again. If your “cached” dashboard mysteriously costs money some mornings, this is usually why.

    Cloud services is free, right up until it isn’t. The parsing, planning, and pruning in the cloud services layer is free — but only while it stays under 10% of your daily warehouse compute. Hammer it with thousands of tiny metadata queries or relentless INFORMATION_SCHEMA polling and you cross that line, and Snowflake starts billing the overage. Most teams never hit it; the ones running huge volumes of trivial queries do, and they never see it coming.

    The local cache doesn’t follow you between warehouses. That warm SSD cache belongs to one specific warehouse. Run your ETL on one warehouse and your reporting on another — sensible workload isolation — and the reporting warehouse gets exactly zero benefit from the partitions ETL just pulled. Each warehouse warms its own cache from cold.

    SELECT * throws away the columnar advantage. Snowflake only reads the columns you ask for. Ask for all of them and you pay to read all of them, even when the dashboard displays three. In a wide table that’s the difference between scanning a few columns and dragging the entire row off storage.

    The mistakes that quietly drain the budget

    Almost every overspending Snowflake account I’ve looked at is making some combination of these five:

    1. Leaving AUTO_SUSPEND at the 600-second default. Ten full minutes of paid idle after every burst of activity. On a warehouse that’s used in short bursts, that idle time can dwarf the actual query time. Most analytics warehouses are fine at 60 seconds.

    2. One giant warehouse for everything. Pile ETL, ad-hoc analysis, and dashboards onto a single warehouse and they fight over the cache and the compute. Separate warehouses per workload keep each cache warm and each workload predictable.

    3. Scaling up when the real fix is pruning. The reflex we opened with. If pct_scanned is high, a bigger warehouse just burns more credits reading the same data. Fix the layout, not the size.

    4. Letting the BI tool emit sloppy SQL. Inconsistent aliases, injected timestamps, and per-user comments mean “the same” dashboard query is never byte-for-byte identical, so it never reuses the result cache. Standardize what the tool sends.

    5. Over-clustering a hot table. Automatic clustering isn’t free — it spends credits reorganizing data in the background. On a table that’s written constantly, that background churn can cost more than the queries it speeds up. Cluster the tables you read far more than you write.

    The one principle to take away

    If you remember nothing else, remember this: scan less, don’t compute faster. Snowflake decides almost everything about a query’s speed and cost before the warehouse ever wakes up — in the pruning, in the caches, in how your data is laid out. The warehouse size is the last lever you should reach for, not the first. Get the layers above it working and most “we need a bigger warehouse” conversations simply stop happening.

    Related reading: Snowflake Time Travel: The Real Architecture · Snowflake Optima: 15x Faster Queries at Zero Cost · Query Snowflake in DuckDB and Cut Costs · Snowflake Streams & Tasks: SCD2 Pipeline Guide

  • How the Warehouse Cache Actually Works in Snowflake

    How the Warehouse Cache Actually Works in Snowflake

    A dashboard that ran in four seconds on Monday took nineteen seconds on Tuesday. Same query, same data, same warehouse size. I spent the better part of an hour convinced Snowflake was having a bad day, checking the status page, refreshing query history, muttering about “platform issues” in our team Slack — before I noticed our DevOps script had quietly added an aggressive auto-suspend policy the night before. The warehouse cache was getting wiped every single morning, and I’d built our entire “fast dashboard” reputation on a cache that reset itself before anyone showed up to work.

    TL;DR

    • Snowflake’s warehouse cache (local disk cache) stores raw compressed micro-partition data on each node’s SSD, not query results.
    • It’s separate from the result cache and metadata cache — three different caches, three different jobs.
    • Auto-suspending a warehouse wipes this cache completely. Resuming starts it cold every time.
    • You can verify it’s working with the ‘percentage scanned from cache’ field in query profile or ACCOUNT_USAGE.QUERY_HISTORY.
    • Multi-cluster warehouses don’t share this cache — a query routed to a new cluster starts cold even if a sibling cluster is warm.
    • You can’t manually size or pin it. The only real lever you control is the auto-suspend timer.

    That mistake is what got me actually reading how this thing works instead of just trusting that Snowflake would “handle it.” Turns out the warehouse cache isn’t magic, isn’t tunable, and isn’t the same thing as the result cache most people learn about first. Here’s what it actually is, where it lives, and how to tell when it’s helping you versus quietly costing you money.

    Diagram showing a cloud data warehouse architecture with three layers: Cloud services (result and metadata cache), compute layer with three SSD nodes, and a database storage layer with immutable micro-partitions. Arrows indicate data flow.

    Three caches, one name people use for all of them

    Snowflake has three distinct caching layers, and the confusion starts because people say “Snowflake caches my query” without specifying which one did the work. The result cache sits in the cloud services layer and stores entire finished query results — if the same exact SQL text runs again within 24 hours and the underlying data hasn’t changed, you get the answer back with bytes_scanned = 0 and zero compute cost. The metadata cache, also in cloud services, holds statistics about every micro-partition — min and max values per column, row counts — so Snowflake can decide which partitions to skip before it ever touches the data.

    The warehouse cache is the third one, and it’s the one this article is actually about. It lives on the local SSD of every node in a running virtual warehouse, and it stores raw, compressed micro-partition data — not query results, not aggregated answers, the actual columnar bytes that got pulled from remote storage to answer a scan.

    Why this distinction actually matters

    If you only know the result cache exists, you’ll misdiagnose a lot of performance issues. Change one character in a WHERE clause, add a comment, swap the role running the query — any of those bypass the result cache entirely, because it requires an exact text match. The warehouse cache doesn’t care about query text at all. It cares about which micro-partitions a query needs and whether those bytes are already sitting on a node’s SSD from a previous scan.

    What a micro-partition actually is

    You can’t understand the warehouse cache without understanding the unit it stores. When data lands in a Snowflake table, it gets automatically carved into micro-partitions — contiguous, immutable blocks holding somewhere between 50MB and 500MB of uncompressed data each, stored in a columnar format. There’s no manual partitioning scheme to design, no index to build. Snowflake just does this on every load.

    Each micro-partition carries its own metadata: minimum and maximum values for every column, which is exactly what the metadata cache is built from. When you filter a query on order_date > '2026-01-01', Snowflake checks that metadata first and skips any micro-partition whose max date falls before that threshold. That’s partition pruning, and it happens before a single byte gets pulled into the warehouse cache. Pruning decides what to read; the warehouse cache decides how fast a repeat read of the same partitions will be.

    The actual lifecycle of the warehouse cache

    Here’s the sequence that matters in practice. A warehouse resumes from suspended state with completely empty SSD — there’s nothing cached because the compute nodes assigned to it are freshly provisioned. The first query that touches a table has to pull every relevant micro-partition from remote cloud storage, which is the slowest tier in the whole architecture. As those partitions get read, they’re written to the local SSD cache as a side effect — not because you asked for caching, just because that’s what happens when a node reads remote data.

    The second query — if it touches the same micro-partitions and the warehouse is still running — can read those bytes off local SSD instead of going back to remote storage. This is meaningfully faster, and it’s why a sequence of similar queries against the same table speeds up the longer a warehouse stays warm. There’s no explicit “build the cache” step. Cache population is a byproduct of usage, which is exactly why a single cold query tells you almost nothing about real-world performance.

    Then the warehouse suspends, and it’s gone

    This is the part that bit me. When a warehouse auto-suspends, the compute nodes it was using get released back into Snowflake’s shared pool. The SSD on those specific nodes goes with them. When the warehouse resumes — even if it’s seconds later, even if it’s the exact same warehouse name — there’s no guarantee you get the same physical nodes back, and the cache starts from zero regardless. There’s no persistence, no “save state before suspending.” It’s just gone.

    Resizing a warehouse up or down does the same thing. A different size means a different set of nodes, which means different SSDs, which means the next queries run cold no matter how warm things were five minutes earlier.

    How to actually see this working

    Stop assuming and go look at it. Open the query profile for any query in Snowsight and check the IO statistics panel — there’s a field literally called percentage_scanned_from_cache. Run the same query twice in a row on a warm warehouse and watch that number jump from near 0% to something much higher on the second run. Suspend the warehouse, resume it, run the same query again, and watch it drop back to 0%. That’s the entire mechanism, visible in about ninety seconds of testing.

    For a wider view across your account, query history gives you the same field at scale. This is the query I run when someone asks “is our caching even helping”:

    Check your real cache hit rate (last 30 days, by warehouse)

    SELECT
        warehouse_name,
        COUNT(*) AS query_count,
        SUM(bytes_scanned) AS bytes_scanned,
        SUM(bytes_scanned * percentage_scanned_from_cache) AS bytes_from_cache,
        SUM(bytes_scanned * percentage_scanned_from_cache)
            / SUM(bytes_scanned) AS pct_scanned_from_cache
    FROM snowflake.account_usage.query_history
    WHERE start_time >= DATEADD(month, -1, CURRENT_TIMESTAMP())
      AND bytes_scanned > 0
    GROUP BY 1
    ORDER BY 5;

    A low percentage here on a warehouse running frequent, similar queries is a signal — either your auto-suspend timer is too aggressive for the workload, or the queries aren’t actually similar enough at the data level to benefit from a warm cache, even if the SQL looks similar to a human reading it.

    Result cache, warehouse cache, metadata cache — side by side

    Cache LayerWhere It LivesWhat It StoresCleared WhenCompute Cost
    Result CacheCloud Services layerFull query results24 hours of inactivity, or DDL on underlying tablesZero — no warehouse needed
    Warehouse CacheSSD on each compute nodeRaw compressed micro-partitionsWarehouse suspends, resizes, or node is replacedWarehouse must be running
    Metadata CacheCloud Services layerMin/max values, row counts, partition statsRarely — persists with the tableZero — used for pruning before scan

    The auto-suspend tradeoff nobody explains clearly

    Snowflake’s own guidance generally points toward short auto-suspend windows to control credit spend, and for spiky, unpredictable workloads that’s the right call. But if a warehouse runs frequent, similar queries back-to-back — a BI tool polling dashboards, an analyst iterating on the same fact table — an aggressive suspend timer means you’re paying the “cold scan” tax on nearly every query, because the cache never gets the chance to stay warm between them.

    The fix isn’t complicated once you see the tradeoff: separate warehouses by access pattern. A reporting warehouse that gets hit constantly during business hours can run a longer suspend window, or stay up during known peak hours, while a warehouse running sporadic ad-hoc analyst queries can suspend aggressively without losing much, since the cache wasn’t going to be useful between unrelated queries anyway.

    The multi-cluster gotcha

    If you’re running a multi-cluster warehouse for concurrency, know that clusters don’t share this cache with each other. Cluster A being fully warm doesn’t help a query that gets routed to Cluster B when Snowflake spins up a new cluster to handle a concurrency spike. That new cluster starts cold, scans from remote storage, and only builds its own local cache from that point forward. Teams chasing consistent query latency under high concurrency often get surprised by this — the warehouse “should” be warm, and on average it is, but any individual query can still land on a cold cluster.

    What the warehouse cache does not do

    It’s worth being precise about the boundaries here, because I’ve seen this cache get credited for things it isn’t responsible for. It doesn’t store query results — that’s the result cache’s job, and it’s a different layer entirely with a different lifetime. It doesn’t help with intermediate computation that spills to local disk during a large sort or hash join — that’s a separate spillage mechanism tracked under bytes spilled to local storage in query profile, not the same SSD allocation conceptually even though it physically lives in a similar place. And it provides no benefit on the write path for a fresh INSERT into new micro-partitions, since there’s nothing previously cached to reuse.

    What’s actually worth doing about this

    You don’t get a dial to resize this cache or pin specific tables into it, so the practical levers are all about behavior, not configuration. Match auto-suspend timers to actual access patterns instead of using one default across every warehouse. If a workload is genuinely cache-sensitive — recurring dashboards, iterative analyst sessions — consider a short warm-up query immediately after resume rather than letting the first real user query eat the cold-start cost. And when you’re debugging a “why did this get slower” ticket, percentage_scanned_from_cache should be one of the first three things you check, right alongside partition pruning stats, before you start blaming the query itself.

    For the deeper mechanics of how partition pruning interacts with clustering keys, the official Snowflake documentation on warehouse cache optimization is worth reading directly — it’s one of the rare vendor docs pages that actually shows the diagnostic query instead of just describing the concept. The micro-partitions and clustering documentation is the right follow-up if pruning efficiency turns out to be your actual bottleneck instead of cache temperature.

    Does the Snowflake warehouse cache get cleared when the warehouse suspends?

    Yes, completely. The warehouse cache lives on the SSD of the compute nodes assigned to that warehouse, and those nodes get released back to the pool on suspend. When the warehouse resumes — even seconds later — it’s starting from zero cached data.

    Why does percentage_scanned_from_cache show 0% on a brand new warehouse?

    Because there’s nothing to scan yet. The first query against any table after a cold start has to pull every micro-partition it needs from remote storage. Cache population happens as a side effect of running queries, not in advance.

    Does the warehouse cache help with INSERT, UPDATE, or DELETE performance?

    📷 the SSD cache layer, mid-rebuild after a warehouse resume — not glamorous, but it’s where the speed comes from

  • Everyone Said SQL Was Dead. It’s Now the Most Valuable Skill in AI (2026)

    Everyone Said SQL Was Dead. It’s Now the Most Valuable Skill in AI (2026)

    In 2018, a wave of Medium posts declared SQL obsolete. NoSQL was the future. Python would handle everything. Data lakes would make relational thinking irrelevant. The hot take had a good run.

    Then AI happened — and SQL came back harder than ever.

    Today, SQL is the connective tissue of every serious AI data stack. It feeds the training pipelines that power large language models. It validates the outputs of ML systems. It runs inside every dbt transformation, every Snowflake query, every Airflow DAG that touches structured data. And in 2026, the rise of text-to-SQL AI agents means that understanding SQL deeply is now more important than ever — not less.

    TL;DR

    For years, pundits called SQL a dying skill. They were wrong. In the AI era, SQL is experiencing a full renaissance — powering LLM pipelines, text-to-SQL agents, dbt models, and Snowflake-backed AI workflows. Senior data engineers with strong SQL command salaries up to $179K. Here’s why SQL is now the most career-defining skill in tech.

    Infographic with four stats: $179K senior data engineer max salary, 150K+ data engineering professionals, 20K+ new jobs created in past year, and 69% of job postings require SQL.

    The Death of SQL Was Always a Myth

    The “SQL is dying” narrative was never based on actual hiring data. It was based on hype cycles. Every new database technology generated thinkpieces about how SQL would be replaced — first by MapReduce, then document stores, then graph databases, then vector DBs.

    None of it displaced SQL as the default language of data work. And there’s a structural reason for that: relational thinking maps directly to how business data is structured. Revenue by region. Users by cohort. Transactions by date. These aren’t graph problems or document problems — they’re table problems, and SQL solves them with surgical precision.

    What the doomsayers missed is that SQL doesn’t compete with new technologies — it sits on top of them. Snowflake runs SQL. BigQuery runs SQL. Delta Lake and Apache Iceberg are queried with SQL. Even Snowflake’s AI features are invoked through SQL-adjacent interfaces.

    “SQL is eternal — it’s the new English of data systems.”

    Why AI Made SQL More Valuable, Not Less

    Here’s the counterintuitive reality: the rise of AI has created more demand for SQL, not less. There are three reasons why.

    1. LLMs Speak SQL

    The text-to-SQL category — where natural language queries get translated into executable SQL — is one of the fastest-growing areas in AI tooling. Tools like Vanna.ai, DataGrip’s AI Assistant, and BlazeSQL are putting SQL generation in the hands of non-technical users.

    But here’s the catch: AI-generated SQL still needs a human expert to validate it. A model hitting 80–85% accuracy on clean data sounds impressive until you realize that the 15% failure rate in production can silently corrupt dashboards, ML training sets, and financial reports. Someone with deep SQL knowledge has to own that validation layer.

    2. AI Models Are Trained on SQL Pipelines

    Every serious ML workflow has a data preparation layer. That layer runs on SQL. Whether it’s dbt transformations cleaning feature tables, Snowflake views materializing training datasets, or window functions creating temporal sequences for time-series models — SQL is the engine underneath.

    A data engineer who can write optimized SQL is not just a “database person.” They’re the person keeping AI models from training on garbage data. That’s a mission-critical role in 2026.

    3. The Semantic Layer Runs on SQL

    As AI agents get wired into data stacks, the “semantic layer” — a metadata-rich translation between business concepts and database schemas — has become critical infrastructure. dbt’s Semantic Layer, Snowflake’s Cortex, and tools like Cube.js all expose this layer through SQL-compatible interfaces. Understanding SQL deeply is what lets engineers build and maintain this layer correctly.

    The Modern SQL Skill Set Is Not What You Learned in 2015

    Basic SELECT * FROM table fluency is table stakes. What the market pays a premium for in 2026 is a completely different tier of SQL mastery.

    WITH user_activity AS (
      SELECT
        user_id,
        event_date,
        revenue,
        SUM(revenue) OVER (
          PARTITION BY user_id
          ORDER BY event_date
          ROWS BETWEEN 6 PRECEDING AND CURRENT ROW
        ) AS rolling_7d_revenue,
        DATEDIFF('day', MAX(event_date) OVER(PARTITION BY user_id), CURRENT_DATE())
          AS days_since_last_event
      FROM events
      WHERE event_date >= DATEADD('day', -90, CURRENT_DATE())
    ),
    
    churn_signals AS (
      SELECT
        user_id,
        event_date,
        rolling_7d_revenue,
        days_since_last_event,
        -- Flag users with declining revenue trend
        CASE
          WHEN rolling_7d_revenue < LAG(rolling_7d_revenue, 7)
               OVER(PARTITION BY user_id ORDER BY event_date) * 0.7
          THEN 'HIGH_RISK'
          WHEN days_since_last_event > 14 THEN 'MEDIUM_RISK'
          ELSE 'LOW_RISK'
        END AS churn_risk
      FROM user_activity
    )
    
    SELECT * FROM churn_signals
    WHERE event_date = CURRENT_DATE() - 1
    ORDER BY rolling_7d_revenue DESC;

    This is what premium SQL work looks like in 2026: window functions generating ML features, CTEs composing complex business logic, and analytical patterns that feed directly into AI systems. It’s not query writing — it’s data architecture expressed in SQL.

    SQL vs. Python: The False Choice That Hurt Careers

    One of the most damaging career myths of the past decade was that SQL and Python were competing skills — as if choosing one meant abandoning the other. That binary thinking led many engineers to underinvest in SQL in favor of chasing Python frameworks, only to find that the highest-value data work required both.

    The truth is more nuanced. Python and SQL are complementary tools with clear division of labor in a modern data stack:

    TaskBest ToolWhy2026 Demand
    Data transformation at scaleSQL (via dbt)Declarative, version-controlled, warehouse-nativeVery High
    Feature engineering for MLSQL + PythonSQL for aggregations, Python for model inputsVery High
    Pipeline orchestrationPython (Airflow/Prefect)DAG logic, branching, retriesVery High
    Ad-hoc data explorationSQLFaster iteration, no environment setupHigh
    Real-time stream processingSQL (Flink/Kafka SQL)Streaming SQL increasingly the standardVery High
    Custom ML model trainingPythonscikit-learn, PyTorch, TensorFlowHigh
    Data quality & validationSQL (dbt tests)Schema-aware, automated, CI/CD-friendlyVery High
    Semantic layer / metricsSQL (dbt Semantic Layer)Business logic lives in SQL modelsEmerging

    What the Job Market Is Actually Saying

    Forget the hot takes. Look at the data. Across job postings, interview processes, and salary surveys, the signal is consistent: SQL is the single most requested skill in data roles, and that demand is accelerating.

    365 Data Science’s 2026 job outlook report found that 69.3% of data analyst postings explicitly require domain expertise that includes SQL as a core component. Data analyst average salaries have risen to $111,000 — up $20,000 from 2025 — driven largely by this demand.

    For data engineers — who live in SQL even more deeply — the numbers are stronger. Motion Recruitment’s 2026 salary guide puts senior data engineer salaries between $147,000 and $179,000. The data engineering sector now employs over 150,000 professionals with more than 20,000 new jobs created in the past year alone.

    “A senior engineer who writes clean, efficient SQL will always be more valuable than a junior who can only configure tools.”

    SQL in the AI-Native Stack: Where It Lives Now

    The modern data stack has evolved, but SQL is woven through every layer of it. Here’s where SQL shows up in a production AI workflow today:

    dbt: SQL as Software Engineering

    dbt (data build tool) transformed SQL from ad-hoc query language into version-controlled, testable, documented software. With the dbt Semantic Layer now powering AI applications directly, SQL models are becoming the canonical source of business logic across the entire organization. Following the Fivetran-dbt Labs merger, the tool’s dominance in the enterprise is only growing.

    Snowflake Cortex: AI Features in SQL

    Snowflake’s Cortex AI suite — rebranded and expanded after Summit 2026 — exposes large language model capabilities through SQL functions. You can run sentiment analysis, text classification, and vector search directly in SQL queries. Engineers who know SQL well have immediate access to AI capabilities without switching tools.

    Apache Flink & Kafka SQL: Streaming Goes SQL-First

    Even the streaming world is going SQL-native. Flink SQL and Kafka’s KSQL bring declarative query patterns to real-time data. As Apache Flink becomes the standard for event-driven AI applications, SQL fluency extends seamlessly from batch to streaming workloads.

    Vector Databases & Hybrid Search

    The newest frontier: hybrid SQL + vector search. Platforms like Snowflake, PostgreSQL with pgvector, and Databricks now support semantic similarity search alongside traditional SQL filtering. The engineers who can combine WHERE clauses with cosine similarity thresholds are building the retrieval layers that power RAG-based AI applications.

    Advanced SQL Concepts Every AI-Era Engineer Must Know

    Being competitive in 2026 means going well beyond JOINs and GROUP BYs. These are the SQL concepts that separate senior engineers from the rest:

    ConceptUse Case in AI WorkflowsDifficulty
    Window FunctionsTime-series feature engineering, rolling metricsIntermediate
    CTEs & Recursive CTEsHierarchical data modeling, lineage graphsIntermediate
    Query Execution PlansOptimizing training dataset queries at scaleIntermediate
    Lateral Joins / UNNESTFlattening JSON/semi-structured ML input dataIntermediate
    Incremental MaterializationEfficient dbt models on large datasetsAdvanced
    Partitioning & ClusteringCost-optimized queries on petabyte warehousesAdvanced
    Vector / Similarity Search SQLRAG retrieval layers, semantic search pipelinesAdvanced

    The Text-to-SQL Trap: Why AI Makes Human SQL Experts More Important

    There’s a seductive argument that text-to-SQL tools will eventually replace SQL expertise. It’s wrong, and understanding why matters for your career strategy.

    The best text-to-SQL tools in 2026 achieve 70–85% accuracy on clean, well-documented schemas. On messy enterprise databases with ambiguous column names and undocumented business logic, that number drops to 50–70%. Even with a proper semantic layer, you top out around 95%.

    That 5–30% failure rate is not a rounding error. It’s the difference between a business decision based on correct revenue data and one based on a silently wrong join. And crucially — AI cannot validate its own SQL output against business intent. A human who understands both the domain and the query language has to do that.

    The engineers who understand SQL deeply are not threatened by text-to-SQL. They’re empowered by it. They can build the semantic layers that make AI-generated queries more accurate, catch the failures that automated tools miss, and govern the data contracts that the entire stack depends on.

    How to Build SQL Mastery That Pays in 2026

    If you want to position yourself in the premium tier of data engineering talent, here’s a practical progression:

    Foundation (Weeks 1–4)

    Master complex multi-table JOINs, aggregations with GROUP BY and HAVING, and subqueries. Get comfortable with the full range of JOIN types and understand when to use each. Practice on real datasets — not toy examples.

    Intermediate (Months 2–3)

    Deep dive into window functions: ROW_NUMBERRANKLAGLEADNTILE, and aggregate windows. Build comfort with CTEs for complex query decomposition. Start reading query execution plans in Snowflake or BigQuery.

    Advanced (Months 4–6)

    Learn how indexes and clustering keys affect performance at scale. Study how dbt compiles SQL and build production dbt models. Experiment with Snowflake Cortex SQL functions. Build a project that combines streaming SQL (Flink or Kafka SQL) with a batch warehouse layer.

    Expert (Ongoing)

    Build the semantic layer. Design data contracts. Validate AI-generated SQL. Architect the query patterns that power ML feature stores. At this level, SQL mastery translates directly into architecture decisions that affect every downstream system in the organization.

  • The Hidden Architecture Behind Snowflake Time Travel: Why It’s Not Really a Backup Feature

    The Hidden Architecture Behind Snowflake Time Travel: Why It’s Not Really a Backup Feature

    TL;DR

    → Time Travel is not a backup — it’s a versioned metadata pointer to immutable micro-partitions you already paid to store
    → Snowflake never overwrites data in place. Every UPDATE or DELETE creates new micro-partitions and marks old ones as expired
    → Standard edition gives you 1 day. Enterprise gives up to 90 days — but storage costs multiply fast on high-churn tables
    → After Time Travel expires, data moves to Fail-safe for 7 more days — but only Snowflake support can retrieve it
    → Zero-copy clones use the same micro-partition pointers — no extra storage until you diverge from the source
    → High-churn tables on 90-day retention can silently balloon your Snowflake bill by 10x


    The misconception that costs people money

    Most engineers who discover Time Travel think: “Great, we have backups.” That’s the wrong mental model — and it’s the one that leads to both security gaps and surprise storage bills. Time Travel is not a backup. It’s a metadata feature built on top of something Snowflake was already doing.

    Understanding why requires understanding how Snowflake actually stores data under the hood.

    How Snowflake stores data: micro-partitions

    Snowflake doesn’t store your tables as traditional database files. It stores them as micro-partitions — small, immutable, columnar files in cloud object storage (S3, Azure Blob, GCS), typically 50–500MB compressed each.

    The word immutable is the key. Snowflake never modifies a micro-partition once it’s written. Every micro-partition is a read-only snapshot of the data at the moment it was created. So what happens when you UPDATE a row? Snowflake writes a new micro-partition with the updated data and marks the old one as expired.

    Diagram showing data partitions before and after an update: before, partitions A and B are active; after, A is expired (for Time Travel), A* contains the updated row, and B remains active and unchanged.

    The old data doesn’t go anywhere immediately — it just gets a metadata flag saying ‘this version is no longer current.’ This is Copy-on-Write, and it’s the architectural foundation that makes Time Travel possible essentially for free.

    Time Travel isn’t a feature Snowflake built on top of backups. It’s a feature Snowflake built on top of an immutable storage model they were already using. The retained partitions are a side effect of how writes work — Time Travel just decides how long to keep them.

    What Time Travel actually is

    Time Travel is Snowflake’s metadata layer keeping pointers to those expired micro-partitions, instead of immediately flagging them for deletion. When you query with AT(TIMESTAMP => ...) or BEFORE, you’re not restoring from a backup. You’re asking Snowflake’s metadata layer to temporarily re-point to the expired partitions. The data was always there — you’re just re-routing the query to read older versions.

    This is why Time Travel queries are fast. There’s no restore process. No data movement. Snowflake reads directly from the older partitions.

    The three-zone model: Active, Time Travel, Fail-safe

    Understanding the full picture requires knowing all three zones data passes through after it’s written and then changed.

    A flowchart illustrates the three-zone data lifecycle: Active (current data, query anytime), Time Travel (1–90 days, billed, SQL access), Fail-safe (7 days, support only, Snowflake cost), and Gone (permanent, no recovery).

    Active data is what your current queries see — the live micro-partitions. Time Travel holds expired micro-partitions for your configured retention window. You can query this with SQL, clone from it, and UNDROP tables dropped within the window. Fail-safe activates when Time Travel expires — Snowflake keeps those partitions for 7 more days, but only Snowflake support can retrieve them. After that, data is permanently gone.

    Time Travel vs Fail-safe — the comparison you need

    FeatureTime TravelFail-safe
    Duration0–90 days (edition dependent)7 days (fixed, non-configurable)
    Who can accessYou — via SQL queriesSnowflake support only
    Query directlyYes — AT / BEFORE syntaxNo — support ticket required
    Clone fromYes — zero-copy clonesNo
    Storage costYes — counts against your billNo additional charge
    ConfigurableYes — per table/schema/databaseNo — always 7 days
    Best forOperational recovery, auditingLast-resort disaster recovery

    The storage cost nobody warns you about

    Every expired micro-partition kept for Time Travel counts against your Snowflake storage bill. The formula is brutal: a table with 90-day retention that sees 100% of its rows updated daily is storing 91 versions of itself simultaneously.

    Bar chart comparing storage multipliers for low churn (green) and high churn (orange) data over different retention periods (1, 7, 30, 90 days), showing high churn increases storage costs, especially at 90 days (30x).

    Most teams set 90-day retention on everything because the docs say Enterprise supports up to 90 days and more seems better. Then they get their first monthly storage invoice and start asking questions.

    ⚠️ The fix for high-churn tables: Set DATA_RETENTION_TIME_IN_DAYS = 0 on transient staging tables, session event tables, or any table where Time Travel has no operational value. You lose time travel on those tables, but you stop paying for micro-partitions you’ll never query.

    Zero-copy clones: same architecture, surprising implications

    Zero-copy clones work through the same micro-partition pointer mechanism. When you CREATE TABLE clone CLONE source, Snowflake doesn’t copy any data. It creates a new table object whose metadata points to the same micro-partitions as the source. Storage only diverges when you write new data to either the source or the clone.

    This is why ‘create a clone before a dangerous operation’ is nearly free — until you start modifying the clone. It’s also why clones on Time Travel windows are powerful: you can clone a table as it existed 7 days ago with zero storage cost at creation time.

    Why Time Travel is not a backup

    Account-level events affect everything. If your Snowflake account is compromised at the account level, or you accidentally drop the entire database, Time Travel data is in the same account. It’s not in a separate system.

    Cloud storage failure. Time Travel data lives in the same cloud storage as your active data. A regional disaster that takes out your Snowflake data takes out Time Travel with it.

    It expires. A backup you can restore from in 6 months is a backup. Time Travel data that’s gone after 90 days is a version history, not a backup. For genuine disaster recovery, you need cross-region replication or dedicated exports.

    Practical SQL patterns


    Here are the patterns I use most in production — from basic time travel queries to monitoring which tables are inflating your storage bill:

    -- Query a table as it existed yesterday
    SELECT * FROM orders
      AT(TIMESTAMP => DATEADD(DAY, -1, CURRENT_TIMESTAMP()));
    
    -- Query using a specific offset in seconds
    SELECT * FROM orders
      AT(OFFSET => -3600);  -- 1 hour ago
    
    -- Restore a dropped table
    UNDROP TABLE orders;
    
    -- Clone a table from 7 days ago (zero-copy, no extra storage)
    CREATE TABLE orders_snapshot
      CLONE orders
      AT(TIMESTAMP => DATEADD(DAY, -7, CURRENT_TIMESTAMP()));
    
    -- Check Time Travel storage usage by table
    SELECT table_name,
           active_bytes / 1e9         AS active_gb,
           time_travel_bytes / 1e9    AS time_travel_gb,
           failsafe_bytes / 1e9       AS failsafe_gb
    FROM information_schema.table_storage_metrics
    WHERE time_travel_bytes > 0
    ORDER BY time_travel_bytes DESC;
    
    -- Set retention to 0 for high-churn tables you don't need to travel
    ALTER TABLE session_events
      SET DATA_RETENTION_TIME_IN_DAYS = 0;

    Frequently Asked Questions

    Q: How does Snowflake Time Travel actually work?
    A: Time Travel works by retaining expired micro-partitions rather than deleting them. When you run UPDATE or DELETE, Snowflake writes new micro-partitions and marks old ones as expired but keeps them for your retention window. When you query with AT or BEFORE, Snowflake re-points to those expired partitions at the metadata level. No data is copied or moved — it’s a metadata operation.

    Q: Is Snowflake Time Travel the same as a backup?
    A: No. Time Travel is not a backup. It’s access to older versions of data in the same system. If your Snowflake account is deleted, compromised at account level, or if cloud storage fails, Time Travel disappears with it. For true disaster recovery you need cross-region replication or separate exports.

    Q: How long does Snowflake Time Travel last?
    A: Standard edition: maximum 1 day. Enterprise and higher: up to 90 days, configurable per table, schema, or database. Transient and temporary tables max out at 1 day regardless of edition.

    Q: What happens after Time Travel expires?
    A: Expired micro-partitions move to Fail-safe — a non-configurable 7-day window managed by Snowflake. You cannot query Fail-safe data yourself. Only Snowflake support can recover it, and recovery is not guaranteed. After Fail-safe expires, data is permanently deleted.

    Q: Does Time Travel affect storage costs?
    A: Yes, significantly. Every expired micro-partition counts toward your storage bill. High-churn tables on 90-day retention can cost 10x more storage than the active data alone. Set DATA_RETENTION_TIME_IN_DAYS = 0 on staging tables or high-churn tables where Time Travel has no operational value.

    Q: What’s the difference between Time Travel and Fail-safe?
    A: Time Travel is user-controlled — you query it with SQL, configure its duration, and clone from it. Fail-safe is Snowflake-controlled — only support can access it, it’s always exactly 7 days, and it exists for Snowflake’s disaster recovery, not yours.