-
How to setup DBT state in Your Snowflake project (step-by-step Config Guide)
You read about dbt State. You understood the pitch — skip unchanged models, cut compute, stop paying to rebuild what didn’t change. Then you opened your project and stared at an empty dbt_project.yml wondering where to actually start. This is that guide. Not the concept. The implementation. Real configs, real gotchas, real Snowflake-specific decisions you’ll face inside…
-
Stop Spinning Up Spark clusters for 50GB Datasets
Your team has a 200GB Parquet file on S3. Someone suggests running the analysis in Spark. You spin up a four-node cluster, configure executors, tune shuffle partitions, wait three minutes for the cluster to initialize, wait fourteen minutes for the job to run, tear the cluster down, and get back a number. The same query…
-
Someone renamed a column.Your Pipeline Died.Here’s the fix
Someone on the backend team renamed order_total to order_amount. Clean name. Makes total sense for their domain model. They shipped it on a Thursday afternoon. By Friday morning, your revenue dashboard was showing zero. Not wrong numbers. Zero. Because your Snowflake pipeline was still selecting order_total from the events table, and the column simply wasn’t there anymore. You found out…
-
Why AI Agents Forget: The Architecture behind Memory failures
Your AI agent isn’t getting dumber over time. It’s getting amnesiac. It forgets a constraint you set ten turns ago, even though it followed it perfectly at turn three. It contradicts itself across sessions. It treats a fact you corrected last week as if it never heard the correction. Teams blame the model. They swap GPT…
-
dbt state: Skip Unchanged Nodes, Cut Warehouse Compute 30%
You have a 400-model dbt project. A junior analyst tweaks one source definition. Every. Single. Model. Rebuilds. Ninety minutes later, you’ve burned $1,200 in warehouse compute for a change that affected nothing downstream. That’s the dbt default. It’s also the most expensive habit in the modern data stack. dbt’s State feature is the principled answer: compare your…
-
dbt Fusion: 30x Faster parsing(And why Migration matters)
You’ve probably heard the buzz: dbt’s new Fusion engine is 30x faster. But what nobody says clearly is faster at what, for whom, and does it break your project? The answer is messier than the marketing. The Fusion engine is a complete rewrite of dbt Core in Rust, released last year and hitting 4,500+ projects already. It’s…
-
Snowflake Iceberg V3: When to Actually Migrate(vs Native Tables)
Most data engineers I talk to still store everything in Snowflake native format. It’s simple: load data, query data, done. But here’s what nobody’s talking about: if you’re querying that data from anywhere else — Spark, Databricks, even just a local Python script — you’re paying a hidden “data tax.” Redundant storage, egress fees, ETL pipeline complexity.…
-
Snowflake Query Execution: what really happens under the hood
Ask ten data engineers what happens when you run a query in Snowflake and most of them will tell you the same thing: the warehouse runs it. SQL goes in, the warehouse chews on it, results come out. Clean, simple, and just wrong enough to cost you money. The truth is that Snowflake query execution is a…
-
How the Warehouse Cache Actually Works in Snowflake
A dashboard that ran in four seconds on Monday took nineteen seconds on Tuesday. Same query, same data, same warehouse size. I spent the better part of an hour convinced Snowflake was having a bad day, checking the status page, refreshing query history, muttering about “platform issues” in our team Slack — before I noticed…
-
Everyone Said SQL Was Dead. It’s Now the Most Valuable Skill in AI (2026)
In 2018, a wave of Medium posts declared SQL obsolete. NoSQL was the future. Python would handle everything. Data lakes would make relational thinking irrelevant. The hot take had a good run. Then AI happened — and SQL came back harder than ever. Today, SQL is the connective tissue of every serious AI data stack.…
