-
Why Larger LLMs Give Incorrect Answers in Production
A team I worked with upgraded their support-ticket triage pipeline to a bigger, newer model expecting fewer mistakes. Benchmarks said it was smarter across the board. Two weeks in, the error rate on a specific ticket category had gotten worse, not better — and worse in a particular way: the new model was wrong just…
-
The Medallion Architecture, Reconsidered: What It Solved and Where It Cracks
Almost every data team says it’s “doing medallion architecture.” Look under the hood and most of them aren’t — they have a Bronze layer that’s a dumping ground, a Silver layer that’s Bronze with nicer column names, and a Gold layer business users technically have access to but can’t actually use. That gap between the…
-
7 Steps to Building and Deploying Your First Autonomous Agent
The Slack message came in at 9:14 on a Tuesday: “why is yesterday’s Snowflake bill $4,200 over budget and who approved it.” Nobody had approved anything. A dashboard refresh job had been silently retrying every five minutes since Saturday after a schema change broke one of its filters, and by the time anyone noticed, it…
-
The Future of Data Engineering in an AI-Driven World
Two facts from 2026 sit uncomfortably next to each other. Databricks has said that most new databases created on its platform are now spun up by AI agents rather than humans. And in the same year, one of the more sober industry write-ups pointed out that actual adoption of agentic data engineering — agents autonomously…
-
How Snowflake Stores Data Internally: Micro-Partitions, FDN & Pruning
A team I worked with ran a nightly job that did something completely reasonable: it updated a single status flag on rows in a 2 TB orders table. One column. A few million rows a night. Harmless. Three months later their storage bill had nearly quadrupled, and no one had loaded any new data. The culprit wasn’t…
-
Why Senior Data Engineers Write SQL Differently (With Examples)
I once inherited a revenue report that had been green for fourteen months. Every morning it ran, populated a dashboard, and nobody questioned it. Then a new CFO asked why the data warehouse said Q2 regional revenue was $4.1M while the finance system said $1.3M. Same source data. The query had no error, no failed…
-
Your Data Pipeline Agent Is a Confused Deputy Waiting to Happen
A support-triage agent I built last quarter had read access to our CRM, could issue refunds under $50 without approval, and could send email on a customer’s behalf to confirm resolutions. All three permissions were individually reasonable. Together, they were a loaded gun. A test ticket, deliberately crafted by our own security review, contained a…
-
What Happens When You Give Your Local Agent a Real Memory
Two weeks ago, an agent I run locally for pipeline maintenance rewrote a retry handler using a flat, fixed-delay retry. It looked reasonable. It was also the exact pattern that caused a duplicate-row incident in May, one I’d personally debugged for four hours and was very sure I’d never see again. I hadn’t told the…
-
How AI Is Reshaping Data Engineering: A Practical Guide (2026)
A team I advised shipped their first AI-generated pipeline on a Thursday. An agent had scaffolded a dbt model from a plain-English request — “roll daily revenue up to weekly by region” — and the SQL was clean, the run was green, the dashboard populated. It looked like the future. Three weeks later finance flagged…
-
5 Real-World SQL Portfolio Projects That Actually Get You Hired
A hiring manager I know keeps a folder of rejected portfolio links. Not to be cruel — to remember the pattern. Out of roughly forty SQL “projects” she reviewed for one analytics-engineer opening, thirty-one were the same customer-churn notebook off the same Kaggle CSV, with the same three queries, ending on the same bar chart.…
