-
Beyond Full Refreshes: How I Redesigned a High-Volume Analytics Pipeline Around Incremental Processing
Full data refreshes are attractive because they are simple. Extract the source data, replace the analytical dataset, refresh downstream reporting, and repeat. When datasets are relatively small, this architecture can be perfectly reasonable. It is straightforward to understand, troubleshoot, and maintain. The problem emerges as historical data grows. In a mature analytics environment, most historical…
-
Snowflake Cortex Sense: The Context Runtime Agents Have Been Missing
Every enterprise AI pilot eventually hits the same wall. The agent demo worked because it ran on curated, well-described data in a controlled environment. Production is different. Production data has column names like rev_adj_v3_final, metric definitions that exist only in the head of the analyst who built the original dashboard in 2019, and a fiscal…
-
The Real Reason Data Engineers Fail Technical Interviews
You can be good at SQL and still struggle in a Data Engineer interview. You can work with Snowflake every day. You can build production pipelines, write dbt models, debug data issues and understand your company’s entire data platform. Then the interviewer gives you a problem you’ve technically seen before: “Find users who have logged…
-
Snowflake DCM Projects: Native Infrastructure as Code for Snowflake
This post is for data platform engineers deciding how to manage Snowflake objects as code. It explains how Snowflake DCM Projects work (DEFINE, PLAN, DEPLOY), when to choose them over Terraform or Schemachange, how to wire them into GitHub Actions, and the production gotchas from the official docs. Why a native option matters Until August…
-
2extract for Web Scraping: Python, Proxies & MCP Guide
This post is for data engineers who need geo-targeted scraping through residential or mobile proxies and want to see how 2extract works in practice. It covers the 2extract MCP server driven by an AI agent, then the same account driven by a Python package (twextract) that builds three datasets: geo/address, eCommerce pricing and market research.…
-
dbt State: Stop Rebuilding What Hasn’t Changed
Every hour, on a schedule, your dbt job rebuilds every model in your project. The staging models run because they always run. The intermediate joins run because they always run. The mart that powers the executive dashboard runs because, well, it always runs — even when not a single row in any upstream table changed…
-
Running LLM Tasks in Apache Airflow with the common.ai Provider
For the past two years, the standard pattern for running LLM calls in Airflow was a PythonOperator that imported the OpenAI client, called the API, and returned the result as XCom. It worked. But when the call failed at 3 AM, Airflow retried the entire task — including the database query that produced the input.…
-
Snowflake Dynamic Data Masking & Row Access Policies: A Production Guide
The governance review lands on a Wednesday. Your company needs to prove that analysts in one region cannot see customer PII from another, that customer emails are masked for anyone below the data steward tier, and that — this one is new — AI agents querying your warehouse cannot extract raw PII even when the…
-
Building Data Pipelines That Feed AI Features Without Breaking the Bill
When the consumer at the end of a pipeline is a language model rather than a dashboard, the expensive step moves from the transform to the last hop, and every row you push through it costs money. This article covers the pipeline shape we keep returning to, a four-question test for choosing batch, event-driven or…
-
Snowflake Cortex AI Token Usage Monitoring: The Complete Guide
Somewhere on your team, an AI_CLASSIFY job is running on a table larger than anyone realised. Or a Cortex Agent is looping through a multi-step workflow that seemed cheap in testing. Or a developer left a search service indexed and running in a dev environment that nobody is querying anymore. None of these will trigger…
