app.dataengineerhub.blog

Tag: databricks

  • Databricks Unity Catalog + Apache Iceberg in 2026

    Databricks Unity Catalog + Apache Iceberg in 2026

    The table format question is settled. Apache Iceberg won. Snowflake, Databricks, AWS, Google, and every serious data platform has committed to it. What hasn’t settled — what’s actively being fought over right now, with real architectural consequences for every data team making lakehouse decisions — is the catalog question. And the catalog matters far more than the format.

    The catalog resolves metadata, controls access, vends credentials, sequences commits, and acts as the single API boundary between every engine and every byte of data your organization owns. Pick the wrong one and you inherit operational debt that grows with each table you add. At Data + AI Summit 2026, Databricks made its position clear: Unity Catalog is the most comprehensive and open catalog across both the Delta Lake and Apache Iceberg ecosystems — with Managed Iceberg GA, Iceberg v3 GA, cross-engine ABAC, and new federation connectors including Snowflake Horizon and Salesforce Data Cloud.

    This is the guide that breaks down what Unity Catalog actually does for Iceberg workloads in 2026 — not the keynote version, but the one that tells you which features are GA, which are preview, what the access model looks like, and where the edges still are.

    TL;DR

    → Unity Catalog now governs both Delta Lake and Apache Iceberg tables from a single catalog. Managed Delta tables are GA. Managed Iceberg tables are in Public Preview (available on Databricks Runtime 16.4 LTS and above).

    → External engines access Unity Catalog managed tables through two open APIs: the Unity REST API (read/write/create for Delta clients) and the Iceberg REST Catalog (IRC) (read/write/create for Iceberg clients). Both support credential vending — temporary, scoped credentials that inherit the requesting principal’s privileges.

    → Iceberg clients that can write to Unity Catalog managed tables: Apache Spark, Apache Flink, Trino, and Snowflake. Path-based access to managed tables is not supported — it bypasses access controls and breaks managed table features.

    → Lakehouse Federation lets Unity Catalog govern tables in foreign catalogs: AWS Glue, Hive Metastore, Snowflake Horizon, Salesforce Data Cloud, Google Cloud Lakehouse, and Palantir. For Snowflake-managed Iceberg tables specifically, Catalog Federation reads directly from object storage (Databricks compute only, no Snowflake compute billed). Non-Iceberg Snowflake tables fall back to Query Federation.

    → Cross-engine ABAC is now GA: column masks and row filters enforced during server-side scan planning through the Iceberg REST Scan APIs. Any engine implementing the Iceberg 1.11 scan-planning client gets the same policies applied without a Databricks runtime.

    → A new FILE type (beta) lets managed Delta and Iceberg tables natively govern unstructured data — PDFs, images, audio, video — in open formats, tracked in Unity Catalog alongside structured tables.

    Why the catalog became the battleground

    Diagram showing how external engines like Apache Spark, Trino, and Snowflake access Unity Catalog managed tables via Unity and Iceberg REST APIs, with cloud object storage managed by Unity Catalog.

    Unity Catalog governs both table formats through one metadata layer. Policies enforce at scan-planning time — before any data file is read — so governance travels with the catalog, not the engine.

    When Delta Lake launched, the catalog was a formality. A Hive Metastore tracked table locations and schemas, and the format handled everything interesting. With Iceberg winning as the shared format, the catalog became the differentiator. Every engine can read Iceberg. The question is which engine decides who can read it, what they can see within each table, and how commits are sequenced when multiple engines write concurrently.

    That’s what Unity Catalog answers for Databricks workloads. It sits between every engine and every table, enforcing access policies at the point where scan planning happens — before any data file is read. Because the Iceberg REST Catalog API exposes those policies at the server-side scan-planning layer, a compliant engine (Spark, Trino, DuckDB via the Iceberg 1.11 client) receives the same row filters and column masks that a Databricks notebook would see, without needing to run inside Databricks. The governance travels with the catalog, not with the runtime.

    Managed tables: what Unity Catalog controls

    The key distinction in Unity Catalog is between managed and external tables. Managed tables are the default and recommended type. Unity Catalog owns everything: where the data files live, how they’re organized, compaction, statistics, optimization. You reference tables by three-part name (catalog.schema.table). Path-based access is explicitly not supported for managed tables — it bypasses Unity Catalog’s access controls and breaks features like Predictive Optimization and Liquid Clustering.

    Managed Delta tables (GA) — Unity Catalog’s default. The Delta format with ACID transactions, schema evolution, and Databricks-specific optimizations. External engines access them read-only through the Unity REST API or as Iceberg via UniForm (Delta tables exposed with an Iceberg read layer). Write access for external Delta clients is in Public Preview.

    Managed Iceberg tables (Public Preview, Databricks Runtime 16.4+) — native Apache Iceberg tables owned by Unity Catalog. External engines with Iceberg REST Catalog support can read, write, and create managed Iceberg tables. Supported write clients today: Apache Spark, Apache Flink, Trino, and Snowflake. Predictive Optimization and Liquid Clustering apply automatically.

    The practical implication: if your workload needs Snowflake to write data that Databricks then transforms, managed Iceberg is the architecture — Snowflake connects via the Iceberg REST Catalog, writes to the managed table, and Databricks reads with full governance. If the flow is Databricks-to-Snowflake reads only, UniForm on a managed Delta table is simpler than standing up a separate managed Iceberg table.

    Cross-engine access: the two APIs

    Unity REST API — for Delta Lake clients. Provides read and write access to managed and external Delta tables. Both modes support credential vending: Unity Catalog issues temporary credentials scoped to the requesting principal’s privileges, so external engines never hold long-lived Databricks credentials and governance policies apply at the storage layer.

    Iceberg REST Catalog (IRC) — for Iceberg clients. Read/write/create access to managed Iceberg tables; read-only access to Delta tables with Iceberg reads enabled (UniForm). The credential vending model is the same: temporary, scoped, inheriting the requesting principal’s privileges from Unity Catalog’s access control list.

    Both APIs hit the Unity Catalog server, not object storage directly. That’s what makes policy enforcement possible at the catalog level rather than being a layer each engine has to implement independently.

    Lakehouse Federation: governing tables you don’t own

    Unity Catalog’s federation model extends governance to tables in foreign catalogs — systems outside Databricks that Unity Catalog can query and, in some cases, govern. The federated catalog list as of mid-2026: AWS Glue, Snowflake Horizon, Hive Metastore, Salesforce Data Cloud, Google Cloud Lakehouse, and Palantir.

    The Snowflake federation case has a meaningful internal split worth understanding separately:

    Catalog Federation (for Snowflake-managed Iceberg tables) — Unity Catalog reads Snowflake Iceberg tables directly from cloud object storage. Databricks compute executes the query; Snowflake compute is never invoked, so there is no Snowflake credit charge for the read.

    Query Federation (for native Snowflake tables) — Non-Iceberg Snowflake tables are always accessed via Query Federation. Unity Catalog sends a query to Snowflake’s compute, which runs it and returns the result. Snowflake credits fire. The distinction is the same split as Salesforce Data Cloud’s File vs Query Federation — the Iceberg format is what enables compute-free storage-layer reads across both platforms.

    Cross-engine ABAC: governance that travels with the catalog

    Cross-engine ABAC is now GA: column masks and row filters defined in Unity Catalog are enforced during server-side scan planning through the Iceberg REST Scan APIs. Any engine that implements the Iceberg 1.11 scan-planning client — Spark, DuckDB, Trino, any compliant engine — gets those policies applied before it reads a single data file.

    Traditional column masking was enforced at query execution time, inside the compute layer. An engine that bypassed the query layer and read files directly could skip the masks. Server-side scan planning enforcement moves the policy check to the catalog, so an Iceberg-compliant client gets an already-filtered manifest — it can only see the files and columns it’s allowed to see, and the catalog decided that before any compute ran.

    Predictive Optimization and Liquid Clustering

    Predictive Optimization automatically identifies tables that need compaction, clustering, or statistics updates based on workload patterns and applies those operations proactively. For managed Iceberg tables, this means the same performance tuning Databricks applies to Delta workloads now runs on open-format tables accessed by external engines.

    Liquid Clustering replaces the manual partition-column decision with an adaptive co-location scheme: you specify clustering keys, and Unity Catalog reorganizes files continuously based on actual query patterns. For Iceberg tables read by Snowflake or Trino, this means better file pruning and lower scan costs even without partition-level optimization on the reader side.

    The gotchas nobody warns you about

    Managed Iceberg tables are Public Preview, not GA. Production workloads should track the GA release — preview status means the API can change.

    Path-based access breaks managed table features. If an external tool or legacy process accesses managed table files directly by path, it bypasses access controls and disables Predictive Optimization and Liquid Clustering. The migration from external tables to managed tables requires updating every access pattern to use three-part names and the catalog APIs.

    Snowflake Catalog Federation requires Iceberg-backed Snowflake tables. The compute-free federation path only works for Snowflake-managed Iceberg tables. Native Snowflake tables fall back to Query Federation with Snowflake compute charges on every federated read.

    Foreign table metadata freshness. For federated tables from Snowflake or other external catalogs, Unity Catalog caches metadata. Tables updated frequently in the external system may appear stale until a metadata refresh runs. For high-frequency foreign tables, configure periodic refresh via Lakeflow jobs.

    The Iceberg v4 roadmap changes the file structure. Databricks engineers are actively proposing Iceberg v4 changes: an adaptive metadata tree (most operations write a single file), relative path support, and a modernized statistics model for VARIANT and GEOMETRY types. Architectures built on Unity Catalog now are well-positioned for v4 because the catalog abstracts format evolution.

    The one principle

    The catalog is a write-path decision, not a read-path one. Any engine can read Iceberg. The question is which catalog controls who writes, how commits are sequenced, and which policies apply at scan time. Unity Catalog’s answer — two open APIs, credential vending, server-side ABAC, foreign catalog federation — is coherent and production-ready for Iceberg workloads today, with managed Iceberg tables a quarter behind on GA. Choose your catalog before you choose your partition strategy, because the catalog is the layer that makes your governance durable as you add engines.

    Related reading: What’s new with Unity Catalog at Data + AI Summit 2026 · Unity Catalog managed tables docs · The 2026 Migration Trap: Native Tables to Dynamic Iceberg v3 · Governing the AI Agent: Snowflake CoCo + MCP Security

  • Build a Databricks AI Agent with GPT-5

    Build a Databricks AI Agent with GPT-5

    The age of AI chatbots is evolving into the era of AI doers. Instead of just answering questions, modern AI can now execute tasks, interact with systems, and solve multi-step problems. At the forefront of this revolution on the Databricks platform is the Mosaic AI Agent Framework.

    This guide will walk you through building your first Databricks AI Agent—a powerful assistant that can understand natural language, inspect your data, and execute Spark SQL queries for you, all powered by the latest GPT-5 model.

    What is a Databricks AI Agent?

    A Databricks AI Agent is an autonomous system you create using the Mosaic AI Agent Framework. It leverages a powerful Large Language Model (LLM) as its “brain” to reason and make decisions. You equip this brain with a set of “tools” (custom Python functions) that allow it to interact with the Databricks environment.

    A diagram showing a Databricks Agent. A user prompt goes to an LLM brain (GPT-5), which then connects to tools (Python functions) and a Spark engine. Arrows indicate the flow between components.

    The agent works in a loop:

    1. Reason: Based on your goal, the LLM decides which tool is needed.
    2. Act: The agent executes the chosen Python function.
    3. Observe: It analyzes the result of that function.
    4. Repeat: It continues this process until it has achieved the final objective.

    Our Project: The “Data Analyst” Agent

    We will build an agent whose goal is to answer data questions from a non-technical user. To do this, it will need two primary tools:

    • A tool to get the schema of a table (get_table_schema).
    • A tool to execute a Spark SQL query and return the result (run_spark_sql).

    Let’s start building in a Databricks Notebook.

    Step 1: Setting Up Your Tools (Python Functions)

    An agent’s capabilities are defined by its tools. In Databricks, these are simply Python functions. Let’s define the two functions our agent needs to do its job.

    # Tool #1: A function to get the DDL schema of a table
def get_table_schema(table_name: str) -> str:
    """
    Returns the DDL schema for a given Spark table name.
    This helps the agent understand the table structure before writing a query.
    """
    try:
        ddl_result = spark.sql(f"SHOW CREATE TABLE {table_name}").first()[0]
        return ddl_result
    except Exception as e:
        return f"Error: Could not retrieve schema for table {table_name}. Reason: {e}"

# Tool #2: A function to execute a Spark SQL query and return the result as a string
def run_spark_sql(query: str) -> str:
    """
    Executes a Spark SQL query and returns the result.
    This is the agent's primary tool for interacting with data.
    """
    try:
        result_df = spark.sql(query)
        # Convert the result to a string format for the LLM to understand
        return result_df.toPandas().to_string()
    except Exception as e:
        return f"Error: Failed to execute query. Reason: {e}"

    Step 2: Assembling Your Databricks AI Agent

    With our tools defined, we can now use the Mosaic AI Agent Framework to create our agent. This involves importing the Agent class, providing our tools, and selecting an LLM from Model Serving.

    For this example, we’ll use the newly available openai/gpt-5 model endpoint.

    from databricks_agents import Agent

# Define the instructions for the agent's "brain"
# This prompt guides the agent on how to behave and use its tools
agent_instructions = """
You are a world-class data analyst. Your goal is to answer user questions by querying data in Spark.

Here is your plan:
1.  First, you MUST use the `get_table_schema` tool to understand the columns of the table the user mentions. Do not guess column names.
2.  After you have the schema, formulate a Spark SQL query to answer the user's question.
3.  Execute the query using the `run_spark_sql` tool.
4.  Finally, analyze the result from the query and provide a clear, natural language answer to the user. Do not just return the raw data table. Summarize the findings.
"""

# Create the agent instance
data_analyst_agent = Agent(
    model="endpoints:/openai-gpt-5", # Using a Databricks Model Serving endpoint for GPT-5
    tools=[get_table_schema, run_spark_sql],
    instructions=agent_instructions
)

    Step 3: Interacting with Your Agent

    Your Databricks AI Agent is now ready. You can interact with it using the .run() method, providing your question as the input.

    Let’s use the common samples.nyctaxi.trips table.

    # Let's ask our new agent a question
user_question = "What were the average trip distances for trips paid with cash vs. credit card? Use the samples.nyctaxi.trips table."

# Run the agent and get the final answer
final_answer = data_analyst_agent.run(user_question)

print(final_answer)

    What Happens Behind the Scenes:

    1. Reason: The agent reads your prompt. It knows it needs to find average trip distances from the samples.nyctaxi.trips table but first needs the schema. It decides to use the get_table_schema tool.
    2. Act: It calls get_table_schema('samples.nyctaxi.trips').
    3. Observe: It receives the table schema and sees columns like trip_distance and payment_type.
    4. Reason: Now it has the schema. It formulates a Spark SQL query like SELECT payment_type, AVG(trip_distance) FROM samples.nyctaxi.trips GROUP BY payment_type. It decides to use the run_spark_sql tool.
    5. Act: It calls run_spark_sql(...) with the generated query.
    6. Observe: It receives the query result as a string (e.g., a small table showing payment types and average distances).
    7. Reason: It has the data. Its final instruction is to summarize the findings.
    8. Final Answer: It generates and returns a human-readable response like: “Based on the data, the average trip distance for trips paid with a credit card was 2.95 miles, while cash-paid trips had an average distance of 2.78 miles.”

    Conclusion: Your Gateway to Autonomous Data Tasks

    Congratulations! You’ve just built a functional Databricks AI Agent. This simple text-to-SQL prototype is just the beginning. By creating more sophisticated tools, you can build agents that perform data quality checks, manage ETL pipelines, or even automate MLOps workflows, all through natural language commands on the Databricks platform.