DeltaLake Foreign Table

Databricks Unity Catalog provides unified data governance and collaboration across the Databricks platform, with centralized metadata management, role-based access control, and fine-grained permissions. Supports unified data storage across clouds and workspaces, audit logging, column- and row-level security, and enables data sharing for analytics and machine learning workflows. Delivers simplified data governance and cross-team collaboration.

The Delta format (Delta Lake) is an open-source table format developed by Databricks, built atop Apache Parquet, designed to handle large-scale data engineering and analytics workloads. Addresses traditional data lake limitations, such as data consistency, transaction support, and performance optimization. Provides ACID transaction support, data versioning (time travel), supports both batch and streaming processing.

Tacnode supports accelerated access to Delta-format data lakes through external tables integrated with Unity Catalog metadata.

Install the Extension

Install the extension (extension installation is a one-time operation; skip this step if already installed):

CREATE EXTENSION delta_fdw;

Create Unity Catalog Foreign Server

CREATE SERVER <server_name> FOREIGN DATA WRAPPER delta_fdw
OPTIONS (
 unity_catalog_endpoint '<endpoint_url>',
 region 'region-name',
 catalog '<catalog_name>'
);

• unity_catalog_endpoint: API endpoint for Unity Catalog
• region: Cloud region where Unity Catalog is deployed
• catalog: Root directory for catalog files

A separate foreign server must be created for each catalog.

-- Example
CREATE SERVER delta_oregon FOREIGN DATA WRAPPER delta_fdw OPTIONS (
        unity_catalog_endpoint 'https://dbc-xxx.cloud.databricks.com',
        region 'us-west-2',
        catalog 'main'
);

Create a User Mapping

Grant access to underlying object storage for specified user accounts.

CREATE USER MAPPING FOR public SERVER delta_server
OPTIONS (
        token 'token',
        access_id 'access-id',
        access_key 'access-key'
);

Import Foreign Tables

Import Delta table metadata from Databricks.

IMPORT FOREIGN SCHEMA remote_schema_name
    { LIMIT TO (table_name [, ...])
    | EXCEPT (table_name [, ...]) }
    FROM SERVER server_name
    INTO local_schema_name
    [ OPTIONS (option 'value' [, ...]) ];

remote_schema_name: schema in Unity Catalog table_name: table in Unity Catalog server_name: foreign server defined above (each catalog uses a different foreign server)

-- Import 'test1' metadata from 'default' schema as a foreign table
import foreign schema "default" LIMIT TO (test1) from server delta_server into public;
 
select * FROM test1;