Delta Sharing

Governed, read-only table sharing into your warehouse or lakehouse via the open Delta Sharing protocol.

Delta Sharing gives Amigo a way to grant governed, read-only access to selected tables so you can consume the data directly inside your analytics platform or warehouse using an open sharing protocol.

New to Delta Sharing?

Delta Sharing is an open protocol for secure data exchange across platforms and clouds. Providers expose read-only tables via a simple REST endpoint. Recipients use a small profile file (credentials plus endpoint) to query those tables directly from tools like pandas or Apache Spark, without copying data.

Learn more:

• Overview
• Protocol spec
• Quickstart
• Python connector
• Spark connector
• Delta Sharing documentation

Key Concepts

• Share: the container your organization is granted access to. A share exposes one or more schemas.
• Schema: a logical grouping of tables within a share.
• Table: a read-only dataset you can query from supported tools.

• Profile file: a JSON file with the endpoint and credentials that clients use.
• Open protocol: REST-based access to the underlying Parquet and Delta data.

Access and Provisioning

Ask your Amigo representative (via Slack) to provision a Delta Share for your organization and specify which tables you need. Delta Sharing is in beta, so schemas and operational policies may still evolve.

You will receive one of the following:

• A share profile file (containing the share endpoint and credentials), or
• An endpoint URL plus recipient credentials, along with the list of shared schemas and tables.

How You Use It

Most platforms support Delta Sharing natively or via an open-source connector. Import the provided share profile, or configure the endpoint and credentials per your platform's instructions. From there, you can browse the shared schemas and tables and query them like native read-only tables in your environment.

Example workflows

• Connect your warehouse or lakehouse to read shared tables for BI dashboards.
• Consume curated datasets for ML feature engineering or offline evaluation.
• Join Amigo-shared tables with your internal datasets without copying data.

Supported Clients

• Python: the delta-sharing library for pandas and PySpark.
• Apache Spark: the Delta Sharing Spark connector (SQL, Python, Scala, Java, R).
• BI and ETL: many tools integrate via the open protocol or vendor connectors.

Quick Start

Trying it quickly?

Use the public demo profile file to explore example datasets: Open datasets profile

Python (pandas)

pip install delta-sharing

import delta_sharing

profile_file = "/path/to/profile.share"  # e.g., ./open-datasets.share
table_url = f"{profile_file}#my_share.my_schema.my_table"

pdf = delta_sharing.load_as_pandas(table_url)
print(pdf.head())

PySpark

Option A: the Python connector in PySpark (requires the Spark connector installed on the cluster).

import delta_sharing

profile_file = "/path/to/profile.share"
table_url = f"{profile_file}#my_share.my_schema.my_table"

df = delta_sharing.load_as_spark(table_url)
df.createOrReplaceTempView("shared_table")
spark.sql("SELECT COUNT(*) FROM shared_table").show()

Option B: the Spark reader directly.

pyspark --packages io.delta:delta-sharing-spark_2.12:3.1.0

profile_file = "/path/to/profile.share"
table_url = f"{profile_file}#my_share.my_schema.my_table"
df = spark.read.format("deltaSharing").load(table_url)
df.show()

Spark (Scala)

spark-shell --packages io.delta:delta-sharing-spark_2.12:3.1.0

val profileFile = "/path/to/profile.share"
val tableUrl = s"$profileFile#my_share.my_schema.my_table"
val df = spark.read.format("deltaSharing").load(tableUrl)
df.selectExpr("count(*) as rows").show()

Table URL format: <profile_file>#<share>.<schema>.<table>. The profile file can live locally or in cloud storage (e.g., s3a://...). See the connector docs for supported path schemes.

Governance and Operations

• Scope: shares are read-only and limited to the schemas and tables you request.
• Security: credentials and recipient tokens can be rotated. IP allowlisting is available on request.
• Observability: access and query activity are logged by Amigo for security and auditing.