Configure a Databricks Unity Catalog or a Snowflake Horizon catalog for the Atlas Stream Processing $iceberg stage. The stage can then update Apache Iceberg tables in that catalog. Complete this setup in your catalog provider first. Then, create a data catalog connection.
About This Task
The $iceberg stage supports the following catalog types:
Catalog Type | Supported Storage | Data Catalog Connection | Setup |
|---|---|---|---|
| AWS S3 and Azure Blob Storage | Not used | No catalog setup. You need only the storage connection. |
| AWSS3 | Not used | Grant your IAM role access to AWS Glue. To learn more, see Add an Atlas Stream Processing Connection. |
| AWS S3 and Azure Blob Storage | Required | Follow the Databricks Unity steps on this page. |
| AWS S3 and Azure Blob Storage | Required | Follow the Snowflake Horizon steps on this page. |
A data catalog connection stores the URL and credentials that Atlas Stream Processing uses to authenticate to your catalog. Databricks Unity uses OAuth M2M authentication with a service principal. Snowflake Horizon uses key-pair authentication.
Before You Begin
Ensure that you have the following:
A stream processing workspace with a connection to the S3 bucket or Azure Blob Storage container where you want to write your Apache Iceberg tables.
To learn more, see Add an Atlas Stream Processing Connection.
A Databricks or Snowflake account with permission to create the resources that this page describes.
Steps
Select your catalog type and cloud provider.
Next Steps
To create the data catalog connection with the credentials that you gathered, see Add an Atlas Stream Processing Connection. Then, to reference the connection in your stream processor, see the catalog field of the $iceberg stage.