> The complete documentation index is at [llms.txt](/llms.txt).

Stream data from any Estuary source into Bauplan Iceberg tables using the [Bauplan connector](https://docs.estuary.dev/reference/Connectors/materialization-connectors/apache-iceberg/bauplan/). Estuary orchestrates Spark jobs on AWS EMR Serverless to merge updates into Bauplan-managed tables in near real-time, including full CDC (Change Data Capture) support.

## When to use this integration

- You want to stream operational data directly into your Bauplan lakehouse with low latency.
- You need row-level changes from the source (inserts, updates, deletes) merged into your Iceberg tables, rather than appended as a raw event log.
- You want to decouple your ingestion pipeline from your transformation logic in Bauplan.

## Prerequisites

- A Bauplan account with the **OAuth client credentials** feature enabled (currently in beta).
- An AWS account where you can create EMR Serverless applications, IAM users and roles, and SSM parameters.
- An Estuary account.

## Step 1. Configure the Estuary source capture

In the [Estuary capture dashboard](https://dashboard.estuary.dev/captures), create a new capture of your data source. Estuary provides connectors out of the box for the main relational databases, SaaS applications (for example, Stripe, and Google Sheets), cloud storage, and generic HTTP APIs.

Refer to the [Estuary capture documentation](https://docs.estuary.dev/reference/Connectors/capture-connectors/) for the specific configuration required for your data source. 

## Step 2. Create a Bauplan OAuth client

The [Bauplan connector](https://docs.estuary.dev/reference/Connectors/materialization-connectors/apache-iceberg/bauplan/) Estuary uses for materialization authenticates against the Bauplan Iceberg REST catalog using OAuth 2.0 client credentials. This feature is currently in **beta:** contact us to enable it for your account.

Once enabled, navigate to your Bauplan account [OAuth apps page](https://app.bauplanlabs.com/oauth-apps) and create a new OAuth application. Specify a name, description, and token lifetime. The system will generate a **Client ID** and a **Client Secret:** keep these; you will need them when configuring the Estuary destination.

> While technically not mandatory, we recommend having one OAuth application _per_ data source.

## Step 3. Configure AWS for EMR Serverless

Estuary uses AWS EMR Serverless to run the Spark jobs that merge data into your Iceberg tables. You need four AWS resources.

### EMR Serverless application

Create an EMR Serverless application with the Spark runtime, release version **emr-7.12.0**, architecture **x86_64**. Enable **Automatically start application on job submission**. Configure VPC, subnets, and security groups so that EMR jobs can reach the Bauplan catalog endpoint (`https://api.use1.aprod.bauplanlabs.com/iceberg`) over the internet. Note the **Application ID:** you will need it later on.

### Staging S3 bucket

Create (or designate) an S3 bucket in the same region as the EMR application. Estuary stages Parquet and CSV data files, as well as the Python scripts executed by Spark jobs, in this bucket before merging them into Iceberg tables.

We recommend using a dedicated prefix (for example, `estuary-staging/`) within your Bauplan data lake bucket.

### IAM execution role

EMR Serverless assumes this role at job runtime, referenced below as ``.

_Permissions policy:_

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "S3ListBucket",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::<BUCKET>"
    },
    {
      "Sid": "S3ReadAll",
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::<BUCKET>/*"
    },
    {
      "Sid": "S3WriteIceberg",
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::<BUCKET>/iceberg/*"
    },
    {
      "Sid": "S3WriteStaging",
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::<BUCKET>/estuary-staging/*"
    },
    {
      "Sid": "SSMReadCatalogCredentials",
      "Effect": "Allow",
      "Action": ["ssm:GetParameter", "ssm:GetParameters"],
      "Resource": "arn:aws:ssm:<REGION>:<ACCOUNT_ID>:parameter<SSM_PREFIX>*"
    }
  ]
}
```

_Trust policy:_

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "emr-serverless.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
```

### IAM user

Estuary uses this user to submit jobs and manage SSM secrets, referenced below as ``.

Attach the following policy:

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "EMRServerlessManageJobs",
      "Effect": "Allow",
      "Action": [
        "emr-serverless:ListApplications",
        "emr-serverless:StartJobRun",
        "emr-serverless:GetJobRun",
        "emr-serverless:ListJobRuns",
        "emr-serverless:CancelJobRun",
        "emr-serverless:GetApplication",
        "emr-serverless:StartApplication"
      ],
      "Resource": [
        "arn:aws:emr-serverless:<REGION>:<ACCOUNT_ID>:/applications/<EMR_APPLICATION_ID>",
        "arn:aws:emr-serverless:<REGION>:<ACCOUNT_ID>:/applications/<EMR_APPLICATION_ID>/jobruns/*"
      ]
    },
    {
      "Sid": "IAMPassExecutionRole",
      "Effect": "Allow",
      "Action": "iam:PassRole",
      "Resource": "arn:aws:iam::<ACCOUNT_ID>:role/<EXECUTION_ROLE>",
      "Condition": {
        "StringEquals": {
          "iam:PassedToService": "emr-serverless.amazonaws.com"
        }
      }
    },
    {
      "Sid": "S3ListBucket",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::<BUCKET>"
    },
    {
      "Sid": "S3ReadWriteStaging",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::<BUCKET>/estuary-staging/*"
    },
    {
      "Sid": "S3ReadWriteIceberg",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::<BUCKET>/iceberg/*"
    },
    {
      "Sid": "SSMManageCatalogCredentials",
      "Effect": "Allow",
      "Action": [
        "ssm:PutParameter",
        "ssm:GetParameter",
        "ssm:GetParameters",
        "ssm:DeleteParameter"
      ],
      "Resource": "arn:aws:ssm:<REGION>:<ACCOUNT_ID>:parameter<SSM_PREFIX>*"
    }
  ]
}
```

Generate an **Access Key ID** (``) and a **Secret Access Key** (``) for this user.

## Step 4. Configure Bauplan

We recommend having Estuary write data not on the `main` branch, but on a separate _ingestion_ branch. In this way, only the data passing quality checks (in the form, for instance, of [expectations](https://docs.bauplanlabs.com/concepts/expectations)) will be eventually merged to `main`. We suggest having one such branch _per_ data source.

Create a source-specific branch from `main` with
```sh
bauplan branch create <USERNAME>.ingestion-<SOURCE_NAME> --from-ref main
```

Additionally, consider creating a dedicated namespace for the data source so that, when merging to `main`, different data sources will live on separate namespaces

```sh
bauplan namespace create <SOURCE_NAME> --branch <USERNAME>.ingestion-<SOURCE_NAME>
```

## Step 5. Configure the Estuary materialization

In the [Estuary materialization dashboard](https://dashboard.estuary.dev/materializations), create a new **Bauplan** materialization and configure it as follows.

### Materialization details

| Field                      | Value                                                         |
| -------------------------- | ------------------------------------------------------------- |
| **Name**                   | Use any name, for instance `/bauplan/` |
| **Data Plane**             | The AWS region that is closest to where Bauplan is deployed                         |  

### Endpoint configuration

| Field                      | Value                                                                                                                                             |
| -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------    |
| **Endpoint**               | `https://api.use1.aprod.bauplanlabs.com/iceberg/` - or to write directly on `main`: `https://api.use1.aprod.bauplanlabs.com/iceberg` |
| **Warehouse**              | `default`                                                                                                                                         |
| **Namespace**              | ``                                                                                                                                |
| **Base Location**          | `s3:///iceberg`                                                                                                                           |

> Setting the namespace here is _not_ sufficient to ensure that tables are written under such namespace in Bauplan. You will need to also specify it for every table (see below).

### Catalog authentication

| Field                      | Value                                                                                                                                          |
| -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| **Catalog Authentication** | Pick "OAuth 2.0 Client Credential". Other authentication methods will _not_ work                                                               |
| **OAuth 2.0 Server URI**   | `v1/oauth/tokens`                                                                                                                              |
| **Catalog Credential**     | `:`                                                                                                   |
| **Scope**                  | _(leave empty)_                                                                                                                                |

### EMR configuration

| Field                      | Value                                                                  |
| -------------------------- | ---------------------------------------------------------------------- |
| **Region**                 | AWS region of your EMR application                                     |
| **Application ID**         | ``                                                 |
| **Execution Role ARN**     | `arn:aws:iam:::role/`                      |
| **Bucket**                 | ``                                                             |
| **Bucket Path**            | `estuary-staging/`                                                     |
| **Access Key ID**          | ``                                                  |
| **Secret Access Key**      | ``                                              |
| **Systems Manager Prefix** | ``                                                         |

### Collections

Add your collections as **bindings**, mapping each Estuary collection to its target Iceberg table name. Here, _for each table_, set the namespace to match your target Bauplan namespace if you want to avoid the default. Save and publish the connector.

If you previously wrote data under a different namespace and need to move collections, you will need to delete the old tables, update the namespace in the binding configuration, and trigger a **backfill** for the affected collections.