Estuary

Stream data from any Estuary source into Bauplan Iceberg tables using the Bauplan connector. Estuary orchestrates Spark jobs on AWS EMR Serverless to merge updates into Bauplan-managed tables in near real-time, including full CDC (Change Data Capture) support.

When to use this integration

You want to stream operational data directly into your Bauplan lakehouse with low latency.
You need row-level changes from the source (inserts, updates, deletes) merged into your Iceberg tables, rather than appended as a raw event log.
You want to decouple your ingestion pipeline from your transformation logic in Bauplan.

Prerequisites

A Bauplan account with the OAuth client credentials feature enabled (currently in beta).
An AWS account where you can create EMR Serverless applications, IAM users and roles, and SSM parameters.
An Estuary account.

Step 1. Configure the Estuary source capture

In the Estuary capture dashboard, create a new capture of your data source. Estuary provides connectors out of the box for the main relational databases, SaaS applications (for example, Stripe, and Google Sheets), cloud storage, and generic HTTP APIs.

Refer to the Estuary capture documentation for the specific configuration required for your data source.

Step 2. Create a Bauplan OAuth client

The Bauplan connector Estuary uses for materialization authenticates against the Bauplan Iceberg REST catalog using OAuth 2.0 client credentials. This feature is currently in beta: contact us to enable it for your account.

Once enabled, navigate to your Bauplan account OAuth apps page and create a new OAuth application. Specify a name, description, and token lifetime. The system will generate a Client ID and a Client Secret: keep these; you will need them when configuring the Estuary destination.

tip

While technically not mandatory, we recommend having one OAuth application per data source.

Step 3. Configure AWS for EMR Serverless

Estuary uses AWS EMR Serverless to run the Spark jobs that merge data into your Iceberg tables. You need four AWS resources.

EMR Serverless application

Create an EMR Serverless application with the Spark runtime, release version emr-7.12.0, architecture x86_64. Enable Automatically start application on job submission. Configure VPC, subnets, and security groups so that EMR jobs can reach the Bauplan catalog endpoint (https://api.use1.aprod.bauplanlabs.com/iceberg) over the internet. Note the Application ID: you will need it later on.

Staging S3 bucket

Create (or designate) an S3 bucket in the same region as the EMR application. Estuary stages Parquet and CSV data files, as well as the Python scripts executed by Spark jobs, in this bucket before merging them into Iceberg tables.

We recommend using a dedicated prefix (for example, estuary-staging/) within your Bauplan data lake bucket.

IAM execution role

EMR Serverless assumes this role at job runtime, referenced below as <EXECUTION-ROLE>.

Permissions policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "S3ListBucket",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::<BUCKET>"
    },
    {
      "Sid": "S3ReadAll",
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::<BUCKET>/*"
    },
    {
      "Sid": "S3WriteIceberg",
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::<BUCKET>/iceberg/*"
    },
    {
      "Sid": "S3WriteStaging",
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::<BUCKET>/estuary-staging/*"
    },
    {
      "Sid": "SSMReadCatalogCredentials",
      "Effect": "Allow",
      "Action": ["ssm:GetParameter", "ssm:GetParameters"],
      "Resource": "arn:aws:ssm:<REGION>:<ACCOUNT_ID>:parameter<SSM_PREFIX>*"
    }
  ]
}

Trust policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "emr-serverless.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

IAM user

Estuary uses this user to submit jobs and manage SSM secrets, referenced below as <iam-user>.

Attach the following policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "EMRServerlessManageJobs",
      "Effect": "Allow",
      "Action": [
        "emr-serverless:ListApplications",
        "emr-serverless:StartJobRun",
        "emr-serverless:GetJobRun",
        "emr-serverless:ListJobRuns",
        "emr-serverless:CancelJobRun",
        "emr-serverless:GetApplication",
        "emr-serverless:StartApplication"
      ],
      "Resource": [
        "arn:aws:emr-serverless:<REGION>:<ACCOUNT_ID>:/applications/<EMR_APPLICATION_ID>",
        "arn:aws:emr-serverless:<REGION>:<ACCOUNT_ID>:/applications/<EMR_APPLICATION_ID>/jobruns/*"
      ]
    },
    {
      "Sid": "IAMPassExecutionRole",
      "Effect": "Allow",
      "Action": "iam:PassRole",
      "Resource": "arn:aws:iam::<ACCOUNT_ID>:role/<EXECUTION_ROLE>",
      "Condition": {
        "StringEquals": {
          "iam:PassedToService": "emr-serverless.amazonaws.com"
        }
      }
    },
    {
      "Sid": "S3ListBucket",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::<BUCKET>"
    },
    {
      "Sid": "S3ReadWriteStaging",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::<BUCKET>/estuary-staging/*"
    },
    {
      "Sid": "S3ReadWriteIceberg",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::<BUCKET>/iceberg/*"
    },
    {
      "Sid": "SSMManageCatalogCredentials",
      "Effect": "Allow",
      "Action": [
        "ssm:PutParameter",
        "ssm:GetParameter",
        "ssm:GetParameters",
        "ssm:DeleteParameter"
      ],
      "Resource": "arn:aws:ssm:<REGION>:<ACCOUNT_ID>:parameter<SSM_PREFIX>*"
    }
  ]
}

Generate an Access Key ID (<AWS_ACCESS_KEY_ID>) and a Secret Access Key (<AWS_SECRET_ACCESS_KEY>) for this user.

Step 4. Configure Bauplan

We recommend having Estuary write data not on the main branch, but on a separate ingestion branch. In this way, only the data passing quality checks (in the form, for instance, of expectations) will be eventually merged to main. We suggest having one such branch per data source.

Create a source-specific branch from main with

bauplan branch create <USERNAME>.ingestion-<SOURCE_NAME> --from-ref main

Additionally, consider creating a dedicated namespace for the data source so that, when merging to main, different data sources will live on separate namespaces

bauplan namespace create <SOURCE_NAME> --branch <USERNAME>.ingestion-<SOURCE_NAME>

Step 5. Configure the Estuary materialization

In the Estuary materialization dashboard, create a new Bauplan materialization and configure it as follows.

Materialization details

Field	Value
Name	Use any name, for instance `<YOUR_ORG>/bauplan/<SOURCE_NAME>`
Data Plane	The AWS region that is closest to where Bauplan is deployed

Endpoint configuration

Field	Value
Endpoint	`https://api.use1.aprod.bauplanlabs.com/iceberg/<BRANCH_NAME>` - or to write directly on `main`: `https://api.use1.aprod.bauplanlabs.com/iceberg`
Warehouse	`default`
Namespace	`<SOURCE_NAME>`
Base Location	`s3://<BUCKET>/iceberg`

caution

Setting the namespace here is not sufficient to ensure that tables are written under such namespace in Bauplan. You will need to also specify it for every table (see below).

Catalog authentication

Field	Value
Catalog Authentication	Pick "OAuth 2.0 Client Credential". Other authentication methods will not work
OAuth 2.0 Server URI	`v1/oauth/tokens`
Catalog Credential	`<CLIENT_ID>:<CLIENT_SECRET>`
Scope	(leave empty)

EMR configuration

Field	Value
Region	AWS region of your EMR application
Application ID	`<EMR_APPLICATION_ID>`
Execution Role ARN	`arn:aws:iam::<ACCOUNT_ID>:role/<EXECUTION_ROLE>`
Bucket	`<BUCKET>`
Bucket Path	`estuary-staging/`
Access Key ID	`<AWS_ACCESS_KEY_ID>`
Secret Access Key	`<AWS_SECRET_ACCESS_KEY>`
Systems Manager Prefix	`<SSM_PREFIX>`

Collections

Add your collections as bindings, mapping each Estuary collection to its target Iceberg table name. Here, for each table, set the namespace to match your target Bauplan namespace if you want to avoid the default. Save and publish the connector.

If you previously wrote data under a different namespace and need to move collections, you will need to delete the old tables, update the namespace in the binding configuration, and trigger a backfill for the affected collections.