# Bauplan Documentation

Bauplan is a serverless data lakehouse platform where data changes follow a Git-like workflow.
You develop and test on isolated data branches, then publish by merging into main.
The platform handles compute, storage, and orchestration automatically.

## Getting Started

- [Bauplan - The Git-for-Data Lakehouse](https://docs.bauplanlabs.com/tutorial/index.md): Bauplan brings Git-style branching to data so agents can safely change production Iceberg tables, validate, publish to main atomically, and roll back on failure.
- [Installation](https://docs.bauplanlabs.com/tutorial/installation.md): Install the Bauplan CLI and Python SDK with uv, then generate and configure your API key via the CLI or config.yml to start building data pipelines.

### Quick Start
Get started with Bauplan. Create a data branch, scaffold a project, run a Python pipeline on the titanic table, and materialize an Iceberg table.

- [Quick start](https://docs.bauplanlabs.com/tutorial/quick-start.md)
- [Data Branches](https://docs.bauplanlabs.com/tutorial/data-branches.md): Work with Bauplan data branches. Import an S3 dataset into an isolated branch, diff it against main, and merge your results back into the data catalog.
- [Import](https://docs.bauplanlabs.com/tutorial/import.md): Import Parquet and CSV files from S3 into Bauplan as Iceberg tables. Create a table, load data, review with branch diff, and resolve schema conflicts.

## Platform Overview
Bauplan is a Python-first lakehouse runtime for building data pipelines on object storage with Git-style branching, atomic publish, and commit-based rollback.

- [Platform Overview](https://docs.bauplanlabs.com/overview/index.md)
- [Execution model](https://docs.bauplanlabs.com/overview/execution-model.md): Understand how Bauplan runs pipelines. Write code locally, execute in a serverless FaaS runtime, read and write your S3, and version every result as a ref.
- [Architecture](https://docs.bauplanlabs.com/overview/architecture.md): Explore Bauplan architecture across three layers - a multi-tenant control plane for planning, a serverless data plane for execution, and user-owned storage.
- [Deployment](https://docs.bauplanlabs.com/overview/deployment.md): An Overview of Bauplan deployment options - a single-tenant Private Link managed cloud or Bring Your Own Cloud (BYOC) running the full runtime inside your own VPC.

## Core Concepts

- [Projects](https://docs.bauplanlabs.com/concepts/projects.md): Structure Bauplan projects with models, expectations, and bauplan_project.yml. Organize pipelines, configure dependencies and secrets, and add apps and orchestrators.
- [Models](https://docs.bauplanlabs.com/concepts/models.md): Build Bauplan models - declarative Python or SQL functions that transform tables. Pin per-model dependencies, materialize Iceberg outputs, and embed DuckDB SQL.
- [Pipelines](https://docs.bauplanlabs.com/concepts/pipelines.md): Build Bauplan pipelines as DAGs by chaining models through their inputs and outputs. Bauplan infers execution order automatically from your model declarations.
- [Iceberg Tables](https://docs.bauplanlabs.com/concepts/tables.md): Manage Apache Iceberg tables in Bauplan with ACID transactions, schema evolution, time travel, and declarative hidden partitioning over your object storage.
- [Namespaces](https://docs.bauplanlabs.com/concepts/namespaces.md): Organize Bauplan tables with namespaces, logical containers like database schemas. Namespaces are branch-specific, inherited on branch creation, and merged back.
- [Data Quality Expectations](https://docs.bauplanlabs.com/concepts/expectations.md): Validate data with Bauplan expectations - inline quality checks that gate Write-Audit-Publish, block invalid data before it reaches production.

### Git for Data
Bring version control to your data lake with Bauplan Git for Data - branches, transactional pipelines, commits, refs, tags, and time travel over Iceberg tables.

- [Git for Data](https://docs.bauplanlabs.com/concepts/git-for-data/index.md)
- [Transactional pipelines](https://docs.bauplanlabs.com/concepts/git-for-data/transactional-pipelines.md): Run transactional data pipelines in Bauplan where every run is atomic. Outputs commit on success via temporary execution branches, leaving your data clean on failure.
- [Commits and refs](https://docs.bauplanlabs.com/concepts/git-for-data/commits-refs.md): Learn how Bauplan versions code, data, and execution environments through commits and immutable refs for point-in-time queries, reproducible runs, and rollback.
- [Tags](https://docs.bauplanlabs.com/concepts/git-for-data/tags.md): Use Bauplan tags as human-readable names pointing to a commit. Mark production releases, validated datasets, and CI deploy points, then query or revert by tag.
- [Data branches](https://docs.bauplanlabs.com/concepts/git-for-data/data-branches.md): Use Bauplan data branches as zero-copy, version-controlled sandboxes to import, test, and merge data changes safely without ever corrupting production data.

## Agents

- [Overview](https://docs.bauplanlabs.com/agents/overview.md): Use AI agents with Bauplan three ways: the MCP server, repository-based CLAUDE.md context, and Agent Skills for guided data pipeline and ingestion workflows.
- [Skills](https://docs.bauplanlabs.com/agents/skills.md): Install Bauplan Skills to guide AI agents through data engineering workflows - build pipelines, ingest safely, run quality checks, and debug failed jobs.
- [MCP Server](https://docs.bauplanlabs.com/agents/mcp.md): Connect AI assistants to your Bauplan lakehouse with the MCP server. Query tables, manage branches, and run pipelines from Claude, Cursor, Codex, and ChatGPT.
- [Context for agents](https://docs.bauplanlabs.com/agents/context.md): Give any LLM or IDE assistant context for Bauplan without an MCP server: add a CLAUDE.md repo file and feed docs as llms.txt or Markdown for accurate code.

## Common Scenarios

- [Multi-stage pipelines](https://docs.bauplanlabs.com/common-scenarios/multi-stage-pipelines.md): Build a multi-stage Bauplan pipeline: chain models into a DAG, push down I/O with column and filter params, choose what to materialize, and dry-run before executing.
- [Create table schema conflicts](https://docs.bauplanlabs.com/common-scenarios/schema-conflicts.md): Fix schema conflicts when creating Bauplan tables from S3 files. Generate a table plan, cast conflicting column types in YAML or Python, then apply and import data.
- [Branching Workflows](https://docs.bauplanlabs.com/common-scenarios/branching-workflows.md): Learn the ways of working with data branches in Bauplan. Compare data-first and code-first workflows for promoting validated changes from a branch to production.
- [Detached Job Runs](https://docs.bauplanlabs.com/common-scenarios/detached-runs.md): Run Bauplan jobs in detached mode for scheduled or long-running pipelines. Submit, poll job status, fetch logs, and cancel jobs from the CLI or Python SDK.
- [Parameterized Runs](https://docs.bauplanlabs.com/common-scenarios/parameterized-runs.md): Pass runtime values to Bauplan pipelines with parameters and encrypted secrets. Use them as variables, filters, or in SQL, and override defaults at run time.
- [Conflict Management](https://docs.bauplanlabs.com/common-scenarios/conflict-management.md): Handle Bauplan merge conflicts safely: resolve concurrency and naming collisions with namespaces, catch schema drift with expectations, and revert corrupted tables.
- [SQL Files](https://docs.bauplanlabs.com/common-scenarios/sql-files.md): Use standalone .sql files as models in Bauplan pipelines - familiar for dbt users. Auto-discover SQL files and set model names with header comments.
- [SDK or CLI?](https://docs.bauplanlabs.com/common-scenarios/sdk-or-cli.md): Choose between the Bauplan CLI, Python SDK, and web app. The stateful CLI suits interactive work, the stateless SDK fits automation, and the web app aids monitoring.

## Integrations
Connect Bauplan to the tools you already use - orchestrators, notebooks, data apps, warehouses, BI tools, and ELT - across your Iceberg data lakehouse.

- [Integrations](https://docs.bauplanlabs.com/integrations/index.md)

### Orchestrators
Orchestrate Bauplan pipelines with Airflow, Prefect, Dagster, Temporal, DBOS, and Orchestra while Bauplan runs the compute on your Iceberg lakehouse.

- [Orchestrators](https://docs.bauplanlabs.com/integrations/orchestrators/index.md)
- [Airflow 3](https://docs.bauplanlabs.com/integrations/orchestrators/airflow.md): Run Bauplan pipelines from an Airflow 3 DAG using the TaskFlow API and Python SDK - authenticate, branch, pass parameters, and fail fast on errors.
- [Temporal](https://docs.bauplanlabs.com/integrations/orchestrators/temporal.md): Run Bauplan pipelines from a Temporal workflow - call the SDK inside Activities, create branches, pass parameters, and get durable retries and failure handling.
- [Dagster](https://docs.bauplanlabs.com/integrations/orchestrators/dagster.md): Run Bauplan pipelines from a Dagster job using the Python SDK - authenticate, create branches, pass parameters, and fail the run if the job fails.
- [DBOS](https://docs.bauplanlabs.com/integrations/orchestrators/dbos.md): Run Bauplan pipelines from a DBOS workflow with durable, exactly-once steps - authenticate, branch, pass parameters, and recover from crashes via the SDK.
- [Prefect](https://docs.bauplanlabs.com/integrations/orchestrators/prefect.md): Run Bauplan pipelines from a Prefect flow using the Python SDK - authenticate, create branches, pass parameters, and run a job from a Prefect flow.
- [Orchestra](https://docs.bauplanlabs.com/integrations/orchestrators/orchestra.md): Connect Bauplan to Orchestra and run pipelines, data quality tests, and custom SDK scripts from the GUI - no servers or local DAG code to manage.

### Interactive notebooks and data apps
Use Bauplan in Jupyter, marimo, and Streamlit via the Python SDK to query tables, run pipelines, and explore lakehouse data on isolated branches.

- [Notebooks and data apps](https://docs.bauplanlabs.com/integrations/notebooks-data-apps/index.md)
- [Jupyter Notebooks](https://docs.bauplanlabs.com/integrations/notebooks-data-apps/jupyter-notebooks.md): Query a Bauplan lakehouse from a Jupyter notebook with the Python SDK - run SQL on an isolated branch and visualize results with Pandas and Matplotlib.
- [marimo](https://docs.bauplanlabs.com/integrations/notebooks-data-apps/marimo.md): Build a reactive marimo notebook on Bauplan with the Python SDK - query a lakehouse branch, add date filters, and chart results, then serve it as a web app.
- [Streamlit](https://docs.bauplanlabs.com/integrations/notebooks-data-apps/streamlit.md): Build a Streamlit data app on Bauplan with the Python SDK - query a lakehouse branch by date range and render results as a table and Matplotlib chart.

### Warehouses and Lakehouses
Connect Bauplan-managed Iceberg tables to Snowflake, BigQuery, Databricks, Trino, and Athena via the Iceberg REST catalog or external tables, with no data copy.

- [Warehouses and Lakehouses](https://docs.bauplanlabs.com/integrations/warehouses-lakehouses/index.md)
- [Snowflake (Inbound)](https://docs.bauplanlabs.com/integrations/warehouses-lakehouses/snowflake-inbound.md): Query Bauplan-managed Iceberg tables from Snowflake. Use Bauplan as an Iceberg REST catalog and external volume to read S3 data with no copy.
- [Snowflake (Outbound)](https://docs.bauplanlabs.com/integrations/warehouses-lakehouses/snowflake-outbound.md): Read Snowflake tables inside Bauplan pipelines as Apache Arrow. Run Python analytics on Snowflake data or export it wholesale to Iceberg.
- [BigQuery (Inbound)](https://docs.bauplanlabs.com/integrations/warehouses-lakehouses/big-query-inbound.md): Query Bauplan-managed Iceberg tables from BigQuery. Register them as external tables over S3 via a cross-cloud connection, with no data copy.
- [BigQuery (Outbound)](https://docs.bauplanlabs.com/integrations/warehouses-lakehouses/big-query-outbound.md): Read BigQuery tables inside Bauplan pipelines as Apache Arrow. Run Python analytics on BigQuery data or export it to Iceberg.
- [GCS (Google Cloud Storage)](https://docs.bauplanlabs.com/integrations/warehouses-lakehouses/gcs.md): Sync a Google Cloud Storage bucket to Bauplan with AWS DataSync and run Iceberg pipelines on GCS-landed data without leaving your landing zone.
- [R2 - Cloudflare object storage](https://docs.bauplanlabs.com/integrations/warehouses-lakehouses/r2-cloudflare-object-storage.md): Connect a Cloudflare R2 bucket to Bauplan and query Parquet files as external Iceberg tables, with zero-copy imports and no egress fees.

### BI tools and Postgres client
Connect BI tools to Bauplan through a PostgreSQL-compatible proxy for read-only SQL access to Iceberg tables stored as Parquet on S3.

- [BI tools and Postgres client](https://docs.bauplanlabs.com/integrations/bi-tools-postgres/index.md)
- [Metabase](https://docs.bauplanlabs.com/integrations/bi-tools-postgres/metabase.md): Connect Metabase to Bauplan via the PostgreSQL proxy. Configure the connection, browse Iceberg tables, and build dashboards over your lakehouse data.

### Data Integration and ELT Tools
Land data in S3 with your ELT tool, then turn it into safe, queryable Iceberg tables in Bauplan. Register external tables or materialize from Parquet files.

- [Data Integration and ELT](https://docs.bauplanlabs.com/integrations/data-int-and-etl/index.md)
- [Fivetran](https://docs.bauplanlabs.com/integrations/data-int-and-etl/fivetran.md): Register Fivetran Managed Data Lake Iceberg tables in Bauplan as external tables. Branch, validate, and query Fivetran data on S3 without copying it.
- [Estuary](https://docs.bauplanlabs.com/integrations/data-int-and-etl/estuary.md): Stream and CDC data from any Estuary source into Bauplan Iceberg tables. Estuary runs Spark on EMR Serverless to merge updates in near real-time.
- [Frequently Asked Questions](https://docs.bauplanlabs.com/faq.md): Bauplan FAQ: how it integrates with Iceberg and S3, handles schema evolution and versioned data, supported file formats, why it avoids Spark, and orchestrator use.

## API Reference

- [CLI Reference](https://docs.bauplanlabs.com/reference/cli.md): Complete reference for the Bauplan command-line interface: run pipelines, query and import tables, manage branches, parameters, jobs, and global CLI options.
- [bauplan](https://docs.bauplanlabs.com/reference/bauplan.md): Reference for the Bauplan Python SDK Client: query tables, run pipelines, import data, and manage branches, commits, tags, and jobs over Apache Iceberg lakehouses.
- [bauplan.exceptions](https://docs.bauplanlabs.com/reference/bauplan-exceptions.md): Reference for bauplan.exceptions: the Python SDK error hierarchy covering HTTP, merge conflict, not-found, forbidden, and table-plan errors raised by Bauplan.
- [bauplan.schema](https://docs.bauplanlabs.com/reference/bauplan-schema.md): Reference for bauplan.schema types: Branch, Ref, Commit, Tag, Namespace, Table, Job, and DAG classes returned by the Bauplan Python SDK over Iceberg catalogs.
- [bauplan.standard_expectations](https://docs.bauplanlabs.com/reference/bauplan-standard-expectations.md): Reference for bauplan.standard_expectations: built-in data quality checks for null, uniqueness, accepted values, mean, and concatenation in Bauplan pipelines.
- [bauplan.state](https://docs.bauplanlabs.com/reference/bauplan-state.md): Reference for bauplan.state types: RunState, RunExecutionContext, and table-create and import state objects returned by Bauplan Python SDK run and import jobs.

## Other
- [Sample Datasets](https://docs.bauplanlabs.com/datasets.md): Sample datasets in the Bauplan lakehouse for tutorials: NYC Citi Bike, NYC taxi trips and zones, Titanic, and German wind energy, with schemas and row counts.
- [dbt Core](https://docs.bauplanlabs.com/integrations/dbt/index.md): The Bauplan dbt Core integration is in active development. Contact us for early access and a preview of running dbt models on your Iceberg lakehouse.
- [S3 Permissions Example](https://docs.bauplanlabs.com/tutorial/s3-permissions.md): Example S3 bucket policy granting public GetObject and ListBucket access so Bauplan can read Parquet and CSV files when importing data in the sandbox.