Partitioning Tables

For why partitioning speeds up reads, see File layout and partitioning.

Bauplan uses Iceberg's hidden partitioning, so you declare partitions on columns and Bauplan manages the partition values.

Supported partition types:

• Time-based: year(col), day(col), hour(col)
• Bucket: partition by any column, such as client ID

note

While partitioning can increase write times to object storage, it significantly improves read performance when filtering by partitioned columns.

Partitioning during import

Use the partitioned_by parameter when creating a table:

import bauplan

client = bauplan.Client()

table = client.create_table(
    table='my_partitioned_table',
    search_uri='s3://your-bucket/data/*.parquet',
    partitioned_by="hour(tpep_pickup_datetime), PULocationID",
    branch='my_branch'
)

The equivalent CLI command is:

bauplan table create my_partitioned_table \
    --partitioned-by "hour(tpep_pickup_datetime), PULocationID" \
    --search-uri 's3://your-bucket/data/*.parquet'

You can also add partitioning via an import plan:

import bauplan
import yaml

client = bauplan.Client()

plan_state = client.plan_table_creation(
    table='my_partitioned_table',
    search_uri='s3://your-data/*.parquet',
    branch='my_branch'
)

plan = yaml.safe_load(plan_state.plan)
plan['schema_info']['partitions'] = [
    {
        'from_column_name': 'datetime_column',
        'transform': {'name': 'year'}
    }
]

client.apply_table_creation_plan(plan)

Partitioning in pipelines

Use the partitioned_by parameter in the @bauplan.model decorator:

from typing import Annotated

import pyarrow

import bauplan

class PartitionedTrips(bauplan.TableSchema):
    """Trip columns kept in the partitioned table."""

    PULocationID: bauplan.Int64 | None
    trip_miles: bauplan.Float64 | None
    tips: bauplan.Float64 | None
    pickup_datetime: bauplan.TimestampMicroUTC | None

@bauplan.python('3.11')
@bauplan.model(
    partitioned_by=['day(pickup_datetime)', 'PULocationID'],
    materialization_strategy='REPLACE'
)
def create_partitioned_table(
    taxi_table: Annotated[
        pyarrow.Table,
        bauplan.Model(
            'taxi_fhvhv',
            projection_schema=PartitionedTrips,
            filter="pickup_datetime >= '2022-12-23T00:00:00-05:00'"
        )
    ],
) -> Annotated[pyarrow.Table, PartitionedTrips]:
    """Creates a partitioned table from taxi data"""
    return taxi_table

In SQL models, use comments:

-- bauplan: materialization_strategy=REPLACE
-- bauplan: partitioned_by=day(pickup_datetime),PULocationID

SELECT
    PULocationID,
    trip_miles,
    tips,
    pickup_datetime
FROM taxi_fhvhv
WHERE
    pickup_datetime >= '2022-12-25T00:00:00-05:00'