Skip to main content

Data Quality Expectations

For what an expectation is and how it stops a run, see Pipelines. This page covers how to write them.

Because Bauplan uses vectorized SIMD operations, expectations run at production speed on full tables or partitions, not just samples.

How this differs from other common patterns

Post-hoc validation frameworks, like dbt tests or Great Expectations, commonly execute validation as separate queries after a job builds a table or view. They detect issues only after the object exists, which means invalid data may already be visible to downstream consumers, and jobs used to build the object must be rerun.

Observability and monitoring platforms, like Monte Carlo, run on schedules independent of pipeline execution. They typically profile tables, track distributions, and alert on anomalies or drift. They are not focused on preventing invalid data from being written and do not integrate with pipeline control flow.

Why use expectations?

Early detection

Expectations catch data quality issues before they propagate through your pipeline, preventing downstream failures and incorrect results.

Data contracts

Well-documented expectations serve as implicit data contracts between teams, systems, and data providers. They codify assumptions about data structure, ranges, completeness, and relationships.

Domain knowledge capture

Expectations encode domain-specific rules and business logic directly in your data infrastructure. For example, "transaction amounts must be positive" or "user IDs must be unique."

Validation at critical points

Key use cases include:

  • External data validation: Validate data from vendors, partner teams, or APIs by writing to an isolated branch, running expectations, then publishing only if all checks pass
  • Model output validation: Ensure transformations produce expected results
  • Pre-deployment checks: Validate data before promoting pipelines to production

How expectations work

Basic structure

Expectations are Python functions decorated with @bauplan.expectation():

from typing import Annotated

import pyarrow

import bauplan
from bauplan.standard_expectations import expect_column_no_nulls

class CriticalField(bauplan.TableSchema):
"""The column this expectation validates."""

critical_field: bauplan.String

@bauplan.expectation()
@bauplan.python('3.11')
def test_critical_field_completeness(
data: Annotated[
pyarrow.Table,
bauplan.Model('my_model', projection_schema=CriticalField),
],
) -> bool:
"""Validate that critical_field has no null values."""
return expect_column_no_nulls(data, 'critical_field')

Standard expectations library

Bauplan provides a built-in library covering common validation scenarios:

  • Nullness checks: expect_column_no_nulls, expect_column_all_null, expect_column_some_null
  • Accepted values: expect_column_accepted_values
  • Uniqueness: expect_column_all_unique, expect_column_not_unique
  • Statistical properties: expect_column_mean_greater_than, expect_column_mean_greater_or_equal_than, expect_column_mean_smaller_than, expect_column_mean_smaller_or_equal_than
  • Derived columns: expect_column_equal_concatenation

You can also write custom expectations. See Custom expectations below.

Nullness checks

from bauplan.standard_expectations import (
expect_column_no_nulls,
expect_column_all_null,
expect_column_some_null
)

# Ensure no missing values in critical fields
expect_column_no_nulls(table, 'user_id')

# Verify optional field is completely empty
expect_column_all_null(table, 'deprecated_field')

# Check that field has at least some nulls (e.g., for optional data)
expect_column_some_null(table, 'middle_name')

Value constraints

from bauplan.standard_expectations import expect_column_accepted_values

# Restrict to enumerated values
expect_column_accepted_values(
table,
'status',
accepted_values=['pending', 'approved', 'rejected']
)

# Validate categorical data
expect_column_accepted_values(
table,
'payment_method',
accepted_values=['credit_card', 'debit_card', 'paypal', 'bank_transfer']
)

Statistical validations

from bauplan.standard_expectations import (
expect_column_mean_greater_than,
expect_column_mean_smaller_than
)

# Validate average order value is reasonable
expect_column_mean_greater_than(data, 'order_amount', 10.0)
expect_column_mean_smaller_than(data, 'order_amount', 10000.0)

Derived column validation

from bauplan.standard_expectations import expect_column_equal_concatenation

# Verify computed field matches expected concatenation
expect_column_equal_concatenation(
data,
target_column='full_name',
columns=['first_name', 'last_name'],
separator=' '
)

Uniqueness checks

from bauplan.standard_expectations import (
expect_column_all_unique,
expect_column_not_unique
)

# Ensure primary keys are unique
expect_column_all_unique(table, 'transaction_id')

# Expect the column to have at least one duplicate value
expect_column_not_unique(table, 'customer_id')

Custom expectations

The standard library covers common checks, but you can also create your own expectations. Say you want to check the freshness of your data:

import datetime
import pyarrow as pa
import pyarrow.compute as pc

def check_data_recency(table: pa.Table, column: str, hours: int) -> bool:
"""Check that the newest timestamp in the column is within the last N hours."""
latest = pc.max(table.column(column)).as_py()
if latest is None:
return False
cutoff = datetime.datetime.now(datetime.UTC) - datetime.timedelta(hours=hours)
return latest >= cutoff

Now to make it run as a part of the pipeline, we need to wrap it in a function with a @bauplan.expectation() decorator:

from typing import Annotated

import pyarrow

import bauplan

class Timestamps(bauplan.TableSchema):
"""The column this expectation checks for freshness."""

timestamp: bauplan.TimestampMicroUTC

@bauplan.expectation()
@bauplan.python('3.11')
def test_data_freshness(
data: Annotated[pyarrow.Table, bauplan.Model('model', projection_schema=Timestamps)],
) -> bool:
return check_data_recency(data, 'timestamp', hours=24)

Handling expectation failures

By default, Bauplan enforces expectations: if any expectation fails, the pipeline halts immediately.

When failures are not critical, you can log them and let the pipeline continue by disabling strict mode on the run command: pass --no-strict in the CLI or strict=False in the Python SDK.

Summary

Expectations provide a powerful framework for maintaining data quality throughout your pipelines. By encoding validation logic as testable functions, you can:

  • Catch issues early before they impact downstream systems
  • Document data assumptions and business rules
  • Build confidence in data quality
  • Create self-documenting pipelines

Well-implemented expectations transform data quality from a reactive problem into a proactive system property.