Data Quality Expectations
For what an expectation is and how it stops a run, see Pipelines. This page covers how to write them.
Because Bauplan uses vectorized SIMD operations, expectations run at production speed on full tables or partitions, not just samples.
How this differs from other common patterns
Post-hoc validation frameworks, like dbt tests or Great Expectations, commonly execute validation as separate queries after a job builds a table or view. They detect issues only after the object exists, which means invalid data may already be visible to downstream consumers, and jobs used to build the object must be rerun.
Observability and monitoring platforms, like Monte Carlo, run on schedules independent of pipeline execution. They typically profile tables, track distributions, and alert on anomalies or drift. They are not focused on preventing invalid data from being written and do not integrate with pipeline control flow.
Why use expectations?
Early detection
Expectations catch data quality issues before they propagate through your pipeline, preventing downstream failures and incorrect results.
Data contracts
Well-documented expectations serve as implicit data contracts between teams, systems, and data providers. They codify assumptions about data structure, ranges, completeness, and relationships.
Domain knowledge capture
Expectations encode domain-specific rules and business logic directly in your data infrastructure. For example, "transaction amounts must be positive" or "user IDs must be unique."
Validation at critical points
Key use cases include:
- External data validation: Validate data from vendors, partner teams, or APIs by writing to an isolated branch, running expectations, then publishing only if all checks pass
- Model output validation: Ensure transformations produce expected results
- Pre-deployment checks: Validate data before promoting pipelines to production
How expectations work
Basic structure
Expectations are Python functions decorated with @bauplan.expectation():
from typing import Annotated
import pyarrow
import bauplan
from bauplan.standard_expectations import expect_column_no_nulls
class CriticalField(bauplan.TableSchema):
"""The column this expectation validates."""
critical_field: bauplan.String
@bauplan.expectation()
@bauplan.python('3.11')
def test_critical_field_completeness(
data: Annotated[
pyarrow.Table,
bauplan.Model('my_model', projection_schema=CriticalField),
],
) -> bool:
"""Validate that critical_field has no null values."""
return expect_column_no_nulls(data, 'critical_field')
Standard expectations library
Bauplan provides a built-in library covering common validation scenarios:
- Nullness checks:
expect_column_no_nulls,expect_column_all_null,expect_column_some_null - Accepted values:
expect_column_accepted_values - Uniqueness:
expect_column_all_unique,expect_column_not_unique - Statistical properties:
expect_column_mean_greater_than,expect_column_mean_greater_or_equal_than,expect_column_mean_smaller_than,expect_column_mean_smaller_or_equal_than - Derived columns:
expect_column_equal_concatenation
You can also write custom expectations. See Custom expectations below.
Nullness checks
from bauplan.standard_expectations import (
expect_column_no_nulls,
expect_column_all_null,
expect_column_some_null
)
# Ensure no missing values in critical fields
expect_column_no_nulls(table, 'user_id')
# Verify optional field is completely empty
expect_column_all_null(table, 'deprecated_field')
# Check that field has at least some nulls (e.g., for optional data)
expect_column_some_null(table, 'middle_name')
Value constraints
from bauplan.standard_expectations import expect_column_accepted_values
# Restrict to enumerated values
expect_column_accepted_values(
table,
'status',
accepted_values=['pending', 'approved', 'rejected']
)
# Validate categorical data
expect_column_accepted_values(
table,
'payment_method',
accepted_values=['credit_card', 'debit_card', 'paypal', 'bank_transfer']
)
Statistical validations
from bauplan.standard_expectations import (
expect_column_mean_greater_than,
expect_column_mean_smaller_than
)
# Validate average order value is reasonable
expect_column_mean_greater_than(data, 'order_amount', 10.0)
expect_column_mean_smaller_than(data, 'order_amount', 10000.0)
Derived column validation
from bauplan.standard_expectations import expect_column_equal_concatenation
# Verify computed field matches expected concatenation
expect_column_equal_concatenation(
data,
target_column='full_name',
columns=['first_name', 'last_name'],
separator=' '
)
Uniqueness checks
from bauplan.standard_expectations import (
expect_column_all_unique,
expect_column_not_unique
)
# Ensure primary keys are unique
expect_column_all_unique(table, 'transaction_id')
# Expect the column to have at least one duplicate value
expect_column_not_unique(table, 'customer_id')
Custom expectations
The standard library covers common checks, but you can also create your own expectations. Say you want to check the freshness of your data:
import datetime
import pyarrow as pa
import pyarrow.compute as pc
def check_data_recency(table: pa.Table, column: str, hours: int) -> bool:
"""Check that the newest timestamp in the column is within the last N hours."""
latest = pc.max(table.column(column)).as_py()
if latest is None:
return False
cutoff = datetime.datetime.now(datetime.UTC) - datetime.timedelta(hours=hours)
return latest >= cutoff
Now to make it run as a part of the pipeline, we need to wrap it in a function with a @bauplan.expectation() decorator:
from typing import Annotated
import pyarrow
import bauplan
class Timestamps(bauplan.TableSchema):
"""The column this expectation checks for freshness."""
timestamp: bauplan.TimestampMicroUTC
@bauplan.expectation()
@bauplan.python('3.11')
def test_data_freshness(
data: Annotated[pyarrow.Table, bauplan.Model('model', projection_schema=Timestamps)],
) -> bool:
return check_data_recency(data, 'timestamp', hours=24)
Handling expectation failures
By default, Bauplan enforces expectations: if any expectation fails, the pipeline halts immediately.
When failures are not critical, you can log them and let the pipeline continue by disabling strict mode on the run command: pass --no-strict in the CLI or strict=False in the Python SDK.
Summary
Expectations provide a powerful framework for maintaining data quality throughout your pipelines. By encoding validation logic as testable functions, you can:
- Catch issues early before they impact downstream systems
- Document data assumptions and business rules
- Build confidence in data quality
- Create self-documenting pipelines
Well-implemented expectations transform data quality from a reactive problem into a proactive system property.