Great Expectations: Unit Tests for Your Data
Great Expectations brings unit testing to your data, letting you assert what a dataset should look like. It validates data within a pipeline, preventing bad data from corrupting models or reports.
Why it exists
Software engineers use automated tests to manage complexity and ensure code reliability. Data pipelines are often just as complex, but frequently lack this discipline, leading to silent failures. Great Expectations was created to bring the same confidence and acceleration to data by making it possible to test the data itself.
The mental model
Think of it as unit tests for your data. Instead of asserting that a function's output matches an expected value, you write assertions about the data's properties. For example, you declare expect_column_values_to_be_between(column, min, max). It's a declarative way to state your assumptions about a dataset's shape, content, and statistical properties.
How it works
You define a collection of "Expectations" in Python, forming an "Expectation Suite." You then run this suite against a batch of data, such as a Pandas DataFrame, Spark DataFrame, or a table in a SQL database. The library returns a validation report detailing which assertions passed or failed, including examples of non-conforming data to speed up debugging. It also automatically generates "Data Docs," a human-readable HTML site from your suites and validation results, creating a living document of your data quality.
When to use it
Use Great Expectations as a quality gate in your data pipelines. Three common use cases: first, validating data from external vendors to catch issues at the source; second, checking the output of a transformation step to ensure its logic is correct; third, preventing low-quality data from entering production data warehouses or machine learning models. It turns implicit knowledge from subject-matter experts into explicit, automated tests.
When not to use it
Great Expectations is not a pipeline execution framework. It integrates with tools like Airflow, Prefect, or Dagster but does not run your DAGs. It is also not a data versioning tool like DVC or lakeFS; it stores metadata about your data (the Expectations and results), not the data itself. Finally, as a Python-native library, it may not be the best choice for a pure R or other non-Python environment.
One canonical example
A team ingesting NYC taxi data could create an Expectation Suite asserting that the passenger_count column must be between 1 and 6, the trip_distance must be positive, and the payment_type must be one of a few known values. Running this validation as the first step in their pipeline prevents bad records from propagating downstream and corrupting analytics.
Interview question
Which statement best describes the primary role of Great Expectations within a data pipeline?
- a.It provides a framework for defining and enforcing data quality assertions.Correct
- b.It serves as a comprehensive tool for data version control and lineage tracking.
- c.It orchestrates the execution of data transformations and workflow dependencies.
- d.It automatically cleanses and transforms raw data into a usable format.
Why? this is the answer
Option A accurately describes Great Expectations' core function: defining 'Expectations' to assert data properties and validate data quality, acting as a quality gate. Option D is incorrect because Great Expectations identifies data quality issues but does not automatically perform cleansing or transformation; it reports on the issues for users to address.
Just read this? Test yourself on what you have been reading.
Read the original → docs.greatexpectations.io
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on mlops — each one lists the topics its interview covers.
See open roles