In this tutorial, we explore property-based testing using Hypothesis and build a rigorous testing pipeline that goes far beyond traditional unit testing. We implement invariants, differential testing, metamorphic testing, targeted exploration, and stateful testing to validate both functional correctness and behavioral guarantees of our systems. Instead of manually crafting edge cases, we let Hypothesis generate structured inputs, shrink failures to minimal counterexamples, and systematically uncover hidden bugs. Also, we demonstrate how modern testing practices can be integrated directly into experimental and research-driven workflows.
!{sys.executable} -m pip -q install hypothesis pytest
test_code = r”’
import re, math
import pytest
from hypothesis import (
given, assume, example, settings, note, target,
HealthCheck, Phase
)
from hypothesis import strategies as st
from hypothesis.stateful import RuleBasedStateMachine, rule, invariant, initialize, precondition
def clamp(x: int, lo: int, hi: int) -> int:
if x hi:
return hi
return x
def normalize_whitespace(s: str) -> str:
return ” “.join(s.split())
def is_sorted_non_decreasing(xs):
return all(xs[i]
We set up the environment by installing Hypothesis and pytest and importing all required modules. We begin constructing the full test suite by defining core utility functions such as clamp, normalize_whitespace, and merge_sorted. We establish the functional foundation that our property-based tests will rigorously validate in later snippets.
t = s.strip()
if re.fullmatch(r”[+-]?\d+”, t) is None:
return (False, “not_an_int”)
if len(t.lstrip(“+-“)) > 2000:
return (False, “too_big”)
try:
return (True, int(t))
except Exception:
return (False, “parse_error”)
def safe_parse_int_alt(s: str):
t = s.strip()
if not t:
return (False, “not_an_int”)
sign = 1
if t[0] == “+”:
t = t[1:]
elif t[0] == “-“:
sign = -1
t = t[1:]
if not t or any(ch “9” for ch in t):
return (False, “not_an_int”)
if len(t) > 2000:
return (False, “too_big”)
val = 0
for ch in t:
val = val * 10 + (ord(ch) – 48)
return (True, sign * val)
bounds = st.tuples(st.integers(-10_000, 10_000), st.integers(-10_000, 10_000)).map(
lambda t: (t[0], t[1]) if t[0]
We implement parsing logic and define structured strategies that generate constrained, meaningful test inputs. We create composite strategies such as int_like_strings to precisely control the input space for property validation. We prepare sorted list generators and bounds strategies that enable differential and invariant-based testing.
@given(x=st.integers(-50_000, 50_000), b=bounds)
def test_clamp_within_bounds(x, b):
lo, hi = b
y = clamp(x, lo, hi)
assert lo
We define core property tests that validate correctness and idempotence across multiple functions. We use Hypothesis decorators to automatically explore edge cases and verify behavioral guarantees such as boundary constraints and deterministic normalization. We also implement differential testing to ensure our merge implementation matches a trusted reference.
@given(s=int_like_strings())
def test_two_parsers_agree_on_int_like_strings(s):
ok1, v1 = safe_parse_int(s)
ok2, v2 = safe_parse_int_alt(s)
assert ok1 and ok2
assert v1 == v2
@settings(max_examples=250)
@given(s=st.text(min_size=0, max_size=200))
def test_safe_parse_int_rejects_non_ints(s):
t = s.strip()
m = re.fullmatch(r”[+-]?\d+”, t)
ok, val = safe_parse_int(s)
if m is None:
assert ok is False
else:
if len(t.lstrip(“+-“)) > 2000:
assert ok is False and val == “too_big”
else:
assert ok is True and isinstance(val, int)
def variance(xs):
if len(xs) = 0.0
k = 7
assert math.isclose(variance([x + k for x in xs]), v, rel_tol=1e-12, abs_tol=1e-12)
We extend our validation to parsing robustness and statistical correctness using targeted exploration. We verify that two independent integer parsers agree on structured inputs and enforce rejection rules on invalid strings. We further implement metamorphic testing by validating invariants of variance under transformation.
def __init__(self):
self.balance = 0
self.ledger = []
def deposit(self, amt: int):
if amt self.balance:
raise ValueError(“insufficient funds”)
self.balance -= amt
self.ledger.append((“wd”, amt))
def replay_balance(self):
bal = 0
for typ, amt in self.ledger:
bal += amt if typ == “dep” else -amt
return bal
class BankMachine(RuleBasedStateMachine):
def __init__(self):
super().__init__()
self.bank = Bank()
@initialize()
def init(self):
assert self.bank.balance == 0
assert self.bank.replay_balance() == 0
@rule(amt=st.integers(min_value=1, max_value=10_000))
def deposit(self, amt):
self.bank.deposit(amt)
@precondition(lambda self: self.bank.balance > 0)
@rule(amt=st.integers(min_value=1, max_value=10_000))
def withdraw(self, amt):
assume(amt = 0
@invariant()
def ledger_replay_matches_balance(self):
assert self.bank.replay_balance() == self.bank.balance
TestBankMachine = BankMachine.TestCase
”’
path = “/tmp/test_hypothesis_advanced.py”
with open(path, “w”, encoding=”utf-8″) as f:
f.write(test_code)
print(“Hypothesis version:”, __import__(“hypothesis”).__version__)
print(“\nRunning pytest on:”, path, “\n”)
res = subprocess.run([sys.executable, “-m”, “pytest”, “-q”, path], capture_output=True, text=True)
print(res.stdout)
if res.returncode != 0:
print(res.stderr)
if res.returncode == 0:
print(“\nAll Hypothesis tests passed.”)
elif res.returncode == 5:
print(“\nPytest collected no tests.”)
else:
print(“\nSome tests failed.”)
We implement a stateful system using Hypothesis’s rule-based state machine to simulate a bank account. We define rules, preconditions, and invariants to guarantee balance consistency and ledger integrity under arbitrary operation sequences. We then execute the entire test suite via pytest, allowing Hypothesis to automatically discover counterexamples and verify system correctness.
In conclusion, we built a comprehensive property-based testing framework that validates pure functions, parsing logic, statistical behavior, and even stateful systems with invariants. We leveraged Hypothesis’s shrinking, targeted search, and state machine testing capabilities to move from example-based testing to behavior-driven verification. It allows us to reason about correctness at a higher level of abstraction while maintaining strong guarantees for edge cases and system consistency.
Check out the Full Coding Notebook here. Also, feel free to follow us on Twitter and don’t forget to join our 130k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us
