The question
How do you make an AI workflow more reliable before adding more complexity?
I started with a deliberately small receipt-review pipeline instead of building a full expense product:
extract_receipt_details(image_path)reads a receipt image and returns structured data.evaluate_receipt_for_audit(receipt_details)decides whether the expense needs review.
That separation matters. Extraction failures and business-rule failures should be visible as different problems.
Why start small
The first goal is not a broad product surface. It is to understand failure modes. Each run saves extraction and audit JSON separately, repeated outputs are preserved, and labeled examples can be compared with a lightweight assessment helper.
The later eval pass turns that baseline into a batch workflow: curated receipt references, small graders, repeated audit reruns, and scoreboard reading that starts from the business decision instead of the lowest component score.
That changed the project shape. The audit policy became more deterministic, closed-form checks moved into code, and noisy line-item extraction stopped being treated as a receipt math error by default. The remaining frontier is upstream extraction quality: totals, subtotals, line-item duplication, and handwriting capture.
Implementation details
The workflow is deliberately boring Python:
def review_receipt(image_path: str | Path) -> ReceiptReviewResult:
settings = load_settings()
client = get_client(settings)
receipt_details = extract_receipt_details(
Path(image_path),
client=client,
model=settings.extraction_model,
)
audit_decision = evaluate_receipt_for_audit(
receipt_details,
client=client,
model=settings.audit_model,
)
return ReceiptReviewResult(
image_path=str(Path(image_path)),
receipt_details=receipt_details,
audit_decision=audit_decision,
models=ReviewModels(
extraction=settings.extraction_model,
audit=settings.audit_model,
),
)
The output preservation was one of the most important choices. Re-running the same receipt never overwrites the previous result:
outputs/reviews/extraction/<receipt>.json
outputs/reviews/extraction/<receipt> (1).json
outputs/reviews/audit_results/<receipt>.json
outputs/reviews/audit_results/<receipt> (1).json
That makes model variability inspectable instead of accidental.
Contract shape
The schemas use strict Pydantic models so missing or invented fields are easier to catch:
class ReceiptDetails(StrictModel):
merchant: str | None
location: Location
time: str | None
items: list[ReceiptItem]
subtotal: str | None
tax: str | None
total: str | None
handwritten_notes: list[str]
class AuditDecision(StrictModel):
not_travel_related: bool
amount_over_limit: bool
math_error: bool
handwritten_x: bool
reasoning: str
needs_audit: bool
What it demonstrates
- Structured output contracts with Pydantic
- Image-to-data extraction
- Explicit separation between extraction and business decisions
- Ground-truth comparison
- An eval-driven approach to iteration
- Deterministic checks for closed-form audit rules
- Triage from business outcome back to upstream extraction failures
Related writing
- intelligence too cheap to meter — the thesis behind the series: cheap intelligence makes judgment and evals more valuable.
- Building an Eval-Driven Receipt Auditor From Scratch — Part 1: the first baseline, failure modes, and why the workflow starts simple.
- Eval Driven Development · Receipt Auditor — Part 2: read the scoreboard from the business outcome backward, push closed-form decisions into code, and isolate extraction as the next frontier.
- Eval-Driven Development · Receipt Auditor — Part 3: expose the misleading aggregate score hiding weak handwritten-X recall, build a balanced repeated-run eval, and use it to choose the intervention that actually fixes the failure.
The code is available on GitHub.