Image models are easy to demo and surprisingly hard to evaluate. A result can look polished while missing the instruction, changing the wrong part of an image, or quietly getting one line of text wrong.
I built Image Evals to make those failures visible. It is a deliberately small Python system for running generation and editing cases, grading each output against a use-case-specific rubric, and saving enough evidence to understand every verdict afterward.
Start with the task, not one universal score
The first suite covers six cases across four kinds of work: a mobile UI mockup, a logo, a marketing flyer, and a multi-reference virtual try-on. Logo and flyer outputs are edited again so the system can test whether a model makes the requested change without disturbing everything else.
Each task gets its own evaluation contract. UI mockups need readable text and plausible affordances. Marketing flyers treat exact copy as a hard gate. Virtual try-on compares the result with both the person and the selected garment, keeping facial similarity, outfit fidelity, and body preservation separate.
The vision judge returns structured observations and evidence for every metric. Python applies the final pass/fail rules. A high visual-quality score cannot compensate for missing copy or a failed instruction.
The useful failure
The first bounded run used GPT Image 2 at low quality and GPT-5.4 as the judge. Five of the six cases passed. The marketing flyer failed.
At first glance, it was one of the strongest images in the set: warm palette, clear hierarchy, appetising photography, and clean typography. But the model added NORTHSTAR COFFEE to the product bag and changed the requested date punctuation. Because exact copy was a hard gate, the result failed instead of being rescued by its 5/5 visual scores.
That was the most useful result in the run. The system caught the kind of attractive-but-unusable output that a broad aesthetic score would hide.
Plain files, inspectable runs
Cases, prompts, model settings, and grader rubrics live in YAML and Markdown. Every run preserves the rendered prompt, source images, output, structured grade, evidence, latency, hashes, and a request ledger. The final verdict remains deterministic and easy to audit.
A light local viewer puts the references, candidate image, grader evidence, cost, configuration, and human feedback in one place. Opening or refreshing the viewer never calls a provider, which makes it safe to inspect results without spending more credit.
The initial smoke run made twelve API calls with no retries and recorded an estimated cost of $0.2273. That is evidence that the plumbing works—not a model ranking. The next step is to add stronger cases and compare the same frozen suite across other image providers.
The code and complete implementation report are available on GitHub.