Overview
Asage-DesignBench measures how well current image models perform professional design work. We collected 50 briefs that professional users wrote for real creative work, covering five areas: logo and brand marks, typography, poster and ad layout, packaging, and infographics. We generated one image per brief with each model and compared every pair of outputs. Three working designers judged each pair blind on six dimensions and overall.
This yields 496 pairwise comparisons and 1,480 expert judgments, which we convert to Elo ratings with a Bradley-Terry model. To test whether cheaper judges reproduce the expert signal, we re-judged all comparisons with a crowd of ten annotators per pair and with two vision-language models. The experts separate the models clearly and identify specific weaknesses in each. The crowd does not.
What's in the benchmark
All 50 briefs used in this benchmark are prompts that professional users wrote for real creative work. Each specifies a deliverable, the required elements, constraints and visual direction, in the form a client or creative director would use. There are ten briefs per area, ranging from a logo request to a packaging mockup with exact copy and finish.
Clean packaging mockup of a chocolate bar wrapper lying flat on a solid pastel background, front view, accurate sharp typography reading "CASTELLO" above "70% Dark · Sea Salt", brand logo centered, realistic foil-and-paper print finish, soft studio lighting, no warped edges, no repeating artifacts.
| Model | Maker |
|---|---|
| GPT Image 2 | OpenAI |
| Nano Banana 2 Pro | |
| Grok Imagine | xAI |
| MAI-2.5 | Microsoft |
| Krea 2 Large | Krea |
Each pair was judged by three designers from a panel of ten designers with experience in brand, print or campaign work. Judges saw the brief and the two images side by side, in random left-right order and without model names, and chose A, B or tie on six dimensions and overall: brief adherence, composition and hierarchy, typographic execution, concept and distinctiveness, style and coherence, and production readiness. The Leaderboard tab defines the ratings, intervals and win rates.
Results
Key findings
- GPT Image 2 ranks first on five of six dimensions and wins every head-to-head matchup, taking 66% to 83% of decisive judgments. Its one relative weakness is concept and distinctiveness, -33 against its own mean.
- Execution quality determines the ranking more than concept quality. Krea 2 Large has the most distinctive concepts relative to its own profile (+32) and ranks last, pulled down by production readiness (-26) and typographic execution (-22). Nano Banana 2 Pro nearly matches GPT Image 2 on brief adherence and ranks second because of composition and hierarchy (-31).
- Text and structured layout are where models break. The three lowest scores in the benchmark are MAI-2.5 on typography (29% win rate), Krea 2 Large on infographics (14%) and Krea 2 Large on packaging (27%).
- Crowd annotators do not reproduce expert judgments. Two crowd annotators agree 53% of the time, chance level, and the crowd majority matches the expert majority on 53% of comparisons. The crowd compresses a 45-point spread between models into 12 and ranks Grok Imagine above Nano Banana 2 Pro, the reverse of the designers.
- VLM judges disagree with the expert majority on 22% of decisive comparisons on average, rising to 30% on concept and distinctiveness, the dimension where they disagree most.
| Rank | Spread | Model | Elo | 95% CI | Win rate | Judgments |
|---|---|---|---|---|---|---|
| 1 | 1–1 | GPT Image 2OpenAI | 1163 | 1133–1190 | 0.72 | 590 |
| 2 | 2–2 | Nano Banana 2 ProGoogle | 1041 | 1016–1065 | 0.56 | 596 |
| 3 | 3–4 | Grok ImaginexAI | 980 | 956–1005 | 0.47 | 596 |
| 4 | 3–4 | MAI-2.5Microsoft | 963 | 937–987 | 0.45 | 585 |
| 5 | 5–5 | Krea 2 LargeKrea | 854 | 828–881 | 0.30 | 593 |
Table 1. Overall verdict on all 496 comparisons. Spread is the 95% range of ranks across bootstrap resamples; win rate counts a tie as half a win.
Model results
GPT Image 2 has the highest Elo on five of six dimensions, with its largest lead on production readiness. Nano Banana 2 Pro is within 65 points of it on brief adherence, the dimension that measures whether the brief was delivered. The gap between the top two models and the remaining three is larger than any gap among the remaining three.
Figure 1. Elo of each model on each dimension, from six separate fits on the same 496 comparisons judged by the expert panel. Models are ordered by their Elo within each dimension. The leading model is drawn in full colour and the others lighter. The thin line through each bar is the 95% confidence interval.
Figure 2 shows each model's Elo on a dimension minus its mean over the other five. GPT Image 2 scores +39 on production readiness and -33 on concept and distinctiveness. Nano Banana 2 Pro scores +48 on brief adherence and -31 on composition and hierarchy. MAI-2.5 scores +28 on composition and -37 on brief adherence. Krea 2 Large scores +32 on concept and -26 on production readiness. These contrasts show where each model falls behind its own level, which is the information needed to prioritise improvements.
Figure 2. Each model's Elo on a dimension minus its mean over the other five dimensions. Bars to the right are relative strengths and bars to the left relative weaknesses. A bar is coloured where the 95% interval excludes zero and grey where it does not.
GPT Image 2 has a win rate of at least 68% in every design area. The lowest scores are Krea 2 Large on infographics (14%) and packaging (27%), and MAI-2.5 on typography (29%).
Figure 3. Win rate of each model against all other models within each design area, on the overall verdict, with a tie counted as half a win. Models are ordered by win rate within each area. The leading model is drawn in full colour. The thin line is the 95% interval. Each area has ten briefs.
Where models fail
Figures 2 and 3 locate each model's weaknesses. A summary is shown in Table 2: the dimension furthest above and below each model's own mean, where the interval excludes zero, and the design area with the lowest win rate.
| Model | Relative strength | Relative weakness | Lowest win rate by area |
|---|---|---|---|
| GPT Image 2 | Production readiness, +39 | Concept & distinctiveness, -33 | Poster & ad layout, 68%* |
| Nano Banana 2 Pro | Brief adherence, +48 | Composition & hierarchy, -31 | Logo & brand marks, 48%* |
| Grok Imagine | None separated from the mean | Style & coherence, -23 | Logo & brand marks, 38% |
| MAI-2.5 | Composition & hierarchy, +28 | Brief adherence, -37 | Typography, 29% |
| Krea 2 Large | Concept & distinctiveness, +32 | Production readiness, -26 | Infographics & data viz, 14% |
Table 2. Where each model is strongest and weakest. The first two columns are the within-model contrasts of Figure 2, restricted to dimensions whose 95% interval excludes zero. The last column is the design area with the lowest win rate against all other models in Figure 3. An asterisk marks a win rate whose interval includes 50%.
For some comparisons the designers also wrote a critique. The defects they name fall into a small number of kinds: garbled or misspelled type, a named weight, typeface, colour or style not followed, an element from the brief missing, the wrong number of items or chart values that contradict their labels, elements the brief did not ask for, weak hierarchy or framing, and rendering artifacts such as stretched or warped forms. One designer wrote that “typography in A inside some of the rooms are misspelled or unrecognizable.” Another, on a missing element: “it only executed the first part of the four key findings along with an icon. Everything that follows, the bar chart and other elements, is missing.” On chart logic: “Option A donut chart shows 100% even if the text reads 82% and 64%.”
Crowd versus experts
We re-judged all 496 comparisons on brief adherence with two alternative judges: a crowd of ten annotators per comparison, and two vision-language models, GPT-6 Astra and Gemini 3.8 Flash, each shown every pair in both image orders.
Crowd and expert rankings disagree: the crowd narrows the gaps between models and reverses two positions. Two designers agree on 70% of decisive votes; two crowd annotators agree on 53%, close to chance. Expert win shares span 45 points across the five models; crowd win shares span 12. The crowd ranks Grok Imagine above Nano Banana 2 Pro, and the designers rank them the other way with non-overlapping intervals.
Figure 4. Share of brief-adherence comparisons won against all other models on the same 496 comparisons, as judged by the expert panel, the crowd and the two VLM judges. Thin lines are 95% intervals. Expert win shares span 45 points, crowd win shares 12 and VLM win shares 77.
The VLM judges fall between the crowd and the experts. They agree with the expert majority on 83% of decisive comparisons on brief adherence and 70% on concept and distinctiveness, with the other dimensions in between. However, they change 13% of their verdicts when the two images are swapped, and they exaggerate the differences between models: the 45-point expert spread becomes 77 points.
The examples below are comparisons where the crowd majority voted against a unanimous expert panel. In each case the crowd chose the image with more visual content, such as a larger element or bolder type, and the experts chose the image that met the requirements stated in the brief.
A brand identity board presented flat on a white background, showing a logo lockup and its color specification. The primary mark is a solid circular badge containing a stylized wave in negative space, above the wordmark "NORHAVN" in a bold grotesque. To the right of the mark, five color swatches are arranged in a horizontal row as rounded rectangles, each labeled beneath with its hex value in small monospaced type: #0B3C5D deep marine, #328CC1 mid blue, #D9B310 brass, #1D2731 near black, and #F4F4F2 off white. The swatch colors must match their stated hex values exactly. Clean, precise layout with even spacing.
A digital graphic featuring text against a solid off-white background (#F5F6F2), a detail shot from a website design. The composition is dominated by text in a left-aligned format. In the top left corner, the words "Half / Measure" are displayed in a large bold serif, rendered in bright neon green (#33FF33). The word "Half" sits directly above "Measure." Starting immediately to the right of "Measure" and continuing downward across the middle and bottom sections, a paragraph appears in a smaller but still large bold serif. This text is black. The visible portion reads: "is Iva Petrova and / Dimov — a video production team / Plovdiv. Iva has honed her craft in st / skate photography and refined it as p". The text cuts off at the right edge of the frame, indicating the paragraph extends beyond the visible area. The visual style is clean and modern, emphasizing the contrast between the vibrant green title text and the black body text.
Generate a clean, branded single-image social post for a business coach offering a weekly success tip. Use a quote layout: center-aligned bold typography reading "Momentum beats motivation.", light background in hex #F6F1EA, two thin accent rules above and below the quote in hex #2D5F4E, and a small circular headshot of the coach in the lower right corner. Include the wordmark "HALVARD & CO" bottom left in hex #2D5F4E. Ample white space, modern, minimal, 4:5 aspect ratio. Colors must match the specified hex values exactly.
hand-drawn logo for a botanical skincare brand, single continuous line illustration of an olive branch, imperfect organic stroke, muted sage green, paired with the wordmark "AVERLY" in a light modern serif, lots of negative space, calm, understated, natural, vector.
experimental typographic composition, the word "DRIFT" repeated eleven times down the page at decreasing weight and increasing letter spacing, stark black on white, no imagery, grid-based, swiss influence, high contrast, flat, poster-scale, minimal.
The two VLM judges split: one chose B, the other changed its answer when the images were swapped.
Limitations
The benchmark has ten briefs per design area, so per-area results are exploratory. Each model generated one image per brief at default settings, and a different seed could change individual comparisons. The panel consists of ten designers with brand, print and campaign backgrounds judging English briefs. The crowd comparison covers brief adherence only, and the VLM results are for two judge models. Elo ratings are relative to the five models in this set.
Conclusion
Working designers rank the five models consistently and separate them clearly. GPT Image 2 ranks first on execution, brief adherence and production readiness, and concept and distinctiveness is the one dimension where it falls behind its own level. Nano Banana 2 Pro follows the brief nearly as well and loses on layout. Krea 2 Large produces the most distinctive concepts and the least finished output.
The choice of judge changes the result. A crowd of ten annotators per comparison agrees with itself at chance on the most objective dimension and compresses the differences between models. VLM judges track the experts more closely but are sensitive to image order and overstate margins. For measuring or training toward professional judgment, expert panels provide the signal, and crowd and VLM judges approximate it with measurable loss.
License and access
Updated October 2026
- Private evaluation
- The 50 briefs, the 250 images and the 1,480 expert judgments are not public. We run DesignBench as a private evaluation for labs and creative product teams, on these briefs or your own. Contact us for access.
- Citation
- If you use Asage-DesignBench, please cite:
@techreport{asage2026designbench,
title = {Asage-DesignBench: An Expert-Judged Benchmark for Text-to-Image Generation in Professional Design},
author = {{Asage}},
institution = {Asage},
year = {2026},
url = {https://asage.ai/research/designbench/},
}
Appendix
Annotator diagnostics
Ten designers judged the 496 comparisons, three per comparison. Each was presented blind, in randomised left-right order with no model names. Brief adherence, decisive votes.
| Metric | Value | Reading |
|---|---|---|
| Inter-annotator agreement | 70% [65, 73] | Pairwise, decisive votes, chance 50% |
| Krippendorff's alpha | 0.36 | Binary, decisive votes |
| Position bias (pooled) | 49.7% to the left image | 979 decisive votes, p = 0.90 |
| Position bias (per rater) | 43% to 55% | No rater deviates from 50% at p < 0.05 |
| Agreement with the other raters | 81% pooled, 75% to 83% | Each vote against the majority of the other two |
| Tie use | 25% (median rater) | Share of judgments recorded as a tie |
Crowd and VLM judge setup
Crowd: ten annotators per comparison from an English-speaking pool (US, GB, CA, AU, IE, NZ), collected in two waves of five with an unsure option, brief adherence only. VLM judges: GPT-6 Astra and Gemini 3.8 Flash, the same rubric as the designers, every pair run in both image orders; a verdict that changed under the swap is excluded as unstable. Judge agreement with the expert majority by dimension ranged from 76% to 85%; verdict flips under swap from 6% to 23%.