DisclosureThe benchmark uses synthetic but production-inspired email fixtures, not live production traffic. It compares two systems under controlled conditions and is not evidence of live production performance. Research conducted April 2026.
Executive summary
This report compares two extraction systems benchmarked in the Taskd workflow repository. The first, which we call Medium Tier, runs the attribute extractor on GPT-4o mini. The second, Structured Outputs, runs the structured-outputs extractor on GPT-5 nano. Across the best current summary, a 75-email trimmed comparison of saved runs, Medium Tier produced higher observed accuracy and substantially lower latency. Structured Outputs used fewer tokens and had lower direct API cost.
| System | Emails | Exact | Semantic | Avg latency | Input tokens | Output tokens | Cost per email |
|---|---|---|---|---|---|---|---|
| Medium Tier · GPT-4o mini | 75 | 96.00% | 96.00% | 1,694.4 ms | 457.8 | 43.3 | $0.000095 |
| Structured Outputs · GPT-5 nano | 75 | 85.33% | 86.67% | 24,213.3 ms | 70.8 | 27.1 | $0.000014 |
Current recommendation: use Medium Tier as the operational default when practical performance matters most. Use Structured Outputs when minimising raw API token cost is the overriding requirement.
The result illustrates why Taskd evaluates complete workflow performance rather than choosing a model on price or benchmark reputation alone. Lower token cost did not produce the best observed accuracy or latency in these runs.
Scope and method
The systems were compared on exact and semantic accuracy, average, p50 and p95 latency, input and output token use, and estimated API cost. Exact accuracy counts outputs that matched the expected result exactly. Semantic accuracy counts outputs whose meaning matched after light normalisation. Cost per email is input tokens times input price plus output tokens times output price, using April 2026 list pricing: $0.15 input and $0.60 output per million tokens for GPT-4o mini, and $0.05 input and $0.40 output per million tokens for GPT-5 nano. Monthly estimates assume 1,500 emails.
The analysis uses saved benchmark artifacts from the extraction repository. The original fixtures include some clean, simplified examples. Newer, harder fixtures were written to resemble realistic email workflows: forwarding and thread context, older instructions mixed with newer ones, agent and customer ambiguity, port and product ambiguity, and quantity corrections. The fixture set should therefore be described as synthetic but production-inspired. A live-production cost model would require sampled production emails or logged production token usage.
Headline readout
- Medium Tier was approximately 14 times faster on average.
- Medium Tier led exact accuracy by 10.67 percentage points.
- Structured Outputs used about 85% fewer input tokens.
- Structured Outputs had approximately 85% lower estimated direct API cost.
- At 1,500 emails a month, absolute direct model cost remained low for both systems: roughly $0.14 against $0.02.
Across all 85 saved runs the picture is the same, with Medium Tier at 96.47% exact accuracy against 87.06%. The Structured Outputs latency average in the untrimmed set is inflated by one stability run containing a major long-tail outlier, which is why the trimmed 75-email view above is the more representative comparison.
Harder, production-inspired slice
| System | Emails | Exact | Semantic | Avg latency | p95 | Cost per email |
|---|---|---|---|---|---|---|
| Medium Tier · GPT-4o mini | 20 | 100.00% | 100.00% | 1,171.4 ms | 1,726 ms | $0.000093 |
| Structured Outputs · GPT-5 nano | 20 | 95.00% | 95.00% | 25,346.9 ms | 40,408 ms | $0.000014 |
On the slice written to look like real inbox traffic, Medium Tier produced higher observed accuracy and remained substantially faster. Structured Outputs remained cheaper in direct token cost.
Trade-off summary
| Dimension | Current leader | Reason |
|---|---|---|
| Baseline accuracy | Tie | Both systems scored 100% on the five-email baseline |
| Combined accuracy | Medium Tier | Higher weighted exact and semantic accuracy |
| Hard-slice accuracy | Medium Tier | 100% versus 95% |
| Speed | Medium Tier | Roughly 1.2 to 1.7 seconds versus 22 to 25 seconds in representative runs |
| Token efficiency | Structured Outputs | Far fewer average input and output tokens |
| Direct API cost | Structured Outputs | Approximately $0.021 versus $0.14 per 1,500 emails |
| Operational default | Medium Tier | Better speed and stronger observed accuracy |
What this report proves, and what it does not
It supports Taskd’s engineering practice of evaluating accuracy, latency, tokens and cost together. It supports the claim that the cheapest extraction approach is not automatically the best operational choice. And it gives a transparent explanation of why a particular extractor was selected.
It does not support a general claim about wasted enterprise tokens. It is not a Taskd-versus-LLM or graph-versus-retrieval comparison. It says nothing about live customer production accuracy or cost, and nothing about performance outside these two systems, fixtures and saved runs.
Next benchmark
The next external-quality evaluation should add a larger held-out dataset, real permissioned production samples, blind scoring, repeated runs with confidence intervals, end-to-end human-review time, error severity rather than match rate alone, a direct comparison between repeated raw-context extraction and reuse of verified structured facts, and a downloadable, versioned artifact bundle.