AI Agent Benchmarks Are Finally Getting Real

Closed models hit ~48% on Zapier's 657-task automation bench — even winners violate business rules half the time.

The Zapier Leaderboard Is Sobering

Artificial Analysis launched a new leaderboard for Zapier's automation bench: 657 tasks across 40 simulated SaaS apps, scored on objectives and guardrails. Results: Fable 5 leads at 48.6%, Opus 4.8 at 48.5%, Gemini 3.5 Flash at 42.6%, GBD 5.5x high at 42.1%. The target floor is 70–80%. The best agent in the world still fails basic guardrails roughly half the time.

Open Models Aren't Close

The top open-weight entry, GLM 5.2 Max, scores 27.8% — a >20-point gap behind the closed Frontier tier. If you need to enforce company policy, open models are not viable for end-to-end agent work yet.

Stop Using Scalar Scores

AA also shipped six domain-specific indices (finance/accounting, legal, healthcare/medical, strategy/ops, engineering, economics). The creator cites Francois: a single number like 75% on XYZ is completely meaningless. Report it as 75% at $10/task instead, because the price-performance frontier has gotten steep — some models are cheap and domain-useless, others are expensive and only worth it in narrow contexts.

Memory Is The Real Bottleneck

Two inference-time fixes shipped this week: - ATMA tackles "ghost memory" — stale and current facts retrieved together. On the LTP benchmark it added 0.24 absolute to conflict accuracy for graphidi. - Recontext is a training-free long-context harness that replaced internal evidence pre-generation across eight 128k datasets. - Block search is pushing million-token in-context retrieval.

The pattern: memory is being engineered at inference, not waited out in the next pretraining run.

Gemini Flash Is The Cost Outlier

Gemini 3.5 Flash stands out for objective-per-guardrail-violation and cost efficiency at 42.6%. If you're running price-sensitive workflows, route traffic through it first.