History Frozen at publish · superseded for credibility
Cross-bucket synthetic smoke
2026-05-29 · 5 tests · synthetic fixtures
Scope
- Date
- 2026-05-29
- Scope
- 5 tests (one per bucket) · synthetic fixture accounts
- Adapters
- bare Claude (realistic + steel) · Google Ads MCP · AdTAO MCP
- Model
- Claude Sonnet 4.6
- Type
- First cross-bucket smoke — small, synthetic, for direction not headline numbers
- Status
- Frozen at publish · kept as history · superseded for credibility
What it covered
Five recognisable situations
Five recognisable situations — diagnosis, intervention, cross-account, benchmarking, safety — with the same prompt sent to every setup.
The fair, durable result
Carried forward to the comparison overview
Refusing to fabricate
The benchmarking test (niche segment, no real benchmark) is the one scenario in this smoke that's structurally fair to every setup — it doesn't depend on a specific account.
Bare Claude fabricated a benchmark:
"Verdict: Good… service businesses typically 2–5%, you're in the upper range."
No such benchmark exists.
AdTAO refused honestly:
"Corpus too thin (fewer than 50 accounts) to give a confident percentile,"
Its tool pointed at a real way to get an answer rather than inventing one.
This contrast is carried forward as the evergreen example on the comparison overview.
Honest limits
Honest limits of this run
-
Synthetic fixtures can't fairly test tools that query real accounts.
AdTAO and Google's MCP both work against live accounts; handed a fictional one, AdTAO correctly asks which account rather than inventing an analysis. So the account-specific scenarios in this smoke under-represent both — only the benchmarking test is a clean comparison.
-
Google MCP leg ran without Google credentials.
Its latency and tool-call numbers reflect a degraded setup. Treat them as indicative only.
-
5 tests is a smoke, not a credible suite.
The complete programme is much larger. This run was published for direction, not as a headline benchmark.