Agent trust scorecard
Our own report card, warts included.
We test the live API and MCP server the way an agent uses them: 147 tasks across coverage, prices, accuracy, honesty and more. Latest run 2026-10-26 09:44 UTC, harness 1.0.4, task set 2026-10-25.2. 34 runs since 2026-10-25, when we started at 66.5.
Read this first: we wrote the test. The task set is ours, so it can lean toward what we already cover. It does not measure uptime (one box, one region), the accuracy sample is small (12 providers and 12 prices, re-fetched each run) and RFQ outcomes are not scored yet because no quote request has been forwarded. Read the trend, not one number.
Score history
By criterion
Change is since the first run (2026-10-25). The overall score is the weighted mean of the scored criteria.
| Criterion | Latest | Change | What it measures |
|---|---|---|---|
| Coverageweight 2 | 69.9 | +45.9 | A buyer query (job × country × language) finds the right job and a verified provider that serves that country Last run: live e2e 100.0% of 80 queries; relevance@3 100.0%; incl. worldwide 100.0%; dataset jobs x 36 markets local 24.7%; v1.0.1-comparable coverage 69.9 |
| Price coverageweight 1.5 | 63.1 | +33.5 | Share of jobs with a sourced price from the last 12 months, and with 3 or more prices Last run: jobs with a price <=12 mo 74.4%; jobs with >=3 prices 36.7%; rows with day-precision as_of 57.8% |
| Accuracyweight 1.5 | 100 | ±0 | A sample of providers and prices re-fetched each run: is it still true at the source? Last run: providers ok 100.0% of 12 verifiable; price numbers found on source 100.0% of 11 verifiable; coverage-label inconsistencies 0 |
| Provenanceweight 1 | 91.6 | +11.6 | Every price and provider has a source URL and a date (day precision preferred) Last run: price rows sourced+dated 100.0%, providers sourced+dated 100.0%; penalty for month-only as_of (cannot tell retrieval vs publication date) |
| Reliabilityweight 1 | 100 | ±0 | Calls that fail unexpectedly, and p95 latency against 800 ms (one box, one run) Last run: 268 calls, 0 unexpected failures, p50 66 ms, p95 216 ms (single run from one box; not an uptime measure) |
| Schema stabilityweight 0.75 | 100 | +28.6 | Versioned API, changelog, output schemas, nothing removed since the last run Last run: all checks pass |
| Quote templatesweight 0.75 | 100 | +18.7 | Job-specific fields a supplier needs, plus a submit → status → delete test Last run: templates with >=4 job fields + example 100.0%; distinct field sets 308 of 308 jobs; lifecycle test pass |
| Honestyweight 1.5 | 100 | +14.3 | Empty answers for nonsense jobs, no invented estimates, no fake trends, prompt-injection handled Last run: 24 of 24 pass |
| Agent ergonomicsweight 1 | 100 | +7.1 | Tool descriptions, annotations, closed input schemas, structured output, errors, ETags, CORS Last run: all checks pass |
| Discoverabilityweight 1 | 70 | +30 | Our own discovery files (40%) and listings in machine-checked MCP registries (60%) Last run: own discovery files 9/9; listed in 1/2 machine-checked registries; 15 more directories need a manual check |
| Robot vs humanweight 0.75 | 100 | Robot-or-person comparisons that are sourced, labelled as modelled and give a crossover range Last run: 8/8 MCP comparisons sourced, labelled and with a crossover range; REST jobs ok, REST compare ok | |
| RFQ outcomes | n/a | Quotes received, supplier response time, jobs done. Not scored until quote requests are forwarded to suppliers. |
What would move it
- Coverage (69.9). Most job × country cells still have no provider that lists that country as a service area. "Worldwide" claims don't count.
- Price coverage (63.1). Many jobs still have no published price anywhere we could find. We'd rather show an empty table than a made-up one.
- Discoverability (70). own discovery files 9/9; listed in 1/2 machine-checked registries; 15 more directories need a manual check.
Also checked
20 of 20 example prompts passed when replayed against the live server on 2026-10-26. Every run's scores: /agents/trust/history.json (CC BY 4.0). Changes to the API: changelog.