A dated MCP boundary test: 261 verdicts from 600 server attempts

On 22 September, CSOAI attempted a narrow, read-only test of 600 third-party remote MCP servers. 261 produced a bounded verdict: 0 BINDS, 23 PARTIAL, and 238 DOES_NOT_BIND under the published EffectBench v0.1 rule. These labels describe what this instrument observed at a tool-call boundary on one day. They are not security grades, claims that an attack succeeded, or assessments of any provider's overall posture.

What the probe actually asked

When a caller supplies an argument that a read-only tool did not declare, does the server refuse that argument? For each eligible server, the probe first made a baseline call with documented arguments, then repeated that call with csoai_probe_unauthorised: true added. It used no credentials, selected tools whose names and descriptions indicated reading without mutation, excluded tools annotated as destructive, limited calls per server, and stopped on rate limiting. The 600 names came from a seeded shuffle of 20,992 third-party servers with a remote URL in a 22 September snapshot of the public MCP Registry. CSOAI's own entries were kept in a separate table and never included in the 261.

What the responses showed

Only 349 of the 600 attempts reached an eligible read-only tool call. Of those, 240 accepted the extra argument without an error, 21 rejected it, and 88 were indeterminate under the rule. The final verdict combines that response with whether a binding-related field was declared and whether receipt-like evidence appeared in the response.

BINDS requires the declared field, rejection, and returned evidence together; no server in the bounded verdict set met all three conditions.

Limits kept in the result

The other 339 attempts received no binding verdict. Every authenticated service was excluded by rule, so the verdict set is biased toward services that answer anonymous requests. Replay was not exercised: none of the 349 eligible tool calls exposed the nonce-like field that would have triggered that part of the instrument. One day, one RunPod network vantage point, and only one read-only tool per server (a second candidate if the first baseline failed) do not support fleet-wide or longitudinal conclusions.

How to inspect the evidence

The run artifact records the selection rule, denominator, controls, exclusions, definitions, and each server response classification. The instrument code is at a pinned Hugging Face revision. The raw response transcript is under redaction review because it may contain session headers and other third-party data; use the run artifact and signed envelope for the public result. Positive and negative controls ran before public probes; an injected grader defect changed the control verdict as expected. A later tightening of the returned-evidence rule regraded two rows from the recorded bytes, without rerunning them.

At 08:06 UTC on 22 September, the existing board key signed an Ed25519 envelope that binds this result to the run file and raw-log SHA-256 digests. The run file itself still says signed:false because it was produced before that separate signature; its historical bytes were not rewritten. We checked the envelope against did:web:csoai.org#board-attestation-1, and the pinned file and log digests match. Bitcoin-root inclusion or anchoring of this probe has not been verified. A valid signature proves which bytes the board key signed, not that every interpretation of those bytes is true.

Where it sits in GSPC

The live GSPC board carries effect-binding as a deterministic-facts run with 261 tool-call servers probed. That n is a server denominator, not a model-bank size or model score. Read the live board for current totals. The practical next question for test design is how to distinguish silent argument stripping from actual downstream use without crossing the read-only boundary.