The CSOAI-census crawler

If you found CSOAI-census/0.1 (+https://councilof.ai/census) in a log, it was us: Council of AI (CSOAI Ltd, UK Companies House 16939677).

What it does

• Public GET requests, and MCP discovery only: initialize and tools/list.
• It never calls a tool, never authenticates and never pays.
• At most one request per second to any host, over one connection per host.
• It reads robots.txt and obeys rules for the product token CSOAI-census.
• It backs off on 429 and 503, honouring Retry-After, and retries once at most.

What it feeds

• csoai/mcp-remote-census — remote MCP endpoints: did initialize answer, and what tools/list returned
• csoai/hf-mcp-spaces-census — Hugging Face Spaces that declare an MCP server
• csoai/a2a-card-census — published A2A agent cards, and whether their signatures verify

Contact

Write to nicholas@csoai.org.

Ask for a re-check

If a row about your endpoint is wrong or out of date, email the endpoint URL with "re-check" in the subject. The next run reads it again, and the dataset row is replaced by what that run observed.

Object or opt out

Email the endpoint or host you want excluded. It goes on an exclusion list that the next run honours, and your objection is recorded. You can also disallow CSOAI-census in your robots.txt, which the crawler reads before it sends anything else.