About the Agent Disco scanner
If you've arrived here from a hit in your access log, this page explains exactly what our scanner does, how often it visits, and how to allow or block it.
User-Agent
Every request we send carries this User-Agent header:
AgentDisco/1.0 (+https://agentdisco.io/bot)
It's stable across all our clients — the target scanner, the
registry lookups (npm, PyPI, GitHub), the conditional-GET
revalidation path. If you want to identify us in access logs,
match on the AgentDisco/ prefix.
Frequency
On-demand only. We don't crawl the web. A request only happens when:
- A human submits your URL via the form on agentdisco.io.
- A human or agent POSTs to
/api/v1/scans. - Our nightly job rescans sites whose badge has gone stale (typically older than 30 days). The job queues a small batch per night, not the whole index.
There is no background crawler, no sitemap walker, no link following. One scan fetches a fixed list of publicly-advertised URIs against your host and stops.
What we fetch
Every path below is either standardised, self-advertised, or directly linked from a standardised path. The scanner does not guess paths or probe for hidden endpoints.
/— homepage (for HTML meta + manifest links)./robots.txt— crawler directives./llms.txtand/llms-full.txt./sitemap.xml(only if/robots.txtadvertises it)./.well-known/ai-plugin.json,/.well-known/agent.json,/.well-known/mcp.json,/.well-known/security.txt, and the other IANA-registered well-known URIs our checks cover.- The OpenAPI / plugin manifest URLs those well-known files link to, if any.
A single scan typically sends 10–20 requests, spread across your origin at no more than 2 requests/second per host. Every response body is capped (tens of KB) before we stop reading.
Allow-listing
The scanner respects robots.txt. If you want to
explicitly allow us without changing your existing rules, add:
User-agent: AgentDisco
Allow: /To explicitly disallow us — the scanner will honour this and the scan will skip path-level checks:
User-agent: AgentDisco
Disallow: /
If you'd prefer not to appear in our public index at all,
every site owner can unlist themselves via DNS-verified
self-service:
POST /api/v1/websites/{host}/unlist.
Unlisting removes your host from search listings, hides the
report page, and disables the badge.
Who we are
Agent Disco is operated by Starsol Ltd (England, company 06002018). The full list of checks the scanner runs — what it looks for, how it scores, and how to fix common failures — is published at /checks. If something about our behaviour looks wrong, send us a message.