AI crawler and agent user-agent directory
These are the AI agent names SynthTrail recognises when a visit announces itself. The directory is free to read; you do not need an account.
$9.99 a month. No contract. Cancel any time.
How to use this list
Search your logs for these names to see which AI tools fetch your pages. Use them in robots.txt rules to allow or disallow a given agent. A name can be faked, which is why SynthTrail also looks at behaviour.
Names versus behaviour
A user-agent is only a claim. Some tokens, like Google-Extended and Applebot-Extended, are control tokens for robots.txt and never appear as a visiting browser. SynthTrail labels sessions by name when a browser announces one, and flags unnamed ones by what they do.
Keeping it current
Vendors add and rename agents often. Treat the list as a starting point and check each vendor's own documentation before you write rules around a name.
Recognised AI agents
| User-agent name | Vendor | What it is for |
|---|---|---|
| GPTBot | OpenAI | Collects pages that may be used to train models |
| OAI-SearchBot | OpenAI | Finds pages for ChatGPT search results |
| ChatGPT-User | OpenAI | Fetches a page when a ChatGPT user asks for it |
| ClaudeBot | Anthropic | Collects pages that may be used to train models |
| Claude-User | Anthropic | Fetches a page when a Claude user asks for it |
| Claude-Web | Anthropic | Older Claude web access name |
| PerplexityBot | Perplexity | Indexes pages for Perplexity search |
| Perplexity-User | Perplexity | Fetches a page when a Perplexity user asks for it |
| Google-Extended | robots.txt control token for Gemini and Vertex AI use | |
| Google-CloudVertexBot | Crawls sites for Vertex AI agents at a site owner's request | |
| Meta-ExternalAgent | Meta | Collects pages for Meta AI products |
| meta-externalfetcher | Meta | Fetches a page when a Meta AI user asks for it |
| FacebookBot | Meta | Crawls pages that may feed Meta AI models |
| Amazonbot | Amazon | Crawls pages for Amazon services including Alexa answers |
| Applebot-Extended | Apple | robots.txt control token for Apple AI training use |
| Bytespider | ByteDance | Crawls pages for ByteDance AI products |
| CCBot | Common Crawl | Builds an open web archive widely used for AI training |
| cohere-ai | Cohere | Fetches pages for Cohere models |
| MistralAI-User | Mistral | Fetches a page when a Le Chat user asks for it |
| DuckAssistBot | DuckDuckGo | Fetches pages to produce DuckAssist answers |
| YouBot | You.com | Crawls pages for You.com AI search |
Descriptions are general summaries. Check each vendor's own documentation before writing rules.
Why behaviour matters too
- Names can be copied by anyone
- Many agents give no name at all
- Behavioural signs catch unnamed agents
- Trails show what each one did
Follow the trail on your own site
Paste one snippet and see the synthetic visitors that use your site. Card details are entered on Stripe, never on this site.
More from SynthTrail
- Understand how RAG crawlers retrieve your content
- Hidden content and rendering audits for AI crawlers
- What to look for in AI agent tracking software
- Bot management versus synthetic traffic auditing
- Server-side versus client-side synthetic visitor verification
- AI agent access audit checklist for webmasters and CTOs
- All guides