AI Visibility
ClaudeBot
Anthropic's web crawler. Like the other major AI crawlers, it fetches raw HTML and does not run JavaScript.
By Shimon Carroll, Founder, SEO for AI Agents · Last updated
ClaudeBot is Anthropic's crawler for gathering web content. Anthropic also operates Claude-User (for user-initiated fetches inside Claude) and Claude-SearchBot (for search-grounded retrieval), each identifiable by user agent. Anthropic publishes the user-agent strings and supports robots.txt directives, so a site can allow or disallow these agents granularly.
As with GPTBot and PerplexityBot, ClaudeBot reads the raw HTML response and does not execute client-side JavaScript. A page whose main content only appears after hydration is, to ClaudeBot, blank. This is why the render-parity diff is the highest-leverage technical check for AI visibility: it measures the gap between what a browser shows and what a non-JS crawler receives.
Allowing ClaudeBot lets Anthropic's systems read and, where applicable, cite your content. Blocking it removes you from that surface. Anthropic has been public about respecting robots.txt and about its crawling practices, so the controls behave predictably. The practical recommendation for a brand that wants to be recommended by Claude is to allow the agents, make sure the content is in the raw HTML, and corroborate the brand's key facts across the sources Claude trusts.