Bot policy

ApexAuditBot is the crawler for SEO for AI Agents, the SEO auditing product operated by Own The Climb. When someone runs an audit, our crawler fetches the requested public web pages so we can analyze their SEO. This page exists because our crawler identifies itself with a contact URL that points here, so any site owner can recognize our traffic and decide how to treat it. We describe our broader data and security practices on our security page.

Every request our crawler makes sends this exact User-Agent string:

ApexAuditBot/0.1 (+https://seoforaiagents.com/bot)

The URL inside the User-Agent points back to this page. We do not impersonate other crawlers: we never claim to be Googlebot, GPTBot, or any agent other than ApexAuditBot.

• We honor robots.txt.Before fetching a URL, our crawler reads the host’s robots.txt and evaluates it against our User-Agent. If a rule disallows the path for us, we do not fetch it. We do not crawl content behind authentication.
• We honor crawl-delay. If the matching robots.txt group sets a crawl-delay, we slow down to at least that spacing. Crawl-delay can only make us slower, never faster.
• We rate-limit per host. By default we make at most one request per second to a given host, and we never exceed five requests per second under any circumstances, so we do not degrade the sites we audit.
• We render lightly by default.Most pages are fetched as plain HTML with no JavaScript execution. We escalate to a full headless-browser render only when a page’s server HTML and its rendered output diverge enough to need it, or when an audit explicitly requests JavaScript rendering.
• We fetch a bounded amount. An audit fetches the requested URL and a limited set of pages linked from it, within a configured depth. We cap each response we read and time out slow requests, so a single audit is a small, polite amount of traffic.

You do not need to do anything to allow us: if your robots.txt has no rule that disallows us, our crawler is permitted, exactly like any well-behaved bot. If you would rather we did not crawl your site, add a group for our User-Agent token to your robots.txt. The token to match is ApexAuditBot (matching is case-insensitive):

User-agent: ApexAuditBot
Disallow: /

To keep us crawling but slow us down, set a crawl-delay in seconds instead:

User-agent: ApexAuditBot
Crawl-delay: 5

Changes to your robots.txt take effect the next time we read it. If you need us to stop immediately or have any question about our traffic, email support@seoforaiagents.com and we will respond promptly.