Tracking AI crawlers and auditing robots.txt with Vurto
Vurto tracks AI crawler visits on your site and audits your robots.txt, to check your pages are actually accessible to those crawlers.
Vurto tracks the passage of AI crawlers (GPTBot, ClaudeBot, PerplexityBot, and others) on your site and audits your robots.txt file, to verify that these bots can actually access the pages you want cited in generative answers.
Why this tracking matters
Content can be perfectly structured and relevant: if generative AI crawlers can't reach it, it will never be cited. The robots.txt file and controlling AI crawlers are the first technical gateway to your generative visibility: before working on a page's content or semantic scoring, you need to make sure it's accessible to the bots feeding ChatGPT, Claude, Gemini or Perplexity.
What Vurto shows about crawler activity
Vurto tracks which identified AI crawlers visit your site and how often, giving a concrete indication of the attention generative engines already pay to your pages. A site regularly visited by GPTBot, ClaudeBot and PerplexityBot is being actively explored as a potential source; a site where these bots rarely or never visit certain sections points to a blind spot to fix before anything else.
A signal to read over time, not as a snapshot
A single AI crawler visit on a given day doesn't tell you much on its own: what matters is how regularly it visits over several weeks. A crawler that used to visit a section of the site regularly and suddenly stops is a signal to prioritize, even with no apparent change on the site — it can point to an infrastructure issue (server response time, intermittent technical errors) that isn't necessarily tied to robots.txt. Conversely, a crawler that has never visited a section doesn't necessarily mean it's blocked: that section may simply never have been judged relevant enough yet by that engine to warrant deeper exploration.
The robots.txt audit
Vurto analyzes your robots.txt file to identify whether rules block, intentionally or by mistake, AI crawler access to all or part of your site. A blocking rule can be old, inherited from a generic configuration, and may never have specifically targeted AI bots — a frequent case, especially on sites whose robots.txt hasn't been reviewed since these new crawlers appeared. The audit pinpoints exactly which rules apply to which bots, rather than guessing from a raw text file.
Interpreting a lack of crawling or a block
Two distinct situations call for different responses. If robots.txt explicitly blocks an AI crawler, the fix is direct: adjust the rule to allow access to the relevant sections. If, on the other hand, nothing blocks the crawler but it simply isn't visiting, or rarely does, the issue isn't technical but editorial or structural — the site may not yet be considered a sufficiently relevant or up-to-date source in your sector, which ties back to the questions covered by analyzing your citation sources.
What to fix, and in what order
Always start with explicit blocks in robots.txt: these are the fastest, highest-impact fixes, since they remove a total access barrier. Once these blocks are lifted, watch for crawlers returning to the affected pages in the following analysis waves — a delay of a few days to a few weeks before seeing an effect on your visibility scores is normal. Build this check into your weekly tracking routine: a robots.txt audit is only needed once, but checking that nothing gets re-blocked after a technical site update is a control worth repeating regularly.
What a gap between site sections reveals
Crawler activity isn't necessarily uniform across a whole site. It's common for category pages and the homepage to be visited regularly, while deeper sections — individual product pages, FAQ pages, older blog content — get much less attention. Before concluding there's an access problem, check that these pages are actually linked from the rest of the site (internal linking, an up-to-date sitemap): a crawler, AI or not, only visits what it can reach by following links or reading the sitemap. A simple oversight in updating the sitemap after adding new product pages can explain a lack of crawling just as well as a robots.txt block.
A common scenario
A technical update to the site (CMS change, hosting migration, new security configuration) can reintroduce a block without any deliberate action — an application firewall or a generic anti-bot rule can treat AI crawlers as unwanted traffic. That's exactly why the robots.txt audit should stay a recurring check rather than a one-time verification done only at the start.