Crawler
SearchVoroBot
SearchVoroBot is the crawler behind SearchVoro's site audits. It visits a site only when a SearchVoro customer asks it to, and it tells you who it is.
User-Agents
What appears in your logs
SearchVoro sends one of three User-Agent strings, depending on what the request is for.
| SearchVoroBot | SearchVoroBot/1.0 (+https://searchvoro.com/bot)Fetches pages to audit them: on-page checks, technical SEO, AI citability, internal links, sitemaps and robots.txt. The same token is used for the public Reddit and Hacker News search APIs behind community monitoring. |
|---|---|
| SearchVoro-CMS | SearchVoro-CMS/1.0 (+https://searchvoro.com/bot)Talks to a WordPress or Shopify site only after its owner connects it with their own credentials, to read content and apply changes they have approved. |
| SearchVoro-Webhook | SearchVoro-Webhook/1.0 (+https://searchvoro.com/bot)Delivers event notifications to webhook endpoints that SearchVoro has been asked to notify. |
Behaviour
How it crawls
| Which sites | SearchVoroBot only visits a site because a SearchVoro customer asked it to: a site they added to their account, or a specific page they submitted for analysis. New sites have to be verified first, with a DNS TXT record or a meta tag that shows the customer controls the domain. It does not crawl the open web or build a search index. |
|---|---|
| robots.txt | Before it fetches pages from a site, SearchVoroBot reads that site's robots.txt (cached for 30 minutes) and skips any path disallowed for the SearchVoroBot token or for *. If robots.txt cannot be fetched, it is treated as having no rules. One request does not check robots.txt today: the single homepage fetch that starts an on-page audit of a site the customer has added. |
| Crawl rate | Each analysis fetches a small, bounded set of pages rather than spidering a whole site, and results are cached for several minutes so repeat views don't trigger new fetches. If your robots.txt sets a Crawl-delay, requests to your host are queued and spaced by that interval (up to 10 seconds). |
| Identification | Requests send an honest User-Agent and a From header of [email protected]. SearchVoroBot does not present itself as a browser to get past bot protection; if your site blocks it, the customer sees that the site blocks crawlers. |
| Network | Requests go only to public addresses. Each connection is checked against private and internal ranges, including after every redirect. |
Control
Blocking or slowing it down
To keep SearchVoroBot out of all or part of your site, add a rule to robots.txt:
User-agent: SearchVoroBot
Disallow: /To slow it down instead, add Crawl-delay: 5 (seconds) under the same User-agent line.
If SearchVoroBot is causing a problem on your site, email [email protected] with your domain and, if you can, a sample log line. For anything else, write to [email protected].