BitMoversBot

It reads public web pages to learn how websites link to each other, what they are built with, and what they are about, and keeps a searchable index of that text. It mostly reads home pages plus a few pages that usually describe a business, such as contact and about pages. It does not fill in forms, log in, run scripts, buy anything or try to get past anything a site protects.

• It reads and obeys robots.txt, including Crawl-delay, and checks it again at least once a day.
• It asks for no more than one page per second from any one site, and backs off when a site says it is busy.
• It identifies itself in every request with a user agent containing BitMoversBot and a link to this page.
• Its total speed is capped, so it never sends a site more than a trickle of requests.
• The link in its user agent ends in a short reference for your site, so if you visit it we can see which site's logs it came from. We record only that reference, the time and your browser's user agent: no IP address and no cookies.

Add this to your robots.txt and it will stop reading your site within a day:

User-agent: BitMoversBot
Disallow: /

Or, to slow it down rather than block it:

User-agent: BitMoversBot
Crawl-delay: 10

The text of the pages it reads, page titles and descriptions, links between sites, the technology a site uses, and business details a site publishes about itself, such as a company name, address, phone number or email address. It does not collect anything behind a login, and it does not sell or publish what it collects.

To have your site or your details removed, or to ask anything about the crawler, email [email protected] with the site address. We remove it and stop crawling it, normally within two working days. Under UK GDPR you can also object to us holding contact details about you; the privacy notice explains your rights.