Our website crawler
Last updated:
What it reads
Gaazzeebo’s free search visibility audit reads a few public pages of websites. Its product token is gaazzeeboaudit.
- The website being audited, at its owner’s or an authorised person’s request: the homepage, robots.txt, the sitemap and up to 8 key pages, read with a browser-compatible User-Agent that ends in
gaazzeeboaudit/1.0. Key pages that robots.txt closes to our token are not read. - For comparison, the websites linked from the Google listings of businesses shown beside it: robots.txt first, then, only if it allows us, the homepage, the sitemap and up to 2 more public pages. These requests identify themselves as
gaazzeeboaudit/1.0 (+https://gaazzeebo.net/audit/crawler; audit@gaazzeebo.net). If robots.txt closes the site to our token or to all crawlers, or cannot be read, we read nothing else.
It does not log in, fill in forms, or try to get past a bot check on a comparison site, and it stops when a site turns it away.
What it keeps
For a comparison site we keep each page’s address, title, main heading and the date we read it, never the page’s content. A report quotes at most a page title, shortened to 80 characters.
How to stop it
Add these two lines to your robots.txt to stop our crawler reading your site:
User-agent: gaazzeeboaudit Disallow: /
Or email contact form with your domain and we will add it to our do-not-read list.