rankiboo-bot

rankiboo-bot is the web crawler of Rankiboo, a new search engine with no ads and no tracking. It reads public web pages so they can be found in Rankiboo search.

How it identifies itself

User-Agent: rankiboo-bot/0.1 (+https://rankiboo.com/bot)

How it behaves

It obeys robots.txt (RFC 9309), including Crawl-delay.
It asks for one page at a time per site, with at least 1 second between requests.
It obeys <meta name="robots" content="noindex">, nofollow and the X-Robots-Tag header.
It does not download pictures, videos or file downloads - only web pages.
It does not log in, fill in forms or run scripts.
If a site keeps refusing it, it stops visiting that site.

How to block or slow it

Add this to your site's robots.txt to block it completely:

User-agent: rankiboo-bot
Disallow: /

Or to slow it down (seconds between requests):

User-agent: rankiboo-bot
Crawl-delay: 10

To keep one page out of Rankiboo, add <meta name="rankiboo-bot" content="noindex"> (or the usual <meta name="robots" content="noindex">) to it.

Contact

Problems with the crawler? Email [email protected] with your site's address and we will look at it.