About RouteReadyBot
RouteReadyBot is the web crawler RouteReady uses to read public bid postings and school committee documents on Massachusetts town, city and school district websites. RouteReady helps school transportation companies find student transportation bids, and every posting we show links back to the official page it came from.
How to recognize it
Its user agent starts with RouteReadyBot/1.0 and includes a link to this page, for example:
RouteReadyBot/1.0 (+https://www.routereadyma.com/about-routeready-bot)It runs on cloud hosting, so its network addresses change. Identify it by its user agent.
What it reads
- Public bid and procurement listings, and the solicitations they link to.
- Public school committee agendas, minutes and meeting calendars, and the documents they link to.
- Nothing behind a login. It doesn't sign in, create accounts, or submit forms that change anything on your site.
How it behaves
- It reads your robots.txt before anything else and follows it, including rules addressed to RouteReadyBot and any Crawl-delay.
- If your robots.txt can't be read because of a server error, it treats your site as off-limits until its next run.
- It waits at least two seconds between requests to the same site, or longer if your Crawl-delay asks for it. When a delay is too long to wait out, it makes one request per run.
- Most pages are checked about once a day, and some once a week.
- If your site refuses a request, for example with HTTP 403, it doesn't retry it or try to get around the block.
How to limit or block it
Add rules for RouteReadyBot to your robots.txt. To keep it off your whole site:
User-agent: RouteReadyBot
Disallow: /To slow it down instead (seconds between requests):
User-agent: RouteReadyBot
Crawl-delay: 10It picks up changes the next time it reads your robots.txt, usually within a day.
Questions or requests
Email support@routereadyma.com with your site's address. We can stop visiting, visit less often, or limit which pages we read.