What robots.txt does
The robots.txt file sits at the root of your site and tells crawlers which paths they may fetch. It can also list your XML sitemaps. A single wrong line can hide your whole site from search, so it is worth testing.
What this tool does
- Fetches your robots.txt and breaks it into groups by crawler.
- Tests whether a path you enter is allowed for Googlebot, Bingbot, or AI crawlers such as GPTBot.
- Finds your sitemaps and reports how many addresses each one lists.
How the rules work
The most specific matching rule wins, meaning the longest path. If an Allow and a Disallow match equally, Allow wins. The * character matches any characters and $ matches the end of the address.
Common mistakes
Disallow: /underUser-agent: *blocks the entire site.- Blocking CSS or JavaScript files stops search engines rendering your pages.
- Using robots.txt to hide private pages. It only asks crawlers to stay away. Use a password or noindex instead.
About sitemaps
A sitemap lists the pages you want indexed. Large sites use a sitemap index that points to several smaller sitemaps. Include lastmod dates so crawlers can focus on what changed.