Free tool
Robots.txt Generator & Validator
Build a clean robots.txt with the right allow, disallow and sitemap rules, or paste your own to check it for mistakes. No sign-up needed.
One full URL per line. Added as Sitemap: lines at the end.
Your robots.txt
Browsers cannot fetch other domains, so to check a live site open yoursite.co.uk/robots.txt, copy what you see, and paste it here.
How to use it
Use the Generate tab to build a robots.txt from scratch. Set a user-agent (use an asterisk to mean all crawlers), then add the paths you want to block under Disallow and any you want to keep open under Allow. Add a rule group for each crawler you want to treat differently, and list your sitemap URLs at the bottom. The preview updates as you type, so you can copy it or download a ready-made file.
Use the Validate tab to check a robots.txt for common mistakes. It flags rules with no user-agent above them, unknown directives, sitemap lines that are not full URLs, groups with no rules, and lines that are missing a colon. A clean file gets a green all-clear.
To check a live site, open its robots.txt at yoursite.co.uk/robots.txt, copy what you see and paste it into the Validate box. This all runs in your browser, which for security reasons cannot fetch files from other domains directly, so the paste step is needed.
A small but important point: blocking a page in robots.txt stops crawlers reading it, but it does not guarantee the URL stays out of search results. To keep a page out of results, leave it crawlable and use a noindex meta tag instead.
Frequently asked questions
What does robots.txt do?
A robots.txt file tells search engine crawlers which parts of your site they can and cannot request. It lives at the root of your domain, for example yoursite.co.uk/robots.txt.
Does robots.txt stop a page from being indexed?
No. Blocking a page in robots.txt stops crawlers reading it, but the URL can still appear in results if it is linked elsewhere. To keep a page out of results, use a noindex meta tag instead and leave it crawlable.
Should I link my sitemap in robots.txt?
Yes. Adding a Sitemap line with the full URL of your sitemap helps search engines find all your pages. You can list more than one sitemap if you have them.
Keep going
Ready to get found by more local customers?
Get a free quote, or grade your site in under a minute.