Robots.txt Generator
Build a robots.txt file with crawl rules for search engines and bots.
A robots.txt file tells search engine crawlers which parts of your site they're allowed or not allowed to crawl - placed at the root of your domain (yoursite.com/robots.txt). This tool lets you build one visually with user-agent rules, rather than writing the directive syntax by hand.
Frequently asked questions
Not exactly - it tells well-behaved crawlers not to CRAWL certain paths, but a page can still occasionally appear in search results without being crawled (e.g. if linked elsewhere). For guaranteed exclusion from search results, use a noindex meta tag or HTTP header on the specific page.
At the root of your domain, so it's accessible at yoursite.com/robots.txt - search engines specifically look for it at that exact location.
Yes - robots.txt supports rules per user-agent, so you can set different crawl permissions for Googlebot versus other crawlers if needed.