Generador de Robots.txt
Build a robots.txt file with crawl rules for search engines and bots.
Acerca de esta herramienta
robots.txt controls crawling, not indexing, and confusing the two is the most consequential mistake in technical SEO. Disallowing a path tells a crawler not to fetch it - but if other sites link to that page, Google can still list the URL in results, showing it without a description because it was never allowed to read the content. Worse, a page you have blocked cannot be read at all, so a noindex tag on it is never seen and never obeyed. The correct pairing is therefore counter-intuitive: to keep a page out of search results, allow crawling and use noindex; to save crawl budget on pages you do not care about, disallow. The file lives at the root of the domain, is publicly readable by anyone, and is honoured voluntarily - well-behaved crawlers respect it while malicious scrapers ignore it entirely. It is a crawling instruction, never a security measure. This builds the syntax in your browser.
Cómo usar esta herramienta
- Choose the user-agent* applies to every crawler. Name a specific one - Googlebot, Bingbot - only when it needs different rules.
- Add your disallow rulesPaths you do not want crawled: admin areas, internal search results, duplicate parameter URLs.
- Add your sitemap URLThe absolute URL of your sitemap.xml. This is how crawlers discover your pages efficiently.
- Upload to your domain rootIt must be reachable at yoursite.com/robots.txt exactly - no other location is checked.
Por qué usarla
- Generates correct directive syntax, where a typo silently changes behaviour rather than erroring.
- Supports per-crawler rules for the cases where different bots need different treatment.
- Includes the sitemap declaration, which is the most useful line in most robots.txt files.
- Runs in your browser with no account.
- Output is plain text you can paste directly into your site root.
Usos comunes
- Keeping crawlers out of admin and login paths.
- Preventing internal search result pages from consuming crawl budget.
- Declaring your sitemap location so crawlers find your pages.
- Blocking parameter URLs that duplicate existing content.
- Setting different rules for a specific bot that is over-crawling.
Consejos para mejores resultados
- Always include your sitemap URL. It is the single most valuable line in the file.
- To keep a page out of search results, do NOT disallow it - allow crawling and add a noindex tag, or the crawler can never see the noindex.
- Test in Google Search Console's robots.txt tester before deploying. A misplaced slash can block your entire site.
- Remember the file is public. Listing "/secret-admin-panel/" advertises exactly where it is.
- Disallow: / blocks everything. Check for it carefully - it is the single most damaging line a site can accidentally ship.
Errores a evitar
- Disallowing a page to keep it out of search results, which prevents the crawler seeing the noindex tag that would actually do it.
- Shipping "Disallow: /" from a staging environment to production, which deindexes the entire site.
- Blocking CSS and JavaScript, which stops Google rendering the page as users see it and can hurt rankings.
- Treating robots.txt as security. It is public, advisory, and ignored by anything malicious.
- Forgetting the sitemap line, making page discovery slower than it needs to be.
- Placing the file anywhere other than the domain root, where nothing will look for it.
Preguntas frecuentes
Not exactly - it tells well-behaved crawlers not to CRAWL certain paths, but a page can still occasionally appear in search results without being crawled (e.g. if linked elsewhere). For guaranteed exclusion from search results, use a noindex meta tag or HTTP header on the specific page.
At the root of your domain, so it's accessible at yoursite.com/robots.txt - search engines specifically look for it at that exact location.
Yes. Rules are grouped per user-agent, so Googlebot can have different permissions from Bingbot or a specific scraper. Use * for the default that applies to everything else.
Allow crawling and add a noindex meta tag or HTTP header. This is the opposite of what people expect: if you disallow the page, the crawler never fetches it, never sees the noindex, and the URL can still appear in results from external links.
No, and treating it as one is dangerous. The file is publicly readable, so listing a private path advertises its location. Compliance is voluntary - malicious scrapers ignore it entirely. Use authentication for anything that genuinely needs protecting.
No. Google renders pages to understand them, and blocking stylesheets or scripts means it sees a broken version of your site. This was once common advice and is now actively harmful.
Yes. Robots Txt Generator is free for normal use with no account required, and ToolBox does not add a watermark to your result.
Most tools in this category run in your browser so the file stays on your device. If a tool needs a temporary server job, files are handled for that job only and are not kept as a lasting archive.
La gente también busca
- robots txt generator free
- robots txt example
- block google from crawling page
- robots txt vs noindex
- disallow rules seo
- robots txt tester
- how to add sitemap to robots txt