Robots.txt & Crawl Budget: What to Allow, Block & Why
Robots.txt is a small text file with outsized SEO impact. Done well, it steers crawlers toward money pages and away from infinite filters. Done poorly, it blocks CSS, wastes crawl budget on junk URLs, or gives a false sense of security for private paths. This guide clarifies what to allow, what to block, and why — then points you to ToolMars generators that keep the file accurate.
Crawl budget matters most on large or parameter-heavy sites, but even modest properties benefit from intentional rules, a sitemap line, and zero accidental Disallows on public content.
Generate a clean robots.txt in seconds
Free robots generator — allow/disallow rules, AI bots, sitemap lines, local only.
Crawl budget in plain language
Search engines allocate limited fetch capacity per site. Every hit on a duplicate tag page, session URL, or redirect hop is a hit not spent on a fresh article or product. Robots.txt cannot "increase" budget magically — it reduces waste so existing capacity lands on URLs that deserve indexing.
- Allow — canonical content, blog posts, tools, key category hubs.
- Disallow — admin, internal search, tracking parameters, thin facets.
- Sitemap — declare XML sitemap URLs so discovery is not left to chance.
Disallow vs noindex (do not confuse them)
Disallow means "do not fetch." Noindex means "you may fetch, but please do not show in results." If you Disallow a URL, Google may be unable to see a noindex tag on that page. For thank-you pages or soft-private content you still need crawled to honor noindex, allow the path and set robots meta / Metadata API robots instead.
What to allow, block, and why
Practical ruleset
- Start from allow-all for Googlebot on public HTML, then add surgical Disallows.
- Block CMS admin, API debug routes, and cart/checkout internals if they create crawl noise.
- Keep URL paths tidy with a slug generator — fewer near-duplicate slugs means less budget waste.
- After migrations, collapse chains with the redirect compiler so bots hit finals in one hop.
- Preview indexable page snippets with the meta generator — crawl access is useless if titles are truncated junk.
- Generate and diff the file in the robots.txt generator before deploy.
User-agent: * Allow: / Disallow: /admin/ Disallow: /api/private/ Disallow: /*?*sort= Disallow: /search User-agent: GPTBot Disallow: / Sitemap: https://www.example.com/sitemap.xml
Stop crawl budget leaks from redirect chains
Compile clean redirect maps for Next.js and static hosts — fewer wasted hops.
AI crawlers and optional blocks
Separating search crawlers from training scrapers is increasingly common. You can Allow Googlebot while Disallowing specific AI bots. Document the business decision — blocking training crawlers does not replace copyright strategy, but it is a clear signal many bots respect.
Related Tools on ToolMars
- Robots Generator — allow/disallow rules and sitemaps
- Redirect Compiler — cut redirect chain waste
- Slug Generator — cleaner indexable paths
- Meta Generator — snippets for pages you allow to crawl
Conclusion
Treat robots.txt as traffic control for bots, not a vault. Allow what should rank, Disallow what burns crawl budget, use noindex when you need exclusion with a fetch, and keep redirects short. Generate the file deliberately — then verify in Search Console after deploy.
Generate a clean robots.txt in seconds
Free robots generator — allow/disallow rules, AI bots, sitemap lines, local only.
Frequently Asked Questions
What is crawl budget?
Crawl budget is roughly how many URLs a search engine is willing to fetch from your site in a given period. Large or messy sites waste it on duplicates, parameters, and low-value paths.
Does robots.txt hide pages from Google Search?
No. Disallow stops crawling, not necessarily indexing. URLs linked elsewhere can still appear without a snippet. Use noindex (and allow crawl) or authentication for true exclusion.
Should I block CSS and JavaScript in robots.txt?
Almost never for Googlebot. Blocking render assets can hurt indexing quality. Allow the resources needed to render key templates.
What should most sites allow by default?
Allow public content, sitemaps, and critical assets. Disallow admin, cart internals, faceted infinite URL spaces, staging clones, and sensitive APIs you do not want hammered.
How do redirects interact with crawl budget?
Long redirect chains burn crawl fetches. Compile clean 1:1 maps and retire dead paths so bots spend hits on final destinations — use a redirect compiler during migrations.
Where does robots.txt live?
At the site root: https://example.com/robots.txt. Subfolder robots files are ignored for host-wide rules.
Can I block AI crawlers separately from Googlebot?
Yes. Many AI user-agents honor Disallow rules (GPTBot, ClaudeBot, CCBot, etc.). Add explicit User-agent blocks while keeping Googlebot rules focused on search.