Generate Robots.txt: How to Create, Allow, Disallow, and Protect Your Crawl Budget
Short answer: to generate robots.txt, you write a plain-text file named robots.txt that lists a User-agent line and the Allow and Disallow paths for that crawler, add your Sitemap URL, and upload it to the root of your domain (for example https://example.com/robots.txt). You can type it by hand or use a robots.txt generator, but you must read the result line by line, because one wrong slash can block your entire site from crawling.
People often search for “generate robot txt” when they only need a quick file. A generator saves typing, but it cannot know which parts of your site waste crawler time and which parts you need crawled. This guide gives you that missing decision framework: what you can allow, what you can disallow, what each choice actually does, which URLs most websites should keep crawlers away from, and how to avoid the mistakes that quietly damage search visibility and ad revenue.
Key Points
- robots.txt controls crawling (whether a bot may fetch a URL). It does not reliably control indexing (whether a URL may appear in search results).
- It is public and voluntary. Never use it to hide private data, and expect only well-behaved crawlers to obey it.
- Block low-value URL spaces (internal search, carts, filters, tracking parameters, admin areas) and never block the CSS, JavaScript, images and pages that search engines need to understand your site.
- Disallow: / on a live site blocks all compliant crawling. It is the most expensive single-character mistake in technical SEO.
- AdSense publishers must not block the Mediapartners-Google crawler on pages that show ads, and a User-agent: * group does not control that crawler.
- Google ignores crawl-delay and the unofficial noindex rule in robots.txt. Test any file before and after publishing.
KEY TAKEAWAY:“robots.txt is a traffic sign for polite crawlers, not a lock on the door. Use it to guide crawling, and use real access controls to protect anything private.”
What Is a Robots.txt File?
A robots.txt file is a plain-text file at the root of a website that tells automated crawlers which URL paths they may or may not request. It follows the Robots Exclusion Protocol, which was standardised as RFC 9309 in 2022, and Google documents how its own crawlers interpret it.
Three facts about the file shape everything else in this guide:
- It works per host, protocol and port. A file at https://www.example.com/robots.txt covers only that exact host. It does not cover example.com, blog.example.com or http://www.example.com. Every subdomain needs its own file.
- It must sit at the root. A file placed at /folder/robots.txt is ignored. The file name is case-sensitive, so use lowercase robots.txt.
- It is a request, not enforcement. Major search engines respect it, but malicious or careless bots can ignore it. Because anyone can read your robots.txt, listing a secret folder in it actually advertises that folder.
What robots.txt Is Not
How to Generate Robots.txt, Step by Step
You can generate robots.txt manually in a text editor, with a CMS setting, or with an online robots.txt generator. The process is the same whichever method you pick.
Step 1: List the URL spaces on your site
Open your site and write down the groups of URLs it produces: content pages, category pages, internal search results, login and account pages, cart and checkout, admin screens, filters and sort options, tracking parameters, scripts, styles and media. This inventory matters more than any tool, because a generator can only format the rules you decide on.
Step 2: Decide what must stay crawlable
Anything you want to rank, plus everything needed to render those pages properly, must stay crawlable. That includes CSS, JavaScript and image files. Google needs them to see pages as users do, and blocking them can distort how your pages are evaluated.
Step 3: Choose what to disallow
From your inventory, pick the URL spaces that are low-value, duplicate, endless or private. The next sections give a tested checklist.
Step 4: Write the groups
Start with User-agent: * for rules that apply to all crawlers. Add named groups only when a specific crawler needs different treatment. Remember that a crawler follows only the single most specific group that matches it and ignores the rest.
Step 5: Add your sitemap
Add one or more absolute Sitemap: lines. Sitemap lines are not tied to a user-agent group.
Step 6: Save, upload and test
Save the file as UTF-8 plain text named robots.txt, upload it to the domain root, open it in a browser, and test sample URLs. Google may cache the file for up to about a day, so allow time before expecting changes.
What to Check in Any Robots.txt Generator
Whether you search for a “robots.txt generator”, a “robots .txt generator” or a “robot txt generator google” tool, check these points before you copy its output:
A note on the word “Google” in tool names. Google publishes documentation for the format and a robots.txt report in Search Console, but I could not confirm an official Google tool that generates the file for you. Pages titled “robots txt google generator” are usually third-party tools, so review their output the same way you would review any other.
Robots.txt Syntax Reference
How Google Picks Between Rules
The longest (most specific) matching rule wins.
- If an Allow and a Disallow match with equal specificity, Google applies the less restrictive rule.
- A crawler uses only the most specific user-agent group that matches it. Groups are not combined, so a Googlebot group fully replaces the * group for Googlebot.
- Invalid lines are ignored.
- Google enforces a 500 KiB limit and ignores anything past it.
Worked Pattern Examples (Including the Gotchas)
KEY TAKEAWAY:“Most robots.txt disasters are not syntax errors. They are pattern surprises: a prefix that catches too much, or a parameter that sits in a position your rule never matched.”
What You Can Allow in Robots.txt
Everything is crawlable by default, so Allow is mainly used to carve exceptions out of a blocked area, or to be explicit for a specific crawler.
What allowing does not do: an Allow rule does not force a search engine to crawl, index or rank a URL. It only removes a crawling restriction.
What You Can Disallow in Robots.txt
What Happens When You Disallow Something
Compliant crawlers stop requesting those URLs. This saves crawl requests and server load.
- The URL can still appear in search. If another page links to a blocked URL, a search engine may list it without a description. Blocking does not remove it.
- A noindex tag on a blocked page is never seen. The crawler cannot read the page, so it cannot see the instruction.
- Content on blocked pages is not read. Any links on blocked pages are not discovered through those pages.
- Blocked resources can limit rendering. If you block CSS or JavaScript, Google may not render pages as intended.
What Happens When You Allow Something
- Crawling becomes possible. The URL may be fetched, and then indexed or not depending on quality, duplication and meta directives.
- Crawler load can rise. Allowing huge parameter spaces can waste requests and server resources.
- Your pages can be evaluated fully. Allowing rendering resources helps search engines understand layout and content.
What to Disallow on Every Website to Protect Crawl Budget
Before the list, a reality check. According to Google’s crawl budget documentation, crawl budget matters mostly for very large sites (on the order of a million or more pages) or sites with many pages that change very frequently, and for sites with large numbers of discovered but unindexed URLs. If you run a small blog or a modest tool site, a clean robots.txt is good hygiene, but it will not transform your rankings. Also, blocking URLs does not automatically push crawl budget to other pages unless your server or Google’s limits were the bottleneck.
With that in mind, these are the URL types most websites should consider blocking:
What Not to Disallow
Robots.txt vs Noindex vs Canonical vs Password Protection
Choosing the wrong tool is more common than writing the wrong syntax. Use this table before you add a rule.
Ready-to-Adapt Robots.txt Templates
These templates show structure. Replace paths and the sitemap URL with ones that match your own site, and test them before publishing.
Minimal template for a small site
User-agent: *
Disallow:
Sitemap: https://example.com/sitemap.xmlAn empty Disallow: blocks nothing. This is a valid “allow everything” file that also advertises your sitemap.
Template for a typical WordPress site
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/
Sitemap: https://example.com/wp-sitemap.xmlThe Allow line keeps the AJAX endpoint available, because some themes and plugins use it on the front end. Check your own site’s sitemap location before copying the sitemap line.
Template for a store with filters and carts
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /*?sort=
Disallow: /*&sort=
Disallow: /*?filter=
Disallow: /*&filter=
Disallow: /*?sessionid=
Sitemap: https://example.com/sitemap.xmlReview your category structure first. If some filtered pages are intentionally optimised to rank, do not block their parameters.
Template for a tools or utility website
User-agent: *
Disallow: /api/
Disallow: /*?session=
Disallow: /*?result=
Disallow: /thank-you/
Sitemap: https://example.com/sitemap.xmlTool sites often generate result URLs or temporary download links. Block only those that create endless thin URLs, and keep your tool pages themselves crawlable.
Template for AdSense publishers who restrict some areas
User-agent: *
Disallow: /search
Disallow: /admin/
User-agent: Mediapartners-Google
Disallow:
Sitemap: https://example.com/sitemap.xmlThe separate Mediapartners-Google group is explicit, so ad crawling is not affected by choices made for other bots.
Custom Robots.txt for Blogger
Blogger users often search for a “custom robots.txt generator for Blogger”. Blogger gives you a default file and, in the blog settings under crawlers and indexing, lets you enable a custom robots.txt. Menu names can change, so check the current Blogger settings page.
A typical Blogger robots.txt looks like this:
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Allow: /
Sitemap: https://yourblog.blogspot.com/sitemap.xmlWhat each part does:
Practical Blogger tips:
Do not paste a robots.txt from a different platform. WordPress paths such as /wp-admin/ do not exist on Blogger.
- If you use a custom domain, use that domain in the sitemap line.
- Blocking /search also blocks label pages that live under that path, so check that you are happy with that before saving.
- After saving, open /robots.txt on your blog in a browser to confirm the live file matches what you entered.
Controlling AI and Other Special Crawlers
robots.txt is also the standard way to state your policy for AI-related crawlers. The right choice depends on your goals, because blocking can reduce how your content is used in AI products, while allowing may increase visibility in them. This table lists commonly referenced tokens. Vendors change names and behaviour, so confirm each against its owner’s own documentation before relying on it.
Example that blocks one training crawler only:
User-agent: GPTBot
Disallow: /Important limits: these rules are requests, not guarantees, and they only affect crawlers that choose to honour them. Decide deliberately, document your policy, and review it when vendors update their crawlers.
Robots.txt and Google AdSense
If your site shows AdSense ads, robots.txt affects revenue and policy compliance in specific ways.
- The AdSense crawler is separate. Google’s AdSense help page says the AdSense crawler (Mediapartners-Google) honours robots.txt, and that pages blocked only through a User-agent: * group may still be visited by the AdSense crawler. To block it, you must name it explicitly. The same help page also names Google-Display-Ads-Bot.
- Blocking it harms ads. If the crawler cannot read a page, Google may not be able to serve relevant ads there, and AdSense help pages describe this as a crawler-access problem to fix by removing the blocking lines.
- Do not hide ad pages from the ad crawler while showing them to users. Showing different content to a crawler than to visitors can be treated as deceptive.
- Do not place ads on pages you block from users or that need login. Ad-serving pages should have real, accessible content.
- A correct robots.txt is not a content-quality fix. It will not make thin pages valuable. Google’s policies still require original, useful content.
A clean file for AdSense sites usually means: allow the ad crawler, allow your content and rendering resources, and block only genuinely low-value or private URL spaces.
How to Test and Monitor Your Robots.txt
- View the live file. Open https://yourdomain.com/robots.txt in a browser and confirm it matches what you meant to publish. Check each subdomain separately.
- Use Google Search Console. The robots.txt report shows the files Google found and any parse problems, and URL Inspection tells you whether a specific URL is blocked by robots.txt.
- Test example URLs. Test URLs that should be blocked, URLs that must stay open (key content, CSS, JS) and the tricky ones from the pattern table above.
- Check server logs. Logs show what crawlers actually requested, which a robots.txt file alone cannot reveal.
- Re-test after every deployment. CMS updates, plugins and migrations can overwrite the file.
How Server Responses Affect robots.txt
Google may also cache the file for up to about a day, so a change might not take effect immediately.
Common Robots.txt Mistakes and How to Fix Them
Pre-Publish Checklist
- ☐ The file is named robots.txt and sits at the root of the correct host.
- ☐ There is no accidental Disallow: /.
- ☐ CSS, JavaScript and key images are crawlable.
- ☐ Search, cart, account, admin and parameter spaces are handled intentionally.
- ☐ Pattern examples (prefix, & variants, $, case) have been tested.
- ☐ The sitemap line is absolute and correct.
- ☐ Mediapartners-Google is not blocked on pages that show ads.
- ☐ No private information is listed in the file.
- ☐ Noindex pages are crawlable so the directive can be read.
- ☐ The file is UTF-8 and well under 500 KiB.
- ☐ You have a reminder to re-test after launches and migrations.
KEY TAKEAWAY:“A good robots.txt is short, deliberate and tested. Every line should answer one question: why should a crawler not fetch this?”
How This Guide Was Prepared
This article was written from Google’s published documentation on how it interprets the robots.txt specification, Google’s AdSense help page on the AdSense crawler, and the Robots Exclusion Protocol standard (RFC 9309). It does not describe private experiments, and it does not claim measured ranking results. Crawler names, defaults and product behaviour change, so check the linked sources before acting on anything for a large or revenue-critical site.
Frequently Asked Questions
How do I generate robots.txt for my website?
Write a plain-text file with User-agent, Allow and Disallow lines plus your Sitemap URL, save it as UTF-8 named robots.txt, and upload it to your domain root. You can type it manually or use a robots.txt generator, but review the output and test sample URLs before publishing.
Is there an official Google robots.txt generator?
Google documents the file format and offers a robots.txt report in Search Console, but I could not confirm an official generator. Tools with “Google” in their names, such as a “robot txt generator google” page, are normally third-party tools, so check what they produce.
Where should the robots.txt file be placed?
At the root of each host, for example https://example.com/robots.txt. A file inside a folder is ignored, and subdomains and different protocols need their own files.
What does Disallow: / do?
It tells compliant crawlers not to crawl any URL on that host. On a live website this normally blocks all crawling, which is why it must only appear on staging or test sites.
What does an empty Disallow: line mean?
It blocks nothing. A group with an empty Disallow: is effectively an explicit “allow everything” instruction for that crawler.
What should I disallow in robots.txt?
Consider low-value or private URL spaces: internal search results, cart and checkout, login and account pages, admin areas, filter and sort parameters, session and tracking parameters, infinite calendars and staging hosts. Keep CSS, JavaScript, images and important content crawlable.
What should I never disallow?
Do not block pages you want to rank, your sitemap, rendering resources such as CSS and JavaScript, pages whose noindex or canonical tags Google must read, and the Mediapartners-Google crawler on pages that show AdSense ads.
Does blocking a page in robots.txt remove it from Google?
No. A blocked URL can still appear in results, sometimes without a description, if other pages link to it. To keep a page out of results, allow crawling and use noindex, or remove the page.
Can I use noindex inside robots.txt?
Not with Google. Google stopped supporting the unofficial noindex rule in robots.txt in 2019. Use a meta robots tag or an X-Robots-Tag header instead.
Does Google support the crawl-delay directive?
No. Google’s documentation lists user-agent, allow, disallow and sitemap as supported fields and says other fields such as crawl-delay are not supported. Some other crawlers may honour it.
What happens if my site has no robots.txt file?
If the file returns a 404 (or most other 4xx errors), Google treats it as having no restrictions and crawls normally. A missing file is not an error, though a short explicit file helps you state your sitemap and rules.
What happens if robots.txt returns a server error?
Google treats persistent server errors cautiously. According to published summaries of its documentation, it stops crawling for a period while it retries, so fix 5xx errors on the file quickly.
How long does it take for robots.txt changes to work?
Google may cache the file for up to about a day, and crawling patterns take time to change, so allow time and then verify in Search Console and your server logs.
What is the maximum size of a robots.txt file?
Google enforces a limit of 500 KiB and ignores anything beyond it. Consolidate rules if your file grows large.
Does robots.txt improve crawl budget?
It can reduce wasted crawling of low-value URL spaces, which matters most on very large or fast-changing sites. For small sites, crawl budget is rarely a problem, and blocking URLs does not automatically give other pages extra crawling.
Do robots.txt rules apply to all crawlers equally?
No. Each crawler follows the most specific group that matches its name, and well-behaved bots honour the file while malicious bots may ignore it. Some special crawlers, such as AdSense’s Media partners-Google, are not controlled by the * group and must be named explicitly.
Will blocking AI crawlers hurt my Google rankings?
Google states that blocking Google-Extended does not affect inclusion or ranking in Google Search, because it is separate from Googlebot. Other vendors have their own crawlers and policies, so check each one’s documentation.
Can robots.txt block images, PDFs and other files?
Yes, with path patterns such as Disallow: /*.pdf$, for compliant crawlers. Blocked files can still be listed by URL if they are linked from elsewhere, and parameters after the extension can bypass an end-anchored pattern.
How do I create a custom robots.txt for Blogger?
Enable the custom robots.txt option in your Blogger settings for crawlers and indexing, then paste a file that keeps the AdSense crawler allowed, blocks /search, allows other pages and lists your blog’s sitemap. Open /robots.txt on your blog afterwards to confirm the live content.
Should AdSense sites do anything special in robots.txt?
Keep Media partners-Google allowed on pages that show ads, do not block the content pages you want evaluated, and block only low-value or private areas. A tidy robots.txt does not replace the need for original, useful content under AdSense policies.
Conclusion
To generate robots.txt well, decide first and format second. Map your URL spaces, keep every page and resource that search engines need open, block only the endless, duplicate or private areas, add your sitemap, and test the result with real URLs. Remember the three limits: robots.txt controls crawling rather than indexing, it protects nothing private, and it only guides crawlers that choose to follow it. For AdSense sites, keep the ad crawler allowed and pair a clean robots.txt with genuinely useful content.
