Rules for Google, Bing and 25+ bots.
Block GPTBot, CCBot and other AI bots.
Check syntax and test any URL.
No login or signup required.
Toolinium Robots.txt Generator creates a properly formatted robots.txt file for your website in seconds. A robots.txt file tells search engine crawlers which pages they are allowed or not allowed to access. It sits at the root of your website and is one of the first files Google checks when crawling your site. Use this tool to allow all crawlers, block specific bots, protect admin pages, or stop AI content scrapers like GPTBot from indexing your content.
A robots.txt file is a plain text file at the root of your website that tells search engine crawlers which URLs they may or may not request. It follows the Robots Exclusion Protocol (RFC 9309).
It is not required, because a missing file simply means everything can be crawled. But it is strongly recommended: it keeps crawlers out of admin and duplicate pages, saves crawl budget and points them to your sitemap.
In the root folder of your domain so it opens at yourdomain.com/robots.txt. A robots.txt inside a subfolder is ignored, and each subdomain needs its own file.
Pick a template in this generator, add any custom Disallow or Allow rules, enter your sitemap URL, then copy or download the result and upload it to your site root. Finally open yourdomain.com/robots.txt to confirm it loads.
The asterisk means all crawlers, and an empty Disallow means nothing is blocked. This is the standard allow-everything file.
Add a rule with the crawler name (or * for all) and the path in Disallow, for example Disallow: /private/. To block a single bot use its name, such as GPTBot, with Disallow: /.
No. Disallow stops crawling, but a blocked URL can still appear in results if other sites link to it. To keep a page out of search results, use a noindex meta tag on a crawlable page or password-protect it.
Disallow controls crawling, while noindex controls whether a page may appear in search results. Google must be able to crawl a page to see its noindex tag, so do not combine both on the same URL.
That is your choice. Blocking AI crawlers asks them not to use your content for training, and it does not affect Google rankings because these are not search ranking crawlers.
Crawl-delay asks a bot to wait a number of seconds between requests. Bing and Yandex honor it, but Google ignores it.
Yes. Add one Sitemap line per file, or paste several URLs separated by spaces or commas in the Sitemap field of this generator. Sitemap lines work independently of any User-agent group.
Yes. /Admin/ and /admin/ are treated as different paths, while directive names such as Disallow are not case-sensitive.
If the file returns a 404, crawlers assume they may crawl everything. If it returns a server error (5xx), Google can pause crawling until it loads again, so keep the file reliably available.
Use the Validator and URL Tester on this page to check syntax and test any URL for a chosen crawler. After uploading, open the robots.txt report in Google Search Console to check fetch errors.
Yes. It is a polite request that reputable crawlers follow, not a security feature. Protect private content with authentication or server rules.
* means every crawler; Googlebot means only Google's crawler.Disallow: /admin/ blocks that folder, and an empty Disallow: allows everything./wp-admin/admin-ajax.php while blocking /wp-admin/.* matches any characters and $ marks the end of a URL, for example Disallow: /*.pdf$.User-agent: * Allow: /wp-admin/admin-ajax.php Disallow: /wp-admin/ Disallow: /wp-login.php Disallow: /?s= Sitemap: https://yourwebsite.com/sitemap.xml
The WordPress template above generates this kind of file. It keeps crawlers out of the admin area and internal search pages while leaving your content, CSS and JavaScript crawlable.
/Admin/ and /admin/ are different.Toolinium Robots.txt Generator creates a properly formatted robots.txt file for your website in seconds.
Explore SEO & Marketing Tools ยท How Toolinium tests tools ยท Editorial policy ยท AI transparency
Updated October 29, 2026