Home

๐Ÿค– Robots.txt Generator

Configuration
Block Googlebot
Block Bingbot
Block AI Crawlers (GPTBot, ClaudeBot, etc.)
Block Social Media Crawlers
Generated robots.txt
Click Generate to build your robots.txt file...
Robots.txt Validator & URL Tester
Paste any robots.txt, or test the one you just generated. Find syntax errors and check whether a URL is allowed or blocked for a specific crawler.

All Search Engines

Rules for Google, Bing and 25+ bots.

Block AI Crawlers

Block GPTBot, CCBot and other AI bots.

Validator & Tester

Check syntax and test any URL.

100% Free

No login or signup required.

About Robots.txt Generator

Toolinium Robots.txt Generator creates a properly formatted robots.txt file for your website in seconds. A robots.txt file tells search engine crawlers which pages they are allowed or not allowed to access. It sits at the root of your website and is one of the first files Google checks when crawling your site. Use this tool to allow all crawlers, block specific bots, protect admin pages, or stop AI content scrapers like GPTBot from indexing your content.

How to Create a robots.txt File (Step by Step)

  1. Pick a template. Choose Allow All, Block All or a platform template (WordPress, eCommerce, Shopify, Blogger, Joomla, Drupal, Next.js, Laravel).
  2. Choose which bots to block. Use the switches for Googlebot, Bingbot, AI crawlers and social crawlers. Red warnings tell you when a setting can hurt your SEO.
  3. Add custom rules. Enter a user-agent and the folders or URLs to Disallow, plus an optional Allow exception.
  4. Add your sitemap. Enter the full sitemap URL (for example https://yourwebsite.com/sitemap.xml). Optionally set a Crawl-delay for Bing and Yandex.
  5. Test it. In the Validator & URL Tester, check for syntax errors and try your important URLs for Googlebot to make sure they are not blocked.
  6. Copy or download, then upload. Save the file as robots.txt in the root folder of your site and open yourdomain.com/robots.txt to confirm it works.

Frequently Asked Questions About robots.txt

What is a robots.txt file?

A robots.txt file is a plain text file at the root of your website that tells search engine crawlers which URLs they may or may not request. It follows the Robots Exclusion Protocol (RFC 9309).

Is a robots.txt file necessary for SEO?

It is not required, because a missing file simply means everything can be crawled. But it is strongly recommended: it keeps crawlers out of admin and duplicate pages, saves crawl budget and points them to your sitemap.

Where do I put the robots.txt file?

In the root folder of your domain so it opens at yourdomain.com/robots.txt. A robots.txt inside a subfolder is ignored, and each subdomain needs its own file.

How do I create a robots.txt file?

Pick a template in this generator, add any custom Disallow or Allow rules, enter your sitemap URL, then copy or download the result and upload it to your site root. Finally open yourdomain.com/robots.txt to confirm it loads.

What does "User-agent: * Disallow:" mean?

The asterisk means all crawlers, and an empty Disallow means nothing is blocked. This is the standard allow-everything file.

How do I block a specific folder, page or bot?

Add a rule with the crawler name (or * for all) and the path in Disallow, for example Disallow: /private/. To block a single bot use its name, such as GPTBot, with Disallow: /.

Does robots.txt remove a page from Google?

No. Disallow stops crawling, but a blocked URL can still appear in results if other sites link to it. To keep a page out of search results, use a noindex meta tag on a crawlable page or password-protect it.

What is the difference between Disallow and noindex?

Disallow controls crawling, while noindex controls whether a page may appear in search results. Google must be able to crawl a page to see its noindex tag, so do not combine both on the same URL.

Should I block AI bots like GPTBot or ClaudeBot?

That is your choice. Blocking AI crawlers asks them not to use your content for training, and it does not affect Google rankings because these are not search ranking crawlers.

What is Crawl-delay and does Google support it?

Crawl-delay asks a bot to wait a number of seconds between requests. Bing and Yandex honor it, but Google ignores it.

Can I add more than one sitemap?

Yes. Add one Sitemap line per file, or paste several URLs separated by spaces or commas in the Sitemap field of this generator. Sitemap lines work independently of any User-agent group.

Are robots.txt paths case-sensitive?

Yes. /Admin/ and /admin/ are treated as different paths, while directive names such as Disallow are not case-sensitive.

What happens if my site has no robots.txt?

If the file returns a 404, crawlers assume they may crawl everything. If it returns a server error (5xx), Google can pause crawling until it loads again, so keep the file reliably available.

How do I test my robots.txt file?

Use the Validator and URL Tester on this page to check syntax and test any URL for a chosen crawler. After uploading, open the robots.txt report in Google Search Console to check fetch errors.

Can malicious bots ignore robots.txt?

Yes. It is a polite request that reputable crawlers follow, not a security feature. Protect private content with authentication or server rules.

Robots.txt Syntax Explained

Example: robots.txt for a WordPress Site

User-agent: *
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-admin/
Disallow: /wp-login.php
Disallow: /?s=

Sitemap: https://yourwebsite.com/sitemap.xml

The WordPress template above generates this kind of file. It keeps crawlers out of the admin area and internal search pages while leaving your content, CSS and JavaScript crawlable.

Common Robots.txt Mistakes to Avoid

Toolinium quick answer

What does Robots.txt Generator do?

Toolinium Robots.txt Generator creates a properly formatted robots.txt file for your website in seconds.

Updated October 29, 2026