Robots.txt Generator

What does Robots.txt Generator do?

Robots.txt Generator creates the robots.txt file that tells search engine crawlers which parts of your website they may visit. You can allow the whole site, block private folders such as /admin/, add your sitemap address and block specific bots, for example AI crawlers. The file is built as you type.

How to use Robots.txt Generator

1. Choose the default rule (normally "Allow").
2. List folders or pages to block, one per line.
3. Add the full URL of your sitemap and tick any bots you want to block.
4. Copy the result or press Download robots.txt and upload it to the main folder of your website.

When is it useful?

• Creating a robots.txt for a new website.
• Keeping crawlers out of admin, login or cart pages.
• Telling search engines where your sitemap is.

Example

Blocking /admin/ and adding a sitemap gives:
User-agent: *
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

Good to know

• robots.txt is public. It does not hide or protect pages; use a password or a noindex tag for private content.
• Blocking a page in robots.txt does not always remove it from search results if other sites link to it.
• Nothing you type is sent to our server.

Frequently asked questions

Is Robots.txt Generator free?

Yes. You can use it as often as you like without creating an account.

Is my text sent to your server?

No. Robots.txt Generator works entirely in your web browser, so what you type or paste stays on your device and is not stored.

Does it work on my phone?

Yes. It runs in any modern browser on a phone, tablet or computer - there is nothing to install.