Skip to content
HQ8T

Robots.txt Generator

Create robots.txt and block AI crawlers.

All search engines
User-agent: *
Disallow: /admin/
Disallow: /private/

Upload it to the root of your site so it opens at https://yoursite.com/robots.txt.

How to use

  1. 1Choose to allow or block search engines.
  2. 2List the paths you want to hide, like /admin/.
  3. 3Add your sitemap URL and choose whether to block AI crawlers.
  4. 4Download robots.txt and upload it to your site's root.

Frequently asked questions

Does robots.txt hide pages from Google?

It asks crawlers not to visit them, but a blocked page can still appear if other sites link to it. Use a noindex meta tag to keep a page out of results.

Is robots.txt a security feature?

No. Anyone can read it, and bad bots ignore it. Protect private pages with a login.

Will blocking AI crawlers hurt my Google ranking?

No. Google-Extended only controls AI training use; regular Googlebot keeps crawling and ranking your site.

Does Google follow Crawl-delay?

No, Google ignores it, but Bing, Yandex and some others respect it.

You may also need