WordPress Robots.txt Generator

Create a safe robots.txt for WordPress and WooCommerce with sitemap, search, feed and AI crawler rules.

---

A robots.txt file tells search engines which parts of your site to skip. A good one saves crawl budget for the pages that matter. A bad one can remove your whole site from Google. This generator writes a clean file for WordPress and WooCommerce sites, with the dangerous options clearly labelled.

How to use it

1. Enter your site address and choose the sitemap your SEO plugin creates.
2. Keep the standard WordPress rules ticked. Add WooCommerce rules if you run a shop.
3. Optionally block AI training crawlers, internal search results and feeds.
4. Copy the result and save it as robots.txt in your site's root folder, or paste it into the file editor in Yoast or Rank Math.
5. Open yoursite.com/robots.txt in a browser to confirm it is live.

What not to block

Never block /wp-content/, /wp-includes/, or your CSS and JavaScript. Google needs them to render your pages, and blocking them can hurt rankings. The one thing in /wp-admin/ that must stay open is admin-ajax.php, because many themes and plugins use it on the public side.

Questions people ask

Does robots.txt stop a page appearing in Google?

Not reliably. It stops crawling, not indexing. A blocked page that other sites link to can still be listed without a description. To keep a page out, use a noindex tag and leave it crawlable.

Will blocking AI crawlers work?

It works for crawlers that choose to respect robots.txt, which the major AI companies say they do. It does nothing against scrapers that ignore it.

What is the staging option for?

Development copies of a site should not be indexed. Blocking everything keeps them out. Remember to switch it off when the site goes live, because that mistake is common and costly.

Does WordPress already create a robots.txt?

It serves a virtual one if no physical file exists. A real file in your root folder takes priority. This tool runs entirely inside your web browser. Nothing you type is uploaded to a server, so there is no queue and no copy of your data left behind when you close the tab.