Features
robots.txt Editor for WordPress
robots.txt is a small file that tells search engine crawlers which parts of your site they may visit. CrawlWP lets you edit it inside WordPress. The editor starts with what your site serves today, including lines other plugins add, so you never start from a blank box.
What it does
What the CrawlWP robots.txt editor does
Edit your robots.txt from your WordPress dashboard, safely, and with the full picture in front of you.
On this page
Edit robots.txt in WordPress
No FTP and no file manager. Edit it from your WordPress dashboard.
Pre-filled with your live file
The editor shows what your site serves now: WordPress defaults plus anything other plugins add.
Read-only until you choose
Editing stays off until you tick Enable robots.txt editing, so nothing changes by accident.
What you save is the whole file
The content you save becomes your complete robots.txt. Nothing is added out of sight.
Sitemap lines included
Your sitemap addresses are already listed so that crawlers can find them.
Easy to undo
Untick Enable robots.txt editing to restore the default WordPress output.
No file to lose
WordPress creates robots.txt on the fly, so there is no file on your server to lose during a move or update.
Control AI crawlers
Add rules for AI crawlers, such as GPTBot, if you do not want them reading your site.
Robots rules for each post
For single pages, set noindex, nofollow, noarchive, nosnippet, or noimageindex in the SEO metabox.
Rules for a whole post type
Set noindex for every post of a type at once.
Canonical URLs
Set the canonical URL for any post in the SEO metabox.
Free
The robots.txt editor is part of CrawlWP Lite, the free version.
Why it matters
Why edit your robots.txt file
robots.txt helps crawlers spend their time on the pages that matter. You can keep them away from areas that do not need crawling, such as internal search results or log files.
But be careful: one wrong line can block your whole site. That is why CrawlWP shows you the current file first, and keeps editing off until you turn it on.
How it works
Edit your robots.txt in three steps
It takes a couple of minutes, and you can undo it anytime.
Step 1
Look at the current file
Open the Robots.txt screen. The editor shows what your robots.txt serves right now, read-only.
Step 2
Turn on editing
Tick Enable robots.txt editing, make your changes, and save. What you save becomes the complete file.
Step 3
Check it live
Open yoursite.com/robots.txt to see the result. Google Search Console also has a robots.txt report.
robots.txt or noindex?
Use the right tool for the job
robots.txt controls crawling. It does not stop a page from appearing in search. If you want a page out of Google, use noindex instead.
What the WordPress default looks like
If you have not changed anything, your robots.txt looks something like this. Your sitemap line is added for you.
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yoursite.com/wp-sitemap.xmlQuestions
robots.txt questions
Answers to the questions people ask most about editing robots.txt in WordPress.
What is a robots.txt file?
It is a plain-text file at yoursite.com/robots.txt. It tells search engine crawlers which parts of your site they may or may not visit.
Where is robots.txt in WordPress?
WordPress creates it on the fly when someone visits /robots.txt. There is no file on your server unless someone uploaded one.
I have a real robots.txt file on my server. Will the editor work?
Not until you remove it. A real file in your site’s root folder is served instead of the WordPress version, so your changes will not show. Keep a copy, then delete or rename it.
Does Disallow remove a page from Google?
No. It stops crawling, not indexing. To keep a page out of search results, use noindex and let crawlers visit the page.
What if I make a mistake?
Untick Enable robots.txt editing and save. WordPress returns to its default output immediately.
Can I block AI crawlers?
Yes. Add a rule for each crawler, such as User-agent: GPTBot followed by Disallow: /. Crawlers that follow robots.txt will stay away, but not every crawler does.
How do I test my robots.txt?
Open yoursite.com/robots.txt in your browser, and check the robots.txt report in Google Search Console.
Is the robots.txt editor free?
Yes. It is part of CrawlWP Lite, the free version.
Pricing
Straightforward pricing, every feature included
Standard
For 1 Site
$59
/ year
Pro
For 5 Sites
$159
/ year
Agency
For Unlimited Sites
$259
/ year
Standard
For 1 Site
$359
once
Pro
For 5 Sites
$559
once
Agency
For Unlimited Sites
$759
once
The robots.txt editor is free in CrawlWP Lite. Get CrawlWP Lite
Take control of your robots.txt
Join thousands of website owners using CrawlWP for their WordPress SEO. The robots.txt editor is free and ready as soon as you install it.
yoursite.com
https://yoursite.com › guides › wordpress-seo
A Practical Guide to WordPress SEO in 2026
Learn how to structure, optimize, and index a WordPress site so search engines find your content within hours rather than weeks.
Google · indexed
Bing · indexed
Yandex · indexed