CrawlWP Premium can automatically find pages in WordPress that Google reports as not indexed and submit them for indexing without you having to submit each page manually. This feature is called auto-indexing.
Auto-indexing is separate from the submissions for indexing to search engines CrawlWP makes when you publish or update content. Those submissions are available independently of auto-indexing. See What IndexNow is and which engines use it for more information about those submissions.
Keep in mind that submitting a URL for indexing by search engines does not guarantee that a search engine will index it. CrawlWP sends the URL to the enabled indexing service and keeps checking its index status afterward.
Before you start
- Connect the Google API under CrawlWP > Settings > API Settings, and make sure the site is available in Google Search Console for the same account. CrawlWP uses Google’s index status to determine which pages need attention. See Integrating WordPress with Google Search API.
- Switch on at least one indexing service under CrawlWP > Settings > Indexing: IndexNow, Google, Bing, or Yandex. Auto-indexing uses the services you have enabled. Categories and other terms are submitted through IndexNow only, and only when Term added is enabled.
- Select the post types and taxonomies you want CrawlWP to track on the Indexing tab under General. See Excluding posts or post types from indexing.
Turn on auto-indexing
- Go to CrawlWP > SEO Indexing.
- Scroll down to the indexing table.
- Click the Enable Auto-Indexing switch to turn it on.
The setting is saved immediately. There is no Save button to click.
How automatic indexing works
After you enable auto-indexing, CrawlWP uses scheduled background jobs to check index status and process pages that need to be submitted. The index status checks continue running whether auto-indexing is enabled or disabled. The submission jobs run only while auto-indexing is enabled.
Index status checks
These checks keep the status in the SEO Indexing table up to date and help CrawlWP identify pages that may need another indexing submission to search engines.
| Job | How often | What it does |
|---|---|---|
| Google check | Every 20 minutes | Checks pages with Not Checked, In Progress, or Not Indexed status when they have not been checked for 72 hours |
| Bing check | Every 20 minutes | Checks pages with Not Checked, In Progress, or Likely Not Indexed status when they have not been checked for 72 hours. This runs only when the Bing API is connected |
| Yandex check | Every 20 minutes | Performs the same type of check as Bing when the Yandex API is connected |
| Re-check indexed pages | Daily | Checks pages that Google previously reported as Indexed when they have not been checked for a month, helping detect pages that later drop out of the index |
| History chart | Hourly | Records the day’s indexed and not-indexed totals for the Google Index History chart |
See Understanding index statuses, including Likely Indexed, for a detailed explanation of the statuses shown in the table.
Submission jobs
Submission jobs run only when Enable Auto-Indexing is turned on. They handle both WordPress pages CrawlWP identifies as not indexed by search engines and pages you manually place in the submission queue.
| Job | How often | What it does |
|---|---|---|
| Submit not-indexed pages | Hourly | Submits pages Google reports as Not Indexed. CrawlWP will not submit the same page again within 7 days of its last submission |
| Submit queued pages | Hourly | Submits up to 50 pages marked In Progress, such as pages added through the Submit for Indexing bulk action. A page remains In Progress until its next Google check, which can take up to 3 days, and it is submitted every 12 hours again until then. |
This means auto-indexing does not continuously resubmit every URL on your WordPress site for search engine indexing. CrawlWP uses the index status and submission timing rules to decide which pages need another submission.
Only pages that Google reports as Not Indexed, along with pages you have manually queued, are submitted by these jobs. Pages with Not Checked or Indexing Excluded status are not automatically submitted.
After a page is submitted, the Last Submitted column records the most recent submission attempt. A corresponding entry will usually also appear in the submission log. See Reading the indexing submission log.
What the different submission statuses mean
The statuses in the SEO Indexing table help you understand where a page is in the process. A page marked Not Indexed is a candidate for automatic submission. A page marked In Progress has already been submitted and is waiting for another index-status check. A page marked Not Checked does not have a current Google index result and is not automatically submitted until CrawlWP receives a relevant status.
Indexing Excluded is different from a page simply being not indexed. It means the page has been excluded from CrawlWP’s indexing workflow and will not be picked up by the auto-indexing submission process.
Best practices for auto-indexing
- Enable auto-indexing only for post types and taxonomies that contain pages you actually want search engines to index.
- Keep your Google API connection working so CrawlWP can identify pages that are still not indexed and update their statuses.
- Use the SEO Indexing table and submission log to check what CrawlWP has submitted instead of repeatedly submitting the same URLs manually.
Auto-indexing is most useful as an ongoing maintenance feature. Once enabled, CrawlWP can continue finding pages that Google reports as not indexed and submit them according to its built-in schedule and limits.
Troubleshooting automatic indexing
Nothing is being submitted. Confirm that Enable Auto-Indexing is switched on and that at least one service is enabled under CrawlWP > Settings > Indexing. Then look at the table for pages marked Not Indexed or In Progress. Pages marked Not Checked are not submitted automatically.
Pages remain Not Checked. Check the Google API connection and make sure the site is available in Google Search Console for the connected account. When CrawlWP cannot find the site in Search Console, it displays a notice near the top of the relevant admin screens.
The auto-indexing switch turns off after reloading. This usually means the setting didn’t save successfully. Reload the page and enable the switch again. If the problem continues, check that you are logged in with an administrator account and that your site is not blocking the request used to save the setting.
Pages are missing from the table. Click Missing Pages? Click Here to refresh the list of published posts and terms and remove entries for content that no longer exists.
A page stays In Progress. This is expected until CrawlWP performs another Google index check. The next check can take up to 3 days, and while the page waits for that check, CrawlWP can resubmit it every 12 hours.
Hourly jobs are running late. Check whether WP-Cron is running normally. On low-traffic sites, use a server cron job to call wp-cron.php every 5 or 10 minutes. If CrawlWP reports missed cron schedules, see How to fix cron events missing their schedules.
Best practices for auto-indexing
- Enable auto-indexing only for post types and taxonomies that contain pages you actually want search engines to index.
- Keep your Google API connection working so CrawlWP can identify pages that are still not indexed and update their statuses.
- Use the SEO Indexing table and submission log to check what CrawlWP has submitted instead of repeatedly submitting the same URLs manually.
Auto-indexing is most useful as an ongoing maintenance feature. Once enabled, CrawlWP can continue finding pages that Google reports as not indexed and submit them according to its built-in schedule and limits.
How to stop auto-indexing
Click the Enable Auto-Indexing switch again to disable the feature.
Pages already marked In Progress keep that status, but the auto-indexing submission jobs no longer submit them while the feature is disabled.
To remove tasks that are still waiting in the background queue, click Clear Queue above the table and confirm the action. Clearing the queue removes waiting tasks but does not change page statuses or your indexing settings.
Troubleshooting automatic indexing
Nothing is being submitted. Confirm that Enable Auto-Indexing is switched on and that at least one service is enabled under CrawlWP > Settings > Indexing. Then look at the table for pages marked Not Indexed or In Progress. Pages marked Not Checked are not submitted automatically.
Pages remain Not Checked. Check the Google API connection and make sure the site is available in Google Search Console for the connected account. When CrawlWP cannot find the site in Search Console, it displays a notice near the top of the relevant admin screens.
The auto-indexing switch turns off after reloading. This usually means the setting was not saved successfully. Reload the page and enable the switch again. If the problem continues, check that you are logged in with an administrator account and that your site is not blocking the request used to save the setting.
Pages are missing from the table. Click Missing Pages? Click Here to refresh the list of published posts and terms and remove entries for content that no longer exists.
A page stays In Progress. This is expected until CrawlWP performs another Google index check. The next check can take up to 3 days, and while the page waits for that check, CrawlWP can resubmit it every 12 hours.
Hourly jobs are running late. Check whether WP-Cron is running normally. On low-traffic sites, use a server cron job to call wp-cron.php every 5 or 10 minutes. If CrawlWP reports missed cron schedules, see How to fix cron events missing their schedules.
Best practices for auto-indexing
- Enable auto-indexing only for post types and taxonomies that contain pages you actually want search engines to index.
- Keep your Google API connection working so CrawlWP can identify pages that are still not indexed and update their statuses.
- Use the SEO Indexing table and submission log to check what CrawlWP has submitted instead of repeatedly submitting the same URLs manually.
Auto-indexing is most useful as an ongoing maintenance feature. Once enabled, CrawlWP can continue finding pages that Google reports as not indexed and submit them according to its built-in schedule and limits.
How to schedule auto-indexing
The schedule is fixed inside CrawlWP. No setting lets you change the hourly submission interval or the frequency of index checks.
CrawlWP uses WordPress scheduled tasks, commonly called WP-Cron, to run these jobs. WP-Cron triggers on site visits, so scheduled tasks can run late when a site doesn’t receive regular traffic.
For a busy site, this may not be noticeable. For a low-traffic site, however, an hourly CrawlWP job may not run at the exact time its schedule expects.
To make the schedule more reliable, ask your hosting provider to create a server cron job that loads https://your-site.com/wp-cron.php every 5 or 10 minutes. Most hosting control panels provide an option for creating cron jobs.
Once the server cron job is working, you can disable WordPress’s normal page-load trigger for WP-Cron by adding the following line to your wp-config.php file:
define( 'DISABLE_WP_CRON', true );This setup lets the server trigger wp-cron.php on a regular schedule instead of relying on visitors to your site.
When CrawlWP’s hourly cron event is more than 10 minutes late, CrawlWP shows the notice: “One or more CrawlWP’s cron events have missed their schedule.” See How to fix cron events missing their schedules if you see this message.
Auto-indexing limits
Auto-indexing does not have a separate daily request counter. Instead, CrawlWP spaces out submissions and index checks and pauses affected requests when an API returns an error or quota limit is reached.
- Google allows 2,000 URL Inspection checks per site per day. If CrawlWP reaches this limit, Google checks are paused for 3 hours.
- The Google Indexing API allows 200 submissions per day by default. If Google returns an error, CrawlWP pauses Google submissions for 6 hours.
Large sites with many pages that are not indexed can therefore take some time to work through the available checks and submissions. Reaching a Google quota does not mean the feature is broken; CrawlWP pauses the affected operation and resumes it later.
See Google Indexing API daily quotas and what happens when you hit them for more information about Google’s quotas and CrawlWP’s handling of quota errors.
How to stop auto-indexing
Click the Enable Auto-Indexing switch again to turn the feature off.
Pages already marked In Progress keep that status, but the auto-indexing submission jobs no longer submit them while the feature is disabled.
To remove tasks that are still waiting in the background queue, click Clear Queue above the table and confirm the action. Clearing the queue removes waiting tasks but does not change page statuses or your indexing settings.
Troubleshooting automatic indexing
Nothing is being submitted. Confirm that Enable Auto-Indexing is switched on and that at least one service is enabled under CrawlWP > Settings > Indexing. Then look at the table for pages marked Not Indexed or In Progress. Pages marked Not Checked are not submitted automatically.
Pages remain Not Checked. Check the Google API connection and make sure the site is available in Google Search Console for the connected account. When CrawlWP cannot find the site in Search Console, it displays a notice near the top of the relevant admin screens.
The auto-indexing switch turns off after reloading. This usually means the setting was not saved successfully. Reload the page and enable the switch again. If the problem continues, check that you are logged in with an administrator account and that your site is not blocking the request used to save the setting.
Pages are missing from the table. Click Missing Pages? Click Here to refresh the list of published posts and terms and remove entries for content that no longer exists.
A page stays In Progress. This is expected until CrawlWP performs another Google index check. The next check can take up to 3 days, and while the page waits for that check, CrawlWP can resubmit it every 12 hours.
Hourly jobs are running late. Check whether WP-Cron is running normally. On low-traffic sites, use a server cron job to call wp-cron.php every 5 or 10 minutes. If CrawlWP reports missed cron schedules, see How to fix cron events missing their schedules.
Best practices for auto-indexing
- Enable auto-indexing only for post types and taxonomies that contain pages you actually want search engines to index.
- Keep your Google API connection working so CrawlWP can identify pages that are still not indexed and update their statuses.
- Use the SEO Indexing table and submission log to check what CrawlWP has submitted instead of repeatedly submitting the same URLs manually.
Auto-indexing is most useful as an ongoing maintenance feature. Once enabled, CrawlWP can continue finding pages that Google reports as not indexed and submit them according to its built-in schedule and limits.