WordPress’s “Discourage search engines” setting, explained

WordPress’s “Discourage search engines” setting is one of those small options that can have a very large effect when it is forgotten at the wrong moment. It is useful while a website is being developed, redesigned or prepared for launch, but leaving it enabled on a production site can prevent search engines from indexing pages you actually want people to find.

The setting lives under:

Settings → Reading → Search Engine Visibility

where WordPress displays the option:

Discourage search engines from indexing this site

The wording sounds simple, but there are several important details behind it. The option does not make a website private, it does not create authentication, it is not a replacement for staging protection and it does not guarantee that every crawler on the internet will behave exactly as requested.

The main practical risk is much simpler: the setting can remain enabled after a staging site becomes production, while the website continues to look completely normal to administrators and visitors.

TheOneWP’s Search Visibility Notice module is designed around that exact problem by making WordPress’s current search visibility state easier to notice inside the administration area.

This guide explains what the setting actually does, how WordPress uses it, when it is useful, when it becomes dangerous and what to check before making a WordPress site public.

Where is the “Discourage search engines” setting?

You can find the setting inside the WordPress administration area:

Settings → Reading

Near the bottom of the screen, WordPress includes a section called Search Engine Visibility.

The checkbox is labelled:

Discourage search engines from indexing this site

The official WordPress Reading Settings documentation describes this option as a way to ask search engines not to index the website.

That word ask matters.

The setting communicates an indexing preference to compatible search engines. It does not physically prevent people, bots or applications from requesting the website.

What happens when the setting is enabled?

When the option is enabled, WordPress changes the search-engine directives it generates so that compatible search engines are instructed not to index the site.

Modern WordPress relies on robots directives that communicate a noindex instruction.

Conceptually, the site is saying:

This content may be accessible,
but it should not be included in search results.

This is an indexing instruction rather than an access restriction.

The website can still load normally in a browser while search engines receive instructions not to include its pages in their indexes.

What does noindex mean?

A noindex directive tells a compatible search engine that a resource should not be included in its search index.

A robots meta tag can look like:

<meta name="robots" content="noindex, follow">

The important instruction is:

noindex

Google documents noindex as a supported method for preventing eligible pages from appearing in Google Search.

See the official Google documentation on blocking search indexing with noindex.

Does the setting block crawling?

Not necessarily, and this is one of the most important distinctions to understand.

Crawling and indexing are different processes.

Crawling means that a search-engine bot requests and reads a URL.

Indexing means that the search engine may store and process information about the page as a candidate for appearing in search results.

A page can therefore be:

• crawlable and indexable;
• crawlable but marked noindex;
• blocked from crawling;
• accessible to visitors but excluded from search;
• completely inaccessible because of authentication or server rules.

These states are related, but they are not interchangeable.

Why WordPress does not simply block everything in robots.txt

It may seem logical to solve the problem using a rule such as:

User-agent: *
Disallow: /

but crawling restrictions and indexing directives solve different problems.

If Google cannot crawl a page because robots.txt blocks it, Google cannot fetch the page to discover a page-level noindex instruction.

Google explicitly explains that a page must remain accessible to its crawler for a noindex directive to be discovered and processed.

See the official Google noindex documentation for this distinction.

robots.txt and noindex solve different problems

A useful simplified comparison is:

robots.txt
→ controls crawler access to URLs

noindex
→ controls whether a crawlable resource should appear in the search index

authentication / authorization
→ controls whether the requester is allowed to access the resource

These mechanisms should not be treated as substitutes for one another.

Google’s robots.txt documentation describes robots.txt as a mechanism for controlling crawler access.

For the WordPress-specific security distinction, see Robots.txt vs real access control.

Does “Discourage search engines” make a WordPress site private?

No.

This is one of the most important limitations of the setting.

Enabling:

Discourage search engines from indexing this site

does not require visitors to enter a password.

If somebody knows the URL, they can normally still open it in a browser.

For example:

https://staging.example.com/

can remain completely accessible even while WordPress asks compatible search engines not to index it.

The setting is therefore an indexing control, not a security boundary.

Do not use it to protect confidential staging sites

A staging website can contain information that should not be publicly accessible, including:

• unfinished content;
• future product information;
• customer data copied from production;
• development notes;
• private documents;
• unreleased campaigns;
• test accounts;
• internal business information.

The Search Engine Visibility option does not protect any of this from somebody who can request the URL directly.

A private staging environment should use real access controls where appropriate, such as:

• HTTP authentication;
• hosting-level password protection;
• VPN access;
• IP allowlists;
• identity-aware proxy access;
• private network access.

See WordPress staging site best practices for the wider staging workflow.

When should you enable the setting?

The setting can be useful when a WordPress installation is publicly reachable but is not yet intended to appear in search results.

Typical examples include:

• a website under initial development;
• a redesign on a temporary environment;
• a pre-launch installation;
• a demonstration site;
• a temporary copy used for testing;
• a site whose public content is not yet ready for indexing.

Even in these cases, the setting should be treated as one layer of the environment configuration rather than as access protection.

When should you disable it?

For a normal public website that you want search engines to index, the setting should generally be unchecked.

Before launch, go to:

Settings → Reading

and confirm that:

Discourage search engines from indexing this site

is not enabled.

This should be part of every production-launch checklist.

Why this setting causes launch problems

The setting itself is not dangerous. Forgetting that it is enabled is the problem.

A common workflow looks like this:

Development begins
↓
Search indexing is discouraged
↓
The website is built and tested
↓
The site is migrated to production
↓
Design, DNS, forms and caching are checked
↓
Search Engine Visibility is forgotten
↓
The production site remains noindex

The site may look completely normal to visitors while search engines receive instructions not to index it.

Make the search visibility state visible to administrators

The setting is buried inside Reading Settings, which means administrators need to remember that it exists before they can check it.

TheOneWP’s Search Visibility Notice module surfaces the current WordPress search visibility state inside the administration interface.

The module does not change the WordPress setting automatically.

Its purpose is to expose the current state so that an installation configured to discourage indexing is harder to overlook during development, migration and launch work.

This is particularly useful on sites that regularly move between staging and production or are managed by several administrators.

Check the setting during every WordPress launch

The Search Engine Visibility checkbox should be included in the final production review.

A basic launch check should include:

• production domain;
• HTTPS;
• Search Engine Visibility;
• robots directives;
• robots.txt;
• canonical URLs;
• XML sitemap;
• redirects;
• internal links;
• staging-domain references.

For the complete process, see A WordPress pre-launch SEO checklist.

What happens if you accidentally leave it enabled after launch?

If the website launches while the setting remains enabled, search engines may encounter noindex instructions while crawling the site.

As those directives are processed, new pages may fail to enter the index and previously indexed pages may eventually be removed.

The effect is not necessarily immediate because search engines need to revisit and process affected URLs.

If you discover the problem, the first step is:

Settings → Reading
↓
Uncheck “Discourage search engines from indexing this site”
↓
Save Changes

Will pages immediately return to Google after disabling it?

Not necessarily.

Removing the setting makes affected pages eligible for indexing again, assuming there are no other indexing restrictions.

Google still needs to recrawl and process the URLs.

Indexing can also depend on factors such as:

• crawl accessibility;
• canonical configuration;
• page quality;
• duplicate content;
• internal linking;
• sitemap discovery;
• other robots directives;
• server responses.

Removing noindex means a page can become eligible for indexing again. It does not command a search engine to index the page immediately.

Check the actual page source after changing the setting

After disabling the WordPress setting, inspect the HTML of an important production page.

Search for:

robots

and verify that an unexpected noindex directive is no longer present.

This matters because WordPress core is not the only component that can generate robots metadata.

SEO plugins can also add noindex

A WordPress website may have several possible sources of indexing directives.

For example:

• WordPress core;
• SEO plugins;
• custom PHP code;
• theme functions;
• HTTP X-Robots-Tag headers;
• CDN or reverse-proxy configuration;
• web-server rules.

Therefore, unchecking Search Engine Visibility does not prove that every page is indexable.

It only removes that particular WordPress-level restriction.

Check individual post and page settings too

An SEO system may allow individual content to be configured as:

noindex

even when WordPress globally allows search engines to index the site.

This can be completely intentional.

For example, you may choose not to index:

• thin utility pages;
• internal search results;
• duplicate archives;
• temporary campaign pages;
• certain taxonomy archives;
• other public URLs without standalone search value.

The global WordPress setting and page-level SEO controls therefore serve different purposes.

When TheOneWP is being used as the site’s SEO layer, its SEO Meta module provides page-level metadata controls separately from the global WordPress Search Engine Visibility setting.

Global search visibility vs page-level noindex

The distinction can be summarized as:

WordPress Search Engine Visibility
→ global site-level state

Page-level SEO robots settings
→ individual URL or content-type decisions

This means a site can be globally indexable while selected pages remain intentionally noindex.

It also means that turning off the global WordPress setting does not automatically remove an independently configured page-level noindex.

When troubleshooting, inspect both layers.

What about X-Robots-Tag?

Robots directives do not have to appear inside HTML.

A server can send an HTTP header such as:

X-Robots-Tag: noindex

This can control indexing for HTML pages and other resource types where a normal HTML meta tag may not be appropriate.

Google documents both robots meta tags and the X-Robots-Tag HTTP header in its robots meta tag specifications.

If a page remains non-indexable after changing WordPress settings, checking its HTTP response headers is therefore worthwhile.

Does enabling the setting remove pages that are already indexed?

It can lead compatible search engines to remove affected pages after they recrawl and process the relevant indexing directives.

This does not normally happen the instant you click Save Changes.

A search engine must first revisit the URL and discover the new instruction.

This is why changing indexing settings on an established production website should be treated carefully.

Can a noindex page still be crawled?

Yes.

In fact, the crawler generally needs to access the page in order to see its noindex instruction.

This leads to an important rule:

If you want Google to process a noindex directive,
do not simultaneously prevent Google from crawling the page
with robots.txt.

Google makes this requirement explicit in its documentation.

Can a robots.txt-blocked URL still appear in search?

Potentially, yes.

If Google knows that a URL exists through links or other discovery mechanisms but cannot crawl the page because of robots.txt, it may still know about the URL while being unable to inspect its page-level metadata.

This is another reason why:

robots.txt ≠ noindex

and why the two should not be treated as interchangeable controls.

Does the setting affect the XML sitemap?

The relationship between a site-wide visibility setting and sitemap output depends on the sitemap implementation being used.

Regardless of implementation, a production site should avoid contradictory signals such as:

XML sitemap:
Discover these URLs.

Robots directives:
Do not index these URLs.

Before launch, inspect both the indexing configuration and the actual sitemap output.

See What is an XML sitemap, and why does it matter? for the complete sitemap explanation.

An XML sitemap does not override noindex

Including a page inside an XML sitemap does not force a search engine to index it.

A page can appear in a sitemap and still carry a noindex directive.

The sitemap helps with URL discovery. It is not an indexing command.

If you want a page indexed, the technical signals should generally agree:

crawlable
+
indexable
+
canonical
+
internally linked
+
included in the relevant sitemap
=
clearer indexing configuration

Even then, the final indexing decision belongs to the search engine.

What happens to custom post types?

The global WordPress Search Engine Visibility setting applies to the website rather than representing a post-type-specific SEO strategy.

When the site is intended to be indexed, individual post types should still be reviewed separately.

For example, a site might contain:

• public posts that should be indexed;
• public pages that should be indexed;
• a product custom post type that should be indexed;
• an internal utility post type that should remain excluded;
• taxonomy archives requiring individual decisions.

See WordPress custom post types and SEO for a deeper explanation of these decisions.

Should you enable the setting on a maintenance-mode site?

Possibly, but do not confuse the two mechanisms.

A maintenance or coming-soon system determines what visitors see.

The Search Engine Visibility setting controls indexing signals.

They may be used together during development, but one does not automatically replace the other.

Also consider whether the site already has indexed content.

Applying broad noindex directives to an established production site merely because it is temporarily under maintenance can create unnecessary indexing consequences.

Should you use it during a redesign?

Use it on the development or staging copy when appropriate, not automatically on the existing production site.

A typical redesign architecture looks like:

example.com
→ existing production site
→ remains indexable

staging.example.com
→ redesign
→ protected from public access
→ discouraged from indexing

At launch, the redesigned website replaces production and the production indexing settings are verified again.

This avoids accidentally removing the existing live website from search while the redesign is still being built.

Do not copy staging settings blindly to production

Website migrations can copy considerably more than content.

Depending on the deployment process, production may inherit:

• WordPress options;
• plugin settings;
• SEO configuration;
• environment URLs;
• robots directives;
• search visibility settings.

If a staging database replaces the production database during deployment, Search Engine Visibility deserves an explicit post-migration check.

How to check whether WordPress is discouraging indexing

The simplest method is:

1. Log in to WordPress.
2. Open Settings → Reading.
3. Find Search Engine Visibility.
4. Check whether Discourage search engines from indexing this site is enabled.

For a public production site that should appear in search, the checkbox would normally be unchecked.

How Search Visibility Notice helps

The difficulty with this setting is not that it is technically complex.

The difficulty is that its current state is easy to forget once administrators leave the Reading Settings screen.

TheOneWP’s Search Visibility Notice module surfaces the WordPress search visibility status inside the administration interface.

This gives administrators a persistent visual reminder when WordPress is configured to discourage indexing.

The module does not:

• force Google to index the website;
• change individual page robots settings;
• replace access control;
• automatically change the WordPress option.

Its responsibility is deliberately focused: expose the existing WordPress status so that a high-impact global setting is less likely to remain unnoticed.

Search Visibility Notice is most useful during staging and launch workflows

The module is particularly useful when a site moves through states such as:

Development
→ staging
→ pre-launch
→ production

During development, discouraging indexing may be intentional.

During production, the same configuration may be a serious mistake.

Making the current state visible reduces dependence on somebody remembering to reopen Settings → Reading after every migration.

A normal visibility state does not mean the site is fully optimized

Allowing indexing is only one part of technical SEO.

A site can allow search engines to index it and still have problems such as:

• broken internal links;
• incorrect canonical URLs;
• poor metadata;
• duplicate pages;
• incorrect redirects;
• weak internal linking;
• missing sitemap entries;
• server errors;
• poor mobile rendering;
• low-value content.

Search visibility only tells you whether one important indexing restriction is active. It does not provide a complete SEO health assessment.

What to check if the site still does not appear in Google

If the WordPress setting is disabled but the site still does not appear in Google, continue checking the wider indexing chain.

Review:

• robots meta tags;
• X-Robots-Tag headers;
• robots.txt;
• canonical URLs;
• HTTP status codes;
• XML sitemaps;
• internal links;
• Google Search Console;
• whether Google has actually discovered and crawled the URLs.

The Search Engine Visibility checkbox is an important first check, but it is not the only reason a WordPress site may be missing from search.

Use Why isn’t my WordPress site showing up on Google? for the complete indexing diagnostic workflow.

Common mistakes with the Search Engine Visibility setting

1. Leaving it enabled after launch

The development setting reaches production and nobody notices because the website continues working normally for visitors.

2. Treating it as password protection

The option does not stop somebody from visiting a known URL.

Use real authentication when content must remain private.

3. Assuming robots.txt and noindex are the same thing

They control different parts of crawler and indexing behavior.

4. Blocking crawling while expecting noindex to be processed

A crawler generally needs access to the page before it can discover the page-level robots directive.

5. Assuming unchecking it guarantees indexing

It only removes one possible indexing restriction.

6. Forgetting page-level SEO settings

Individual pages or content types may still have their own robots directives.

7. Using it as the only staging safeguard

A development website can remain publicly accessible even when indexing is discouraged.

8. Forgetting to verify the setting after migration

A staging database or configuration import can reintroduce the previous visibility state on production.

WordPress Search Engine Visibility checklist

Before launching a WordPress website, verify the following:

• Open Settings → Reading.
• Check the Search Engine Visibility setting.
• Uncheck “Discourage search engines” when the production site should be indexed.
• Save the setting.
• Inspect representative page source for unexpected noindex.
• Check HTTP headers for unexpected X-Robots-Tag directives.
• Review page-level SEO robots settings.
• Review robots.txt separately.
• Check the XML sitemap.
• Make sure staging remains genuinely protected.
• Verify the configuration again immediately after deployment.
• Use Search Console to inspect representative production URLs after launch.

Using TheOneWP to make the setting harder to forget

The WordPress Search Engine Visibility option already exists in core, so the problem does not require another mechanism that duplicates the setting.

The practical problem is visibility.

TheOneWP’s Search Visibility Notice module exposes the current WordPress state where administrators are more likely to notice it during normal backend work.

This creates a clearer workflow:

WordPress setting
→ controls the global indexing preference

Search Visibility Notice
→ surfaces the current state

SEO Meta
→ manages page-level SEO directives where needed

Search Console
→ shows how Google processes the live URLs

The modules do not replace WordPress’s own setting or Google’s indexing systems.

They make the configuration easier to understand and less dependent on remembering where one high-impact checkbox is hidden.

The setting is simple; the distinction is not

WordPress’s “Discourage search engines from indexing this site” option gives administrators a simple global way to tell compatible search engines that a website should not currently appear in search results.

But the setting should be understood for what it is.

It is an indexing signal.

It is not authentication.

It is not a password.

It is not a firewall.

It is not the same thing as robots.txt.

And disabling it does not guarantee that a website will suddenly appear in Google.

During development, use the setting deliberately. During production launch, check it explicitly. When a site needs to remain genuinely private, use real access control instead.

And when several people maintain the same WordPress installation, making the state visible through TheOneWP’s Search Visibility Notice can reduce the chance that an intentional staging configuration quietly becomes an accidental production SEO problem.

For the official behavior of the WordPress option, refer to the WordPress Reading Settings documentation. For Google’s indexing controls, see the official noindex documentation and robots meta tag specifications.