All Pages of UAT site are not to be Indexed by Google

Khine Lwin Myint

Posted on 27/08/2019 05:40:22

Hi,

Is it possible to exclude the UAT website from google search entirely?

We suggested the customer check these fields for all pages in UAT.

But, they don't want to go into each page to check the boxes and they think there should be a default logic in place to ensure that any UAT webpage is not indexed by Google.

And also, is it possible that all new pages created in UAT site will not be indexed by Google by default?

Regards,

Khine

Replies

Morten Bengtson

Posted on 27/08/2019 09:08:58

Hi Khine,

Here are some options...

1) Add the following to robots.txt for the website (see https://doc.dynamicweb.com/documentation-9/content/content/websites-language-layers#3217)

User-agent: *
Disallow: /

2) Use robots meta tag in the master template

<meta name="robots" content="noindex,nofollow" />

3) Set X-Robots-Tag header in web.config

<configuration>
   <system.webServer>
      <httpProtocol>
         <customHeaders>
            <add name="X-Robots-Tag" value="noindex,nofollow" />
         </customHeaders>
      </httpProtocol>
   </system.webServer>
</configuration>

Best regards,
Morten

Hsu Sandar Swe

Posted on 27/09/2019 11:41:04

Hi Morten,

We tried with both robot.txt and meta tag method. But the pages which are using product catalog module are still able to index by google.

Nicolai Pedersen

Posted on 27/09/2019 12:14:44

Google will index whatever it wants - then you can use these directives to let Google know to take out the pages - but it does not happen, and it does not happen fast.

Read this article or google for other solutions: https://www.voog.com/blog/how-to-remove-your-website-or-web-page-from-google

You can also give the UAT a new name with the new settings from Mortens post, then Google will soon remove its references.

BR Nicolai

You must be logged in to post in the forum

Developer forum

All Pages of UAT site are not to be Indexed by Google

Replies