Developer forum

Forum » CMS - Standard features » All Pages of UAT site are not to be Indexed by Google

All Pages of UAT site are not to be Indexed by Google

Khine Lwin Myint
Reply

Hi, 

Is it possible to exclude the UAT website from google search entirely?

We suggested the customer check these fields for all pages in UAT. 

screenshot-1.png

But, they don't want to go into each page to check the boxes and they think there should be a default logic in place to ensure that any UAT webpage is not indexed by Google.

And also, is it possible that all new pages created in UAT site will not be indexed by Google by default?

 

Regards,

Khine

Replies

 
Morten Bengtson
Morten Bengtson
Reply

Hi Khine,

Here are some options...

1) Add the following to robots.txt for the website (see https://doc.dynamicweb.com/documentation-9/content/content/websites-language-layers#3217)

User-agent: *
Disallow: /

2) Use robots meta tag in the master template

<meta name="robots" content="noindex,nofollow" />

3) Set X-Robots-Tag header in web.config

<configuration>
   <system.webServer>
      <httpProtocol>
         <customHeaders>
            <add name="X-Robots-Tag" value="noindex,nofollow" />
         </customHeaders>
      </httpProtocol>
   </system.webServer>
</configuration>

Best regards,
Morten

 
Hsu Sandar Swe
Hsu Sandar Swe
Reply

Hi Morten, 

We tried with both robot.txt and meta tag method. But the pages which are using product catalog module are still able to index by google. 

robot.png
 
Nicolai Pedersen
Nicolai Pedersen
Reply

Google will index whatever it wants - then you can use these directives to let Google know to take out the pages - but it does not happen, and it does not happen fast.

Read this article or google for other solutions: https://www.voog.com/blog/how-to-remove-your-website-or-web-page-from-google

You can also give the UAT a new name with the new settings from Mortens post, then Google will soon remove its references.

BR Nicolai

 

You must be logged in to post in the forum