Developer forum

Not sure I understand what you mean...

Is it there or is it not? What do you get on /robots.txt? Remember it depends on your 404 settings.

Swift has an empty robots.txt that I forgot to remove - because it sends out some disallow on local.dynamicweb.dk domains that gave us testing issues.

The entries in robots.txt doesn't necessarily prevent Google from indexing your content.
Instead you can either make the content protected (password/permissions) or use noindex... or remove the content ;)

You can enable noindex on websites or pages:​
https://doc.dynamicweb.com/documentation-9/content/content/pages#3239
https://doc.dynamicweb.com/documentation-9/content/content/websites#3213

See detailed advice from Google here:
https://developers.google.com/search/docs/advanced/robots/intro
https://developers.google.com/search/docs/advanced/crawling/control-what-you-share

@Adrian - do you happen to check these things on *.dynamicweb-cms.com?

Because that will always send out 

ser-agent: *
Disallow: /

Hi guys,

What I got initially was a 404 error.

I have checked this post: https://doc.dynamicweb.com/forum/cms-standard-features/cms-standard-features/sitemap-xml-and-robots-txt-not-working and checked my setup and I have found that my errorMode was set to Detailed. I have changed it to Custom but now I get an empty response even if I have something in my Robots.

When checking with Google Search console Robots tester, they report it as a 404 response. The 404 redirect is configured properly on the solution.

My check is on <!--Removed-->

@Morten it looks like the default Rapido master files don't read the settings for noindex, nofollow from the website settings, they are hardcoded in the template. That was the first one I tried.

I have attached my robots settings.

Adrian

I do not get a 404 when visiting robots.txt on the above url

I cannot explain it... Do you happen to have a robots.txt file in the root of the website?

Hi Nicolai,

No. This is all I have:

<!-- Removed-->

Adrian

Add a robots.txt file there and put what you need into it. That will work here and now.

Hi Nicolai,

Ok. I can do that.

Thank you,

Adrian

Hi guys,

I know it's an old post but we just found another site with the same issue.

The solution in my case was to disable Tracking.

It could be related to one of the Tracking settings but in my case I have just disabled tracking and the error went away.

These are our current settings:

Solution is 9.13.11.

Thank you,
Adrian