Developer forum

Hi Jose.

You cannot see the friendly urls in the log. It is perfectly normal to see Default.aspx in the logs, because that is the file that URL rewrite targets.

BR Nicolai

Hi Nicolai,
I understand that in the log there must be Default.aspx urls.
What I don't understand is the lines of the IIS log containing "googlebot". Only
those lines.The page code has no links to URLs with a Default.aspx format. 
Moreover, the log of the server (located at DW cloud) shows two types of entries:Type 1) URLs that are rewritten by DW before they are sent to the actual execution. No problem with this type of lines
Example:
Line 167: 2021-10-20 00:00:36 10.21.142.46
GET /Default.aspx ID=2399&productid=PROD3632&Feed=True&redirect=true 443 - 195.154.126.60
Mozilla/5.0+(compatible;+AhrefsSiteAudit/6.1;++http://ahrefs.com/robot/)
https://mydomain.com/nameofmyproduct 200 0 0 203Type 2) URLs that are read and crawled by googlebot
Example:
Line 2004: 2021-10-20 00:19:28 10.21.142.46
GET /Default.aspx ID=2399&GroupID=GROUP502 443 - 66.249.66.203
Mozilla/5.0+(compatible;+Googlebot/2.1;++http://www.google.com/bot.html) - 200 0 0 2390.
Here the log indicates that google has entered and found the page not friendly
Default.aspx..... and that the response from the server was 200.
If you type that url in the browser, the answer is 301, as it redirects you to the friendly url.
So why does the log put 200 instead of 301?All the googlebot entries are with that Default format instead of the friendly URL. 
This is a very serious problem because, if we have 4000 products, 
googlebot read first the 4000 non-friendly URLs and, after that it almost doesn't read the friendly URLs. As a result the pages are no indexed.This is what we have to solve. 
We don't know if it is an issue of the server setting, DW settings or how to solve it. 
We have this problem since six months ago.

Thanks,

Jose.

Hi,

Thanks Nicolai for your suggestion. We will do it.

But please, to understand it well, is it correct that in googlebot lines that i indicate the server resturns 200 instead of 301?

Regards,

Jose

There are server level 301s and Application level 301s.

1. So when you request /productgroup/productName Dynamicweb (which is the application) will translate that into /Default.aspx?ID=123&Group=GROUP1 and execute that on the server that returns a 200 in the logs. That is what you see in the log, and that is fine.
    
2. When you access /Default.aspx?ID=123&Group=GROUP1 Dynamicweb (the application) will see that it is not the friendly, and send out a 301 to the browser or Googlebot and tell to use the /productgroup/productName - which then hits number one above...

I do not think you will see any 301s in the logfile that Dynamicweb is in charge of.

Hi Nicolai,

With the point one is well and i don´t have doubts.

About the second point, my doubts are related with googlebot. Not with the browser.

Our SEO auditors says that googlebot reads a lot of defautl.aspx urls as 200 status, because of that, then googlebot has no time to crawl the friendly URLs.

Here is a log fragment of tonight.

2021-10-28 00:00:13 10.21.142.46 GET /Default.aspx ID=2399&GroupID=GROUP974&ProductID=PROD7486 443 - 66.249.66.67 Mozilla/5.0+(compatible;+Googlebot/2.1;++http://www.google.com/bot.html) - 200 0 0 765

This is not correspond with that you say in the second point.

It will be helpful know if the googlebot lines of another dynamicweb solutions contains urls default with status 200, or the log entries are like this example:

2021-10-28 00:00:13 10.21.142.46 GET /https://mydoimain.com/myurlfriendly 443 - 66.249.66.67 Mozilla/5.0+(compatible;+Googlebot/2.1;++http://www.google.com/bot.html) - 200 0 0 765

By the way, the website contains a blog with a quite a lot post and only 5 post have been indexed (so the problem is not about the content)

Maybe the problem is related to some ISS server configuration.

Regards,

Jose.

Hi Jose,

I am not sure if this is helpful but I had some trouble with the links before with some SEO experts.

1. We have noticed that enabling "Use strict URL recognition" creates a secondary redirect between non-friendly to the friendly link .

2. We have noticed that the standard "Force SSL" redirection was also causing some issues with indexing and we have implemented the SSL redirect directly in IIS.

On my project, I had to fix both the above issues before my client's SEO experts were satisfied.

I am not sure if there are better solutions :)

I hope it helps.

Adrian

Hi Adrian,

Thanks for your answer.

Im going to disable the "Use strict URL recognition" check. I tested it in the staging eviroment but i notice that any wrong url redirects to the homepage. Not to the 404 page.

Do you know if disabling this button the website can still having a 404 page?

On the other hand, we have asked support to add the certificate to the iis server , not in the dw application.

Finally, it will be a great help know if your server googlebot log contains default urls with 200 status. 

Thanks again,

Jose

Hi Jose,

In my case, the 404 issue did not happen.

The SSL certificate configuration has nothing to do with the redirection. You can set an SSL redirect in IIS using Url Rewrite module from IIS.

In my case, the agency asked me to install a Chrome extension to check the redirections:

The project in question is this: https://magazin.agricover.ro/

I am not sure if the issue you experienced is replicable on my end or not but you are free to have a look at it.

Thank you,
Adrian