Hi,
The Google robot is not correctly crawling our website. The Google search "site:mysite.com" displays all the PDFs of the website but only five pages of about 5000.
According the post, we are only using robots.txt. We don't use robots meta tag in the master template, neither the X-Robots-Tag header in web.config.
The robots.txt is:
Sitemap: https://mysite.com/Sitemap.xml
User-agent: *
Disallow: /Files/Papirkurv/
Disallow: /*?Promotion=True
Disallow: *.pdf$
Disallow: /*?cartcmd=*
Sitemap.xml contain the URLs to all categories, products, etc.
What we may do to list all the site pages in Google searches?
Regards,
Tomas