Developer forum

Forum » Swift » Using Robots.txt

Using Robots.txt

E-mail notifications
Daniel Hollmann
Reply
Posted on 09/12/2021 09:37:46

Using Roboxts.txt

Hi. In Swift Robots.txt, is a service under Swift Tools and a page, like any other page, setup with the ”Swift_PageClean.cshtml” template. But how do you use it correctly? I can insert elements like the swift text module, but this renders HTML as well as the text, making the file not readable for search enignes. I have tested this using this tool: https://technicalseo.com/tools/robots-txt/

I tried to change the content type as well, but the file was still not valid. I can’t change the item type in a standard solution as well, so how do I use it correctly?

Would be nice if that was added to the documentation here:

https://doc.dynamicweb.com/swift/seo/guides/basic-website-seo


Replies

 
Nicolai Pedersen
Posted on 09/12/2021 09:57:32
Reply
This post has been marked as an answer

Hi Daniel

I think that page is left over from a test we made. You can just delete it (or rename the URL) and Dynamicweb will fall back to the robots.txt created based on website settings.

BR Nicolai

Votes for this answer: 1
 
Daniel Hollmann
Posted on 09/12/2021 10:17:05
Reply

Thanks Nicolai. That worked as expected! 

 
Viktor Kvarnström
Posted on 20/02/2023 09:58:38
Reply

Hi Nicolai,

i have the same problem. Using dymaicweb 9.14.6. I have deleted the robots file under swift tools... And set correct values in website settings. when i go to my site then /robots.txt i still get this: 

User-agent: *
Disallow: /Files/Papirkurv/
User-agent: *
Disallow: /
Disallow: /*?cartcmd=*

 
Nicolai Pedersen Dynamicweb Employee
Nicolai Pedersen
Posted on 21/02/2023 13:31:14
Reply

That seems fine. This is coming from Dynamicweb.

The

Disallow: /

you see because you access the website from a domain under dynamicweb.dk

On website settings you can add whatever else you want.

 
Aki Ruuskanen
Aki Ruuskanen
Posted on 24/02/2023 12:28:10
Reply

Nicolai, I'm not sure I understand. 

This URL:

https://mindflower.se/robots.txt

Gives us:

User-agent: User-agent: *
Disallow: /Files/Papirkurv/
User-agent: *
Disallow: /
Disallow: /*?cartcmd=*

In the old Robots.txt setting we have:

User-agent: *
Disallow: /admin

Regards / Aki

 

 
Nicolai Pedersen Dynamicweb Employee
Nicolai Pedersen
Posted on 24/02/2023 15:34:40
Reply

It came from one of the disabled websites robots settings.

I deleted it...

 
Aki Ruuskanen
Aki Ruuskanen
Posted on 28/03/2023 15:50:12
Reply

Hi,

This one is still rendering the wrong robots. 

From the URL : https://mindflower.se/robots.txt

User-agent: *
Disallow: /Files/Papirkurv/

Disallow: /*?cartcmd=*

Robots.txt in DW

User-agent: *
Disallow: /admin

https://mindflower.se/sitemap.xml

I check the Area table. There is nothing in the AreaRobotsTxt in the other inactive websites. Where does this one come from?

Regards / Aki

 
Nicolai Pedersen Dynamicweb Employee
Nicolai Pedersen
Posted on 28/03/2023 17:00:36
Reply

Do you happen to have a real robots.txt sitting on disk in the root of the website?

 
Aki Ruuskanen
Aki Ruuskanen
Posted on 28/03/2023 17:56:17
Reply

I sure hope not :)  I'll check. 

/Aki

 
Aki Ruuskanen
Aki Ruuskanen
Posted on 29/03/2023 07:58:44
Reply

Hi. Checked the root. No robots there. 

 
Aki Ruuskanen
Aki Ruuskanen
Posted on 29/03/2023 09:15:49
Reply

I removed all the inactive sites so now we get a mix of the content in the settings for robot.txt and other DW stuff. 

In settings (with link to sitemap checked) :

User-agent: *
Disallow: /admin

In https://mindflower.se/robots.txt

Sitemap: https://mindflower.se/Sitemap.xml
User-agent: *
Disallow: /Files/Papirkurv/
User-agent: *
Disallow: /admin
Disallow: /*?cartcmd=*

 

You must be logged in to post in the forum