This question is not answered. Helpful answers available: 2. Correct answers available: 1.


Permlink Replies: 2 - Pages: 1 - Last Post: 04-Jan-2019 13:29 Last Post By: Laza
Lobata

Posts: 3
Registered: 30-Nov-2016
Option to tell a web robot that it should not visit any pages on the site?
Posted: 04-Jan-2019 08:31
 
  Click to reply to this thread Reply
I noticed in the index.html a line:

<meta name="robots" content="index,follow">

I actually don't want my site indexed by web robots - just want share with relatives and close friends, not the rest of the world. Is there a Turtle setting that tells the indexing programs not to index the directory and all its subdirectories? I could edit index.html and the robots.txt file directly to indicate that I don't want the robot to not visit any pages where it is found or any contained directories. In this age of privacy slipping away, I would think this would be a useful option to let the user be somewhat in control of their privacy.

I know malware robots can and often do ignore /robots.txt and the line in index.html, but at least the reputable web crawlers would honor the directives.

Thanks
RobM

Posts: 3,105
Registered: 4-Aug-2006
Re: Option to tell a web robot that it should not visit any pages on the site?
Posted: 04-Jan-2019 09:50   in response to: Lobata in response to: Lobata
 
  Click to reply to this thread Reply
Tiger has an option for this, but Turtle does not. See this thread for further information on stopping robots.
Laza

Posts: 2,222
Registered: 6-Sep-2005
Re: Option to tell a web robot that it should not visit any pages on the site?
Posted: 04-Jan-2019 13:29   in response to: Lobata in response to: Lobata
 
  Click to reply to this thread Reply
What if you change the line in question (Ctrl-Shift-E -> page-header.inc) to this:
<meta name="robots" content="noindex, nofollow">

(Unfortunately you'll have to redo this change after a skin update, but it's not too hard.)
Legend
Forum admins
Helpful Answer
Correct Answer

Point your RSS reader here for a feed of the latest messages in all forums