Robot Exclusion Protocol

page cannot be crawled
or displayed due to robots
from robot system

disallow modules
disallow add-on modules
disallow hopper

exclude search results
disallow cache disallow
texts disallow help

disallow starting
points disallow articles
disallow index

disallow stress test
disallow stress disallow
agent disallow

copyright the next
line is a spam bot trap for
the doc tree logs you

should really change this
to something else disallow
all our manual

results all our art
disallow all new pages
disallow all old

user ideas
disallow Latin version
but allow robots

[Via a Wikipedia link on “Eleusinian Mysteries” page]

http://www.perseus.tufts.edu/robots.txt

 

Advertisements

Tags: , , , ,