|
|
| |
This page exists to check how different search crawlers obey the noindex and nofollow content-values in a robots-metatag.
Note that this page bears a <meta name="robots" content="noindex,nofollow" /> in its html-head. This specific meta-line has been in this page from the very beginning of its existence.
The website as a whole does not have a robots.txt file because we want to differentiate its indexing on a page-by-page basis. Every single page of this site has a <meta name="robots" ... directive.
We have also made sure that this page validates with 0 errors with the W3C validator.w3.org HTML-validator service, and we have deliberately used an old-fashioned and extremely simple make-up style. Analysing the content of this page should be extremely simple for any web crawler.
Here we have a few keywords to use when checking the indexing/retrieving: limousine huckleberry abacadabra weezenhof
This site is a hobby project of Hendrik-Jan Thomassen < hjt@myDS420.info >