14,974
edits
Robots.txt: Difference between revisions
Jump to navigation
Jump to search
no edit summary
No edit summary |
No edit summary |
||
| Line 1: | Line 1: | ||
== Robots.txt for golbal website or specific folders == | == Robots.txt for golbal website or specific folders == | ||
manual | |||
* [http://www.robotstxt.org/wc/norobots.html A Standard for Robot Exclusion] | * [http://www.robotstxt.org/wc/norobots.html A Standard for Robot Exclusion] | ||
* [http://www.searchengineworld.com/cgi-bin/robotcheck.cgi Robots.txt Validator] | * [http://www.searchengineworld.com/cgi-bin/robotcheck.cgi Robots.txt Validator] | ||
| Line 5: | Line 6: | ||
* [http://www.google.com.tw/intl/zh-TW/webmasters/remove.html#outdated 給網站管理員的 Google 資訊] | * [http://www.google.com.tw/intl/zh-TW/webmasters/remove.html#outdated 給網站管理員的 Google 資訊] | ||
: [http://www.google.com/webmasters/remove.html Google Information for Webmasters] (last visited: February 10, 2006) | : [http://www.google.com/webmasters/remove.html Google Information for Webmasters] (last visited: February 10, 2006) | ||
robots.txt validator | |||
* [http://tool.motoricerca.info/robots-checker.phtml New Robots.txt Syntax Checker: a validator for robots.txt files] | |||
== meta tag for each webpage == | == meta tag for each webpage == | ||