Robots.txt

Robots.txt for golbal website or specific folders

manual

• A Standard for Robot Exclusion
• Robots.txt Validator
• 網站管理員說明中心 - 我不想將要攔截的每個檔案都列出。我可以使用* 和 $ 模式媒合嗎？(EN)
• 給網站管理員的 Google 資訊

Google Information for Webmasters (last visited: February 10, 2006)

robots.txt validator

• New Robots.txt Syntax Checker: a validator for robots.txt files

meta tag for each webpage

<meta name="robots" content="noindex,nofollow" />

related:

• HTML Author's Guide to the Robots META tag
• Add meta tag to movabletype