收藏两个robots.txt文件。
wordpress2.3的robots文件示例：
#
# robots.txt for WordPress
# Version 2.3
#
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-includes/
引用自：http://www.grick.net/archives/92
下面是phpcms2007的robots示例，自己写的，错了管换，仅供参考：
User-agent: *
Disallow: /admin/
Disallow: /ads/
Disallow: /announce/
Disallow: /article/
Disallow: /ask/
Disallow: /bill/
Disallow: /cert/
Disallow: /corpandresize/
Disallow: /data/
Disallow: /digg/
Disallow: /down/
Disallow: /editor/
Disallow: /error_report/
Disallow: /fckeditor/
Disallow: /formguide/
Disallow: /guestbook/
Disallow: /house/
Disallow: /images/
Disallow: /include/
Disallow: /info/
Disallow: /languages/
Disallow: /link/
Disallow: /mail/
Disallow: /member/
Disallow: /message/
Disallow: /module/
Disallow: /movie/
Disallow: /mypage/
Disallow: /news/
Disallow: /page/
Disallow: /pay/
Disallow: /picture/
Disallow: /product/
Disallow: /spider/
Disallow: /stat/
Disallow: /templates/
Disallow: /union/
Disallow: /uploadfile/
Disallow: /vote/
Disallow: /wenba/
Disallow: /yp/