以下只是截取了google蜘蛛日志,对360和bing(必应)也是可以的
百度的效果就不是很明显,不过提交后,效果还是有一点的,只是不明显罢了
robots.txt的设置:
User-agent: *
Sitemap:http://www.linuxyw.com/sitemap.xml
以下是我提取sitemap地图的一部分:
http://www.linuxyw.com/a/wenjianguanli/20130505/207.html
http://www.linuxyw.com/a/wenjianguanli/20130505/208.html
http://www.linuxyw.com/a/xunihua/20130424/128.html
http://www.linuxyw.com/a/xunihua/20130506/228.html
http://www.linuxyw.com/a/xunihua/20130506/229.html
http://www.linuxyw.com/a/xunihua/20130506/230.html
http://www.linuxyw.com/a/xunihua/20130506/231.html
http://www.linuxyw.com/a/xunihua/20130506/232.html
http://www.linuxyw.com/a/xunihua/20130506/233.html
http://www.linuxyw.com/a/xunihua/20130506/234.html
http://www.linuxyw.com/a/xunihua/20130506/235.html
http://www.linuxyw.com/a/yenei/20130329/10.html
http://www.linuxyw.com/a/yenei/20130329/9.html
http://www.linuxyw.com/a/yenei/20130506/211.html
http://www.linuxyw.com/a/yenei/20130506/212.html
http://www.linuxyw.com/a/yenei/20130506/213.html
http://www.linuxyw.com/a/yenei/20130506/214.html
http://www.linuxyw.com/a/yenei/20130506/215.html
http://www.linuxyw.com/a/yenei/20130506/222.html
http://www.linuxyw.com/a/yenei/20130507/242.html
http://www.linuxyw.com/a/yenei/20130507/243.html
http://www.linuxyw.com/a/yenei/20130507/249.html
http://www.linuxyw.com/a/yenei/20130508/254.html
http://www.linuxyw.com/a/yenei/20130508/256.html
http://www.linuxyw.com/a/yenei/20130508/257.html
http://www.linuxyw.com/a/yenei/20130508/258.html
http://www.linuxyw.com/a/yenei/20130508/259.html
http://www.linuxyw.com/a/yenei/20130508/261.html
http://www.linuxyw.com/a/yenei/20130508/262.html
http://www.linuxyw.com/a/yunweiguzhang/20130331/12.html
http://www.linuxyw.com/a/yunweiguzhang/20130403/18.html
http://www.linuxyw.com/a/yunweiguzhang/20130420/109.html
http://www.linuxyw.com/a/yunweiguzhang/20130504/151.html
http://www.linuxyw.com/a/yunweijingyan/20130329/8.html
http://www.linuxyw.com/a/yunweisixiang/20130329/7.html
http://www.linuxyw.com/a/yunweisixiang/20130424/131.html
http://www.linuxyw.com/a/wangzhanhuodong/2013/0418/28.html
http://www.linuxyw.com/a/wangzhanhuodong/2013/0418/29.html
http://www.linuxyw.com/a/wangzhanhuodong/2013/0507/253.html
http://www.linuxyw.com/a/wangzhanhuodong/2013/0508/255.html
http://www.linuxyw.com/a/ziyuang ... n/20130420/107.html
http://www.linuxyw.com/a/ziyuang ... u/20130507/251.html
在google提交sitemap地图后的蜘蛛爬取效果日志:
203.208.60.36 - - [09/May/2013:04:57:21 +0800] "GET /robots.txt HTTP/1.1" 200 217 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:04:57:21 +0800] "GET / HTTP/1.1" 200 10433 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.37 - - [09/May/2013:05:07:56 +0800] "GET /a/wangzhanhuodong/2013/0507/253.html HTTP/1.1" 200 14819 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.37 - - [09/May/2013:05:07:57 +0800] "GET /a/yenei/20130508/254.html HTTP/1.1" 200 9094 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.37 - - [09/May/2013:05:08:04 +0800] "GET /a/yenei/20130508/256.html HTTP/1.1" 200 8640 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.37 - - [09/May/2013:05:08:11 +0800] "GET /a/yenei/20130508/257.html HTTP/1.1" 200 24537 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.40 - - [09/May/2013:05:08:19 +0800] "GET /a/yenei/20130508/259.html HTTP/1.1" 200 15699 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.42 - - [09/May/2013:05:08:26 +0800] "GET /a/yenei/20130508/262.html HTTP/1.1" 200 9476 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:08:33 +0800] "GET /a/yenei/20130508/261.html HTTP/1.1" 200 14776 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.41 - - [09/May/2013:05:08:40 +0800] "GET /a/yenei/20130507/242.html HTTP/1.1" 200 10792 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.38 - - [09/May/2013:05:08:47 +0800] "GET /a/Mysql/20130428/141.html HTTP/1.1" 200 10215 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.37 - - [09/May/2013:05:08:54 +0800] "GET /a/yenei/20130508/258.html HTTP/1.1" 200 8962 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.38 - - [09/May/2013:05:09:07 +0800] "GET /a/tiaoyou/20130506/241.html HTTP/1.1" 200 16925 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.38 - - [09/May/2013:05:09:14 +0800] "GET /a/tiaoyou/20130506/237.html HTTP/1.1" 200 10165 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.37 - - [09/May/2013:05:09:21 +0800] "GET /a/xunihua/20130506/231.html HTTP/1.1" 200 11780 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.38 - - [09/May/2013:05:09:28 +0800] "GET /a/jiaoben/20130423/115.html HTTP/1.1" 200 42972 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.42 - - [09/May/2013:05:09:34 +0800] "GET /a/shujuku/20130506/225.html HTTP/1.1" 200 13725 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.38 - - [09/May/2013:05:09:41 +0800] "GET /a/xunihua/20130506/234.html HTTP/1.1" 200 10686 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:09:47 +0800] "GET /a/tiaoyou/20130429/147.html HTTP/1.1" 200 132549 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.40 - - [09/May/2013:05:09:54 +0800] "GET /a/xunihua/20130506/228.html HTTP/1.1" 200 10891 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.40 - - [09/May/2013:05:10:00 +0800] "GET /a/tiaoyou/20130506/227.html HTTP/1.1" 200 9021 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.41 - - [09/May/2013:05:10:07 +0800] "GET /a/xunihua/20130506/230.html HTTP/1.1" 200 11964 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:10:13 +0800] "GET /a/shujuku/20130506/217.html HTTP/1.1" 200 10406 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.38 - - [09/May/2013:05:10:19 +0800] "GET /a/tiaoyou/20130506/238.html HTTP/1.1" 200 12611 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.37 - - [09/May/2013:05:10:25 +0800] "GET /a/xunihua/20130506/235.html HTTP/1.1" 200 9498 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.40 - - [09/May/2013:05:10:31 +0800] "GET /a/shujuku/20130506/216.html HTTP/1.1" 200 15696 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.42 - - [09/May/2013:05:10:37 +0800] "GET /a/xunihua/20130506/229.html HTTP/1.1" 200 11135 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.41 - - [09/May/2013:05:10:43 +0800] "GET /a/jiaoben/20130423/112.html HTTP/1.1" 200 12754 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.37 - - [09/May/2013:05:10:49 +0800] "GET /a/xunihua/20130506/232.html HTTP/1.1" 200 12049 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.42 - - [09/May/2013:05:10:55 +0800] "GET /a/shujuku/20130506/224.html HTTP/1.1" 200 11270 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.41 - - [09/May/2013:05:11:01 +0800] "GET /a/tiaoyou/20130506/236.html HTTP/1.1" 200 11827 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.39 - - [09/May/2013:05:11:07 +0800] "GET /a/jiaoben/20130423/113.html HTTP/1.1" 200 8617 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:11:13 +0800] "GET /a/tiaoyou/20130506/240.html HTTP/1.1" 200 13916 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.42 - - [09/May/2013:05:11:18 +0800] "GET /a/xunihua/20130506/233.html HTTP/1.1" 200 9502 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.41 - - [09/May/2013:05:11:24 +0800] "GET /a/tiaoyou/20130506/239.html HTTP/1.1" 200 12368 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.40 - - [09/May/2013:05:11:30 +0800] "GET /a/qitafuwu/20130502/150.html HTTP/1.1" 200 9049 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.42 - - [09/May/2013:05:11:35 +0800] "GET /a/gongxiang/20130506/226.html HTTP/1.1" 200 9555 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.38 - - [09/May/2013:05:11:41 +0800] "GET /a/gongxiang/20130507/252.html HTTP/1.1" 200 8570 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.39 - - [09/May/2013:05:11:46 +0800] "GET /a/gongxiang/20130505/162.html HTTP/1.1" 200 8376 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.37 - - [09/May/2013:05:11:52 +0800] "GET /a/gongxiang/20130505/161.html HTTP/1.1" 200 9243 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:11:57 +0800] "GET /a/fuzaijunheng/20130504/156.html HTTP/1.1" 200 61409 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.41 - - [09/May/2013:05:12:02 +0800] "GET /a/fuzaijunheng/20130504/155.html HTTP/1.1" 200 147140 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.41 - - [09/May/2013:05:12:07 +0800] "GET /a/fuzaijunheng/20130504/153.html HTTP/1.1" 200 41982 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.37 - - [09/May/2013:05:12:12 +0800] "GET /a/fuzaijunheng/20130509/264.html HTTP/1.1" 200 202106 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.37 - - [09/May/2013:05:12:17 +0800] "GET /a/fuzaijunheng/20130504/154.html HTTP/1.1" 200 99367 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:12:28 +0800] "GET /a/wenjianguanli/20130505/187.html HTTP/1.1" 200 8560 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.38 - - [09/May/2013:05:12:30 +0800] "GET /a/wenjianguanli/20130505/192.html HTTP/1.1" 200 8744 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.42 - - [09/May/2013:05:12:38 +0800] "GET /a/wenjianguanli/20130505/202.html HTTP/1.1" 200 8731 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.40 - - [09/May/2013:05:12:43 +0800] "GET /a/wenjianguanli/20130505/176.html HTTP/1.1" 200 8688 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.40 - - [09/May/2013:05:12:48 +0800] "GET /a/wenjianguanli/20130505/184.html HTTP/1.1" 200 9377 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.38 - - [09/May/2013:05:12:56 +0800] "GET /a/wenjianguanli/20130505/195.html HTTP/1.1" 200 8699 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.38 - - [09/May/2013:05:12:58 +0800] "GET /a/jinglizhitian/20130504/158.html HTTP/1.1" 200 9091 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.40 - - [09/May/2013:05:13:03 +0800] "GET /a/wenjianguanli/20130505/189.html HTTP/1.1" 200 8541 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.40 - - [09/May/2013:05:13:08 +0800] "GET /a/wenjianguanli/20130505/179.html HTTP/1.1" 200 8838 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.41 - - [09/May/2013:05:13:12 +0800] "GET /a/wenjianguanli/20130427/132.html HTTP/1.1" 200 10554 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.40 - - [09/May/2013:05:13:17 +0800] "GET /a/wenjianguanli/20130428/134.html HTTP/1.1" 200 10489 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:13:22 +0800] "GET /a/wenjianguanli/20130505/193.html HTTP/1.1" 200 10500 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.41 - - [09/May/2013:05:13:26 +0800] "GET /a/wenjianguanli/20130505/168.html HTTP/1.1" 200 8615 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.37 - - [09/May/2013:05:13:31 +0800] "GET /a/wenjianguanli/20130505/204.html HTTP/1.1" 200 9261 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:13:36 +0800] "GET /a/wenjianguanli/20130505/188.html HTTP/1.1" 200 8824 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.38 - - [09/May/2013:05:13:40 +0800] "GET /a/wenjianguanli/20130429/148.html HTTP/1.1" 200 11091 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.37 - - [09/May/2013:05:13:45 +0800] "GET /a/wenjianguanli/20130505/198.html HTTP/1.1" 200 8741 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.39 - - [09/May/2013:05:13:49 +0800] "GET /a/wenjianguanli/20130505/181.html HTTP/1.1" 200 8673 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.39 - - [09/May/2013:05:13:53 +0800] "GET /a/wenjianguanli/20130505/197.html HTTP/1.1" 200 9190 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.37 - - [09/May/2013:05:13:58 +0800] "GET /a/wenjianguanli/20130505/203.html HTTP/1.1" 200 8801 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.38 - - [09/May/2013:05:14:02 +0800] "GET /a/wenjianguanli/20130505/207.html HTTP/1.1" 200 8637 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.42 - - [09/May/2013:05:14:06 +0800] "GET /a/wenjianguanli/20130505/208.html HTTP/1.1" 200 8771 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:14:11 +0800] "GET /a/wenjianguanli/20130505/180.html HTTP/1.1" 200 9062 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:14:15 +0800] "GET /a/wenjianguanli/20130505/169.html HTTP/1.1" 200 8761 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.37 - - [09/May/2013:05:14:22 +0800] "GET /a/wenjianguanli/20130505/185.html HTTP/1.1" 200 8580 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.37 - - [09/May/2013:05:14:23 +0800] "GET /a/wenjianguanli/20130505/205.html HTTP/1.1" 200 9181 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:14:27 +0800] "GET /a/wenjianguanli/20130505/190.html HTTP/1.1" 200 8692 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:14:31 +0800] "GET /a/wenjianguanli/20130505/191.html HTTP/1.1" 200 9143 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:14:35 +0800] "GET /a/wenjianguanli/20130505/199.html HTTP/1.1" 200 8774 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:14:39 +0800] "GET /a/wenjianguanli/20130429/149.html HTTP/1.1" 200 10101 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:14:47 +0800] "GET /a/wenjianguanli/20130427/133.html HTTP/1.1" 200 10117 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:14:51 +0800] "GET /a/wenjianguanli/20130505/186.html HTTP/1.1" 200 8580 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:14:55 +0800] "GET /a/yunweiguzhang/20130504/151.html HTTP/1.1" 200 8483 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:14:59 +0800] "GET /a/wenjianguanli/20130505/175.html HTTP/1.1" 200 9286 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:15:03 +0800] "GET /a/wenjianguanli/20130505/194.html HTTP/1.1" 200 8746 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:15:07 +0800] "GET /a/wenjianguanli/20130505/178.html HTTP/1.1" 200 8786 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:15:10 +0800] "GET /a/jinglizhitian/20130505/164.html HTTP/1.1" 200 282884 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:15:14 +0800] "GET /a/wenjianguanli/20130505/201.html HTTP/1.1" 200 8649 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:15:18 +0800] "GET /a/wenjianguanli/20130505/170.html HTTP/1.1" 200 9266 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:15:21 +0800] "GET /a/wenjianguanli/20130505/173.html HTTP/1.1" 200 10428 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:15:25 +0800] "GET /a/wenjianguanli/20130505/182.html HTTP/1.1" 200 8717 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:15:29 +0800] "GET /a/wenjianguanli/20130505/171.html HTTP/1.1" 200 8903 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:15:32 +0800] "GET /a/wenjianguanli/20130505/172.html HTTP/1.1" 200 8730 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:15:36 +0800] "GET /a/wenjianguanli/20130505/177.html HTTP/1.1" 200 10278 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:15:39 +0800] "GET /a/wenjianguanli/20130505/165.html HTTP/1.1" 200 9012 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:15:43 +0800] "GET /a/wenjianguanli/20130505/200.html HTTP/1.1" 200 9622 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:15:46 +0800] "GET /a/yunweisixiang/20130424/131.html HTTP/1.1" 200 11818 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.213 - - [09/May/2013:05:15:49 +0800] "GET /a/yenei/20130508/261.html HTTP/1.1" 200 14776 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:15:53 +0800] "GET /a/wenjianguanli/20130505/167.html HTTP/1.1" 200 10014 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:15:56 +0800] "GET /a/wenjianguanli/20130505/174.html HTTP/1.1" 200 8842 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:15:59 +0800] "GET /a/wenjianguanli/20130505/206.html HTTP/1.1" 200 8519 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:16:03 +0800] "GET /a/wenjianguanli/20130505/196.html HTTP/1.1" 200 8687 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:16:06 +0800] "GET /a/wenjianguanli/20130505/166.html HTTP/1.1" 200 9015 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:16:12 +0800] "GET /a/wangzhanhuodong/2013/0508/255.html HTTP/1.1" 200 8495 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:16:15 +0800] "GET /a/ziyuangongxiang/yunweigongju/20130507/251.html HTTP/1.1" 200 8083 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:16:18 +0800] "GET /a/Shell/Shell/2013/0428/140.html HTTP/1.1" 200 11016 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:16:22 +0800] "GET /a/Shell/Shell/2013/0428/136.html HTTP/1.1" 200 8918 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:16:25 +0800] "GET /a/Shell/Shell/2013/0504/157.html HTTP/1.1" 200 8580 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.217 - - [09/May/2013:05:16:28 +0800] "GET /a/yenei/20130508/259.html HTTP/1.1" 200 15699 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:16:31 +0800] "GET /a/yunweijingyan/xitongguanli/2013/0423/123.html HTTP/1.1" 200 8625 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:16:34 +0800] "GET /a/yunweijingyan/xitongguanli/2013/0424/129.html HTTP/1.1" 200 14711 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:16:37 +0800] "GET /a/yunweijingyan/xitongguanli/2013/0423/122.html HTTP/1.1" 200 8647 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:16:40 +0800] "GET /a/yunweijingyan/xitongguanli/2013/0505/163.html HTTP/1.1" 200 396149 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:16:43 +0800] "GET /a/bushu/jiankongbaojing/2013/0509/263.html HTTP/1.1" 200 128907 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:16:45 +0800] "GET /a/yenei/20130506/222.html HTTP/1.1" 200 8511 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:16:48 +0800] "GET /a/yenei/20130506/215.html HTTP/1.1" 200 8848 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:16:51 +0800] "GET /a/yenei/20130506/214.html HTTP/1.1" 200 8386 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:16:54 +0800] "GET /plus/feedback.php?aid=261 HTTP/1.1" 200 4379 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.36 - - [09/May/2013:05:16:57 +0800] "GET /plus/feedback.php?aid=262 HTTP/1.1" 200 4406 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.216 - - [09/May/2013:05:17:00 +0800] "GET /a/yenei/20130508/262.html HTTP/1.1" 200 9476 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.41 - - [09/May/2013:05:17:03 +0800] "GET /a/yenei/20130507/243.html HTTP/1.1" 200 10062 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.41 - - [09/May/2013:05:17:05 +0800] "GET /a/yenei/20130506/223.html HTTP/1.1" 200 9610 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.41 - - [09/May/2013:05:17:08 +0800] "GET /plus/feedback.php?aid=259 HTTP/1.1" 200 4386 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.41 - - [09/May/2013:05:17:11 +0800] "GET /a/yenei/20130507/249.html HTTP/1.1" 200 8933 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.41 - - [09/May/2013:05:17:13 +0800] "GET /plus/feedback.php?aid=264 HTTP/1.1" 200 4392 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.41 - - [09/May/2013:05:17:16 +0800] "GET /a/yenei/20130508/260.html HTTP/1.1" 404 986 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.41 - - [09/May/2013:05:17:19 +0800] "GET /a/yenei/20130506/211.html HTTP/1.1" 200 9058 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.41 - - [09/May/2013:05:17:21 +0800] "GET /a/tiaoyou/20130428/137.html HTTP/1.1" 200 50268 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.41 - - [09/May/2013:05:17:24 +0800] "GET /a/xunihua/20130424/128.html HTTP/1.1" 200 9637 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.41 - - [09/May/2013:05:17:26 +0800] "GET /a/tiaoyou/20130428/138.html HTTP/1.1" 200 10042 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.41 - - [09/May/2013:05:17:29 +0800] "GET /a/tiaoyou/20130428/135.html HTTP/1.1" 200 26460 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.41 - - [09/May/2013:05:17:31 +0800] "GET /a/Mysql/20130428/141_2.html HTTP/1.1" 200 8437 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.41 - - [09/May/2013:05:17:34 +0800] "GET /a/qitafuwu/20130423/126.html HTTP/1.1" 200 12411 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.41 - - [09/May/2013:05:17:39 +0800] "GET /a/qitafuwu/20130423/124.html HTTP/1.1" 200 9537 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.41 - - [09/May/2013:05:17:41 +0800] "GET /a/fuzaijunheng/20130423/116.html HTTP/1.1" 200 10155 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.41 - - [09/May/2013:05:17:44 +0800] "GET /a/fuzaijunheng/20130423/117.html HTTP/1.1" 200 10601 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.41 - - [09/May/2013:05:17:46 +0800] "GET /a/fuzaijunheng/20130423/118.html HTTP/1.1" 200 10156 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.39 - - [09/May/2013:05:17:51 +0800] "GET /a/fuzaijunheng/20130423/125.html HTTP/1.1" 200 10156 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.39 - - [09/May/2013:05:17:53 +0800] "GET /a/fuzaijunheng/20130429/146.html HTTP/1.1" 200 57705 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.39 - - [09/May/2013:05:17:56 +0800] "GET /a/fuzaijunheng/20130504/153_3.html HTTP/1.1" 200 29166 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.211 - - [09/May/2013:05:17:49 +0800] "GET /a/fuzaijunheng/20130509/264.html HTTP/1.1" 200 202106 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.39 - - [09/May/2013:05:17:58 +0800] "GET /a/fuzaijunheng/20130504/153_2.html HTTP/1.1" 200 23237 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.39 - - [09/May/2013:05:18:00 +0800] "GET /a/fuzaijunheng/20130504/153_6.html HTTP/1.1" 200 8491 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
本文来自linux系统运维:http://www.linuxyw.com/a/yenei/20130509/271.html