Robots 限制搜索引擎蜘蛛抓取哪些文件

2019独角兽企业重金招聘Python工程师标准>>>

编辑一份 Robots 文件，另存为robots.txt，存在到服务器的根目录里

Java代码  

1. User-agent: *   
2. Disallow: /plus/ad_js.php  
3. Disallow: /plus/advancedsearch.php  
4. Disallow: /plus/car.php  
5. Disallow: /plus/carbuyaction.php  
6. Disallow: /plus/shops_buyaction.php  
7. Disallow: /plus/erraddsave.php  
8. Disallow: /plus/posttocar.php  
9. Disallow: /plus/disdls.php  
10. Disallow: /plus/feedback_js.php  
11. Disallow: /plus/mytag_js.php  
12. Disallow: /plus/rss.php  
13. Disallow: /plus/search.php  
14. Disallow: /plus/recommend.php  
15. Disallow: /plus/stow.php  
16. Disallow: /plus/count.php  
17. Disallow: /include  
18. Disallow: /templets  

 

效果图：

 

打开百度站长链接：http://zhanzhang.baidu.com/robots/index

输入域名或子域名

 

 

robots.txt文件用法举例

转载于:https://my.oschina.net/pvpCC9IFwqz4/blog/505174