闲聊Robots协议
其实，我了解搜索引擎方面的知识是比较晚的~~~对robots协议还是来自2012年的“3B大战“也就是360和百度之间的一场争论！！
360呢，在2012年推出了自己的一款搜索引擎”360搜索“，并在发布没多久就一跃成为中国第二大搜索引擎，超越搜狗，仅次于百度！！！
但是呢，百度就指出自己的Robots文本中已设定不允许360爬虫进入，而360的爬虫依然对“百度知道”、“百度百科”等百度网站内容进行抓取。
违反了国际上”Robots协议“。一下是关于这方面大家可以查看:http://baike.baidu.com/view/9230864.htm 至此呢，我才了解到了”Robots协议“
百度一下，了解到”
robots协议（也称为爬虫协议、爬虫规则、机器人协议等）也就是 robots.txt，网站通过robots协议告诉搜索引擎哪些页面可以抓取，哪些页面不能抓取。Robots协议是网站国际 互联网界通行的道德规范，其目的是保护网站数据和敏感信息、确保用户个人信息和隐私不被侵犯。因其不是命令，故需要搜索引擎自觉遵守。一些 病毒如 malware（马威尔病毒）经常通过忽略robots协议的方式，获取网站后台数据和个人信息。
robots.txt文件是一个 文本文件，使用 任何一个常见的文本编辑器，比如 Windows系统自带的Notepad，就可以创建和编辑它。 robots.txt是一个协议，而不是一个 命令。robots.txt是 搜索引擎中访问网站的时候要查看的第一个文件。robots.txt文件告诉 蜘蛛程序在服务器上什么文件是可以被查看的。
当一个搜索蜘蛛访问一个 站点时，它会首先 检查该站点 根目录下是否存在robots.txt，如果存在，搜索机器人就会按照该文件中的内容来确定访问的范围；如果该文件不存在，所有的搜索蜘蛛将能够访问网站上所有没有被口令保护的页面。百度官方建议，仅当您的网站包含不希望被 搜索引擎收录的内容时，才需要使用robots.txt文件。如果您希望搜索引擎收录网站上所有内容，请勿建立robots.txt文件。
如果将 网站视为酒店里的一个房间，robots.txt就是主人在房间门口悬挂的“ 请勿打扰”或“欢迎打扫”的提示牌。这个文件告诉来访的搜索引擎哪些房间可以进入和参观，哪些房间因为存放贵重物品，或可能涉及住户及访客的隐私而不对搜索引擎开放。但robots.txt不是 命令，也不是 防火墙，如同守门人无法阻止窃贼等恶意闯入者。
以上是来自百度的解释！！Robots仅仅是一种协议而已！如果你不遵循它，那也没办法！只能通过打官司解决了！！
我们来看一下各大网站的Robots.txt吧~~~
www.baidu.com/robots.txt
User-agent: Baiduspider Disallow: /w? User-agent: Googlebot Disallow: /update Disallow: /history Disallow: /usercard Disallow: /usercenter User-agent: MSNBot Allow: / User-agent: Baiduspider-image Disallow: /w? User-agent: YoudaoBot Allow: / User-agent: Sogou web spider Disallow: /update Disallow: /history Disallow: /usercard Disallow: /usercenter User-agent: Sogou inst spider Disallow: /update Disallow: /history Disallow: /usercard Disallow: /usercenter User-agent: Sogou spider2 Disallow: /update Disallow: /history Disallow: /usercard Disallow: /usercenter User-agent: Sogou blog Disallow: /update Disallow: /history Disallow: /usercard Disallow: /usercenter User-agent: Sogou News Spider Disallow: /update Disallow: /history Disallow: /usercard Disallow: /usercenter User-agent: Sogou Orion spider Disallow: /update Disallow: /history Disallow: /usercard Disallow: /usercenter User-agent: JikeSpider Allow: / User-agent: Sosospider Allow: / User-agent: YYspider Allow: / User-agent: PangusoSpider Allow: / User-agent: yisouspider Allow: / User-agent: EasouSpider Allow: / User-agent: * Disallow: /
上面是什么意思就不用多说了吧、？User-agent后面跟的也就是网络爬虫的名字了！！！
正如百度所说，确实没允许360spider进行爬取！！
www.google.com/robots.txt
User-agent: * Disallow: /search Disallow: /sdch Disallow: /groups Disallow: /images Disallow: /catalogs Allow: /catalogs/about Allow: /catalogs/p? Disallow: /catalogues Disallow: /news Allow: /news/directory Disallow: /nwshp Disallow: /setnewsprefs? Disallow: /index.html? Disallow: /? Allow: /?hl= Disallow: /?hl=*& Disallow: /addurl/image? Disallow: /pagead/ Disallow: /relpage/ Disallow: /relcontent Disallow: /imgres Disallow: /imglanding Disallow: /sbd Disallow: /keyword/ Disallow: /u/ Disallow: /univ/ Disallow: /cobrand Disallow: /custom Disallow: /advanced_group_search Disallow: /googlesite Disallow: /preferences Disallow: /setprefs Disallow: /swr Disallow: /url Disallow: /default Disallow: /m? Disallow: /m/ Disallow: /wml? Disallow: /wml/? Disallow: /wml/search? Disallow: /xhtml? Disallow: /xhtml/? Disallow: /xhtml/search? Disallow: /xml? Disallow: /imode? Disallow: /imode/? Disallow: /imode/search? Disallow: /jsky? Disallow: /jsky/? Disallow: /jsky/search? Disallow: /pda? Disallow: /pda/? Disallow: /pda/search? Disallow: /sprint_xhtml Disallow: /sprint_wml Disallow: /pqa Disallow: /palm Disallow: /gwt/ Disallow: /purchases Disallow: /hws Disallow: /bsd? Disallow: /linux? Disallow: /mac? Disallow: /microsoft? Disallow: /unclesam? Disallow: /answers/search?q= Disallow: /local? Disallow: /local_url Disallow: /shihui? Disallow: /shihui/ Disallow: /froogle? Disallow: /products? Disallow: /products/ Disallow: /froogle_ Disallow: /product_ Disallow: /products_ Disallow: /products; Disallow: /print Disallow: /books/ Disallow: /bkshp?*q=* Disallow: /books?*q=* Disallow: /books?*output=* Disallow: /books?*pg=* Disallow: /books?*jtp=* Disallow: /books?*jscmd=* Disallow: /books?*buy=* Disallow: /books?*zoom=* Allow: /books?*q=related:* Allow: /books?*q=editions:* Allow: /books?*q=subject:* Allow: /books/about Allow: /booksrightsholders Allow: /books?*zoom=1* Allow: /books?*zoom=5* Disallow: /ebooks/ Disallow: /ebooks?*q=* Disallow: /ebooks?*output=* Disallow: /ebooks?*pg=* Disallow: /ebooks?*jscmd=* Disallow: /ebooks?*buy=* Disallow: /ebooks?*zoom=* Allow: /ebooks?*q=related:* Allow: /ebooks?*q=editions:* Allow: /ebooks?*q=subject:* Allow: /ebooks?*zoom=1* Allow: /ebooks?*zoom=5* Disallow: /patents? Disallow: /patents/download/ Disallow: /patents/pdf/ Disallow: /patents/related/ Disallow: /scholar Disallow: /citations? Allow: /citations?user= Allow: /citations?view_op=new_profile Allow: /citations?view_op=top_venues Disallow: /complete Disallow: /s? Disallow: /sponsoredlinks Disallow: /videosearch? Disallow: /videopreview? Disallow: /videoprograminfo? Allow: /maps?hq=http://maps.google.com/help/maps/directions/biking/mapleft.kml&ie=UTF8&ll=37.687624,-122.319717&spn=0.346132,0.727158&z=11&lci=bike&dirflg=b&f=d Allow: /maps/api/js? Disallow: /maps? Disallow: /mapstt? Disallow: /mapslt? Disallow: /maps/stk/ Disallow: /maps/br? Disallow: /mapabcpoi? Disallow: /maphp? Disallow: /mapprint? Disallow: /maps/api/js/ Disallow: /maps/api/staticmap? Disallow: /mld? Disallow: /staticmap? Disallow: /places/ Allow: /places/$ Disallow: /maps/preview Disallow: /maps/place Disallow: /help/maps/streetview/partners/welcome/ Disallow: /help/maps/indoormaps/partners/ Disallow: /lochp? Disallow: /center Disallow: /ie? Disallow: /sms/demo? Disallow: /katrina? Disallow: /blogsearch? Disallow: /blogsearch/ Disallow: /blogsearch_feeds Disallow: /advanced_blog_search Disallow: /uds/ Disallow: /chart? Disallow: /transit? Disallow: /mbd? Disallow: /extern_js/ Disallow: /xjs/ Disallow: /calendar/feeds/ Disallow: /calendar/ical/ Disallow: /cl2/feeds/ Disallow: /cl2/ical/ Disallow: /coop/directory Disallow: /coop/manage Disallow: /trends? Disallow: /trends/music? Disallow: /trends/hottrends? Disallow: /trends/viz? Disallow: /notebook/search? Disallow: /musica Disallow: /musicad Disallow: /musicas Disallow: /musicl Disallow: /musics Disallow: /musicsearch Disallow: /musicsp Disallow: /musiclp Disallow: /browsersync Disallow: /call Disallow: /archivesearch? Disallow: /archivesearch/url Disallow: /archivesearch/advanced_search Disallow: /base/reportbadoffer Disallow: /urchin_test/ Disallow: /movies? Disallow: /codesearch? Disallow: /codesearch/feeds/search? Disallow: /wapsearch? Disallow: /safebrowsing Allow: /safebrowsing/diagnostic Allow: /safebrowsing/report_badware/ Allow: /safebrowsing/report_error/ Allow: /safebrowsing/report_phish/ Disallow: /reviews/search? Disallow: /orkut/albums Allow: /jsapi Disallow: /views? Disallow: /c/ Disallow: /cbk Allow: /cbk?output=tile&cb_client=maps_sv Disallow: /recharge/dashboard/car Disallow: /recharge/dashboard/static/ Disallow: /translate_a/ Disallow: /translate_c Disallow: /translate_f Disallow: /translate_static/ Disallow: /translate_suggestion Disallow: /profiles/me Allow: /profiles Disallow: /s2/profiles/me Allow: /s2/profiles Allow: /s2/photos Allow: /s2/static Disallow: /s2 Allow: /s2/search/social Disallow: /transconsole/portal/ Disallow: /gcc/ Disallow: /aclk Disallow: /cse? Disallow: /cse/home Disallow: /cse/panel Disallow: /cse/manage Disallow: /tbproxy/ Disallow: /imesync/ Disallow: /shenghuo/search? Disallow: /support/forum/search? Disallow: /reviews/polls/ Disallow: /hosted/images/ Disallow: /ppob/? Disallow: /ppob? Disallow: /ig/add? Disallow: /adwordsresellers Disallow: /accounts/o8 Allow: /accounts/o8/id Disallow: /topicsearch?q= Disallow: /xfx7/ Disallow: /squared/api Disallow: /squared/search Disallow: /squared/table Disallow: /toolkit/ Allow: /toolkit/*.html Disallow: /globalmarketfinder/ Allow: /globalmarketfinder/*.html Disallow: /qnasearch? Disallow: /app/updates Disallow: /sidewiki/entry/ Disallow: /quality_form? Disallow: /labs/popgadget/search Disallow: /buzz/post Disallow: /compressiontest/ Disallow: /analytics/reporting/ Disallow: /analytics/admin/ Disallow: /analytics/web/ Disallow: /analytics/feeds/ Disallow: /analytics/settings/ Disallow: /alerts/ Disallow: /ads/search Disallow: /phone/compare/? Allow: /alerts/manage Allow: /alerts/remove Disallow: /travel/clk Disallow: /hotelfinder/rpc Disallow: /hotels/rpc Disallow: /flights/rpc Disallow: /commercesearch/services/ Disallow: /evaluation/ Disallow: /chrome/browser/mobile/tour Disallow: /compare/*/apply* Disallow: /forms/perks/ Disallow: /baraza/*/search Disallow: /baraza/*/report Disallow: /shopping/suppliers/search Disallow: /ct/ Disallow: /edu/cs4hs/ Disallow: /trustedstores/s/ Disallow: /trustedstores/tm2 Disallow: /trustedstores/verify Disallow: /adwords/proposal Disallow: /shopping/product/ Disallow: /shopping/seller Disallow: /shopping/reviewer Sitemap: http://www.gstatic.com/culturalinstitute/sitemaps/www_google_com_culturalinstitute/sitemap-index.xml Sitemap: http://www.google.com/hostednews/sitemap_index.xml Sitemap: http://www.google.com/sitemaps_webmasters.xml Sitemap: http://www.gstatic.com/sitemaps/websearch_hreflang/sitemap_index.xml Sitemap: http://www.google.com/ventures/sitemap_ventures.xml Sitemap: http://www.gstatic.com/dictionary/static/sitemaps/sitemap_index.xml Sitemap: http://www.gstatic.com/earth/gallery/sitemaps/sitemap.xml Sitemap: http://www.gstatic.com/s2/sitemaps/profiles-sitemap.xml Sitemap: http://www.gstatic.com/trends/websites/sitemaps/sitemapindex.xml
诶？为什么多出了Sitemap这个元素呢？
前面说过爬虫会通过网页内部的链接发现新的网页。但是如果没有连接指向的网页怎么办?或者用户输入条件生成的动态网页怎么办?能否让网站管理员通知搜索引擎他们网站上有哪些可供抓取的网页?这就是sitemap，最简单的 Sitepmap 形式就是 XML 文件，在其中列出网站中的网址以及关于每个网址的其他数据(上次更新的时间、更改的频率以及相对于网站上其他网址的重要程度等等)，利用这些信息搜索引擎可以更加智能地抓取网站内容。
sitemap是另一个话题，足够开一篇新的文章聊的，这里就不展开了，有兴趣的同学可以参考sitemap
新的问题来了，爬虫怎么知道这个网站有没有提供sitemap文件，或者说网站管理员生成了sitemap，(可能是多个文件)，爬虫怎么知道放在哪里呢?
由于robots.txt的位置是固定的，于是大家就想到了把sitemap的位置信息放在robots.txt里。这就成为robots.txt里的新成员了。
以上是跟的xml文件形式，大家可以打开看一下~~~其实还可以后跟txt格式的~~如：
大家打开看看！！！！！！！当然还可以是压缩包的形式哦~~我们看一下亚马逊的
http://www.amazon.cn/robots.txt
User-agent: * Disallow: /buycar Disallow: /cart Disallow: /checkout Disallow: /class Disallow: /com Disallow: /common Disallow: /css Disallow: /dll Disallow: /doc Disallow: /dp/e-mail-friend/ Disallow: /dp/manual-submit/ Disallow: /dp/product-availability/ Disallow: /dp/rate-this-item/ Disallow: /dp/shipping/ Disallow: /dp/twister-update/ Disallow: /gp/aws/ssop Disallow: /gp/cart Disallow: /gp/css/homepage.html Disallow: /gp/customer-reviews/common/du Disallow: /gp/flex Disallow: /gp/gfix Disallow: /gp/history Disallow: /gp/item-dispatch Disallow: /gp/music/clipserve Disallow: /gp/music/wma-pop-up Disallow: /gp/offer-listing Disallow: /gp/product/e-mail-friend Disallow: /gp/product/product-availability Disallow: /gp/product/rate-this-item Disallow: /gp/recsradio Disallow: /gp/slredirect Disallow: /gp/twitter/ Disallow: /gp/vote Disallow: /gp/voting/ Disallow: /gp/yourstore Disallow: /inc Disallow: /js Disallow: /lib Disallow: /mn/bookLookInsideApp Disallow: /mn/checkInitApp Disallow: /mn/checkoutAlertMsgApp Disallow: /mn/checkoutredirectApp Disallow: /mn/giftCardApp Disallow: /mn/loginApplication Disallow: /mn/loyaltyApp Disallow: /mn/orderAddrApp Disallow: /mn/orderCfmApp Disallow: /mn/orderDetailApp Disallow: /mn/orderFailApp Disallow: /mn/orderHistoryApp Disallow: /mn/orderModifyApp Disallow: /mn/orderSummaryApp Disallow: /mn/paymentRedriveApp Disallow: /mn/recommendReviewApp Disallow: /mn/releaseReviewApp Disallow: /mn/reviewVoteApplication Disallow: /mn/selectPaymentMethodApp Disallow: /mn/selectShippingOpptionApplication Disallow: /mn/shipmentTraceApp Disallow: /mn/shoppingCartApplication Disallow: /mn/tellFriend Disallow: /mn/thankYouApplication Disallow: /mn/virtualAccountApp Disallow: /mn/yourAccountApp Disallow: /paper Disallow: /xml Disallow: /youraccount Disallow: /ap/signin Disallow: /gp/registry/wishlist/ Disallow: /wishlist/ Allow: /wishlist/universal* Allow: /wishlist/vendor-button* Allow: /wishlist/get-button* Disallow: /gp/wishlist/ Allow: /gp/wishlist/universal* Allow: /gp/wishlist/vendor-button* Allow: /gp/wishlist/ipad-install* Disallow: /registry/wishlist/ Disallow: /gp/help/customer/display.html*nodeId=200843370 Disallow: /gp/help/customer/display.html*nodeId=200877580 Disallow: /gp/help/customer/display.html*nodeId=200877590 Disallow: /gp/help/customer/display.html*nodeId=200879080 Disallow: /gp/help/customer/display.html*nodeId=200879100 Disallow: /gp/help/customer/display.html*nodeId=200879120 Disallow: /gp/help/customer/display.html*nodeId=200879160 Disallow: /gp/help/customer/display.html*nodeId=200879140 Disallow: /gp/help/customer/display.html*nodeId=200877610 Disallow: /gp/help/customer/display.html*nodeId=200878960 Disallow: /gp/help/customer/display.html*nodeId=200878980 Disallow: /gp/help/customer/display.html*nodeId=200879000 Disallow: /gp/help/customer/display.html*nodeId=200879040 Disallow: /gp/help/customer/display.html*nodeId=200879020 Disallow: /gp/help/customer/display.html*nodeId=200877630 Disallow: /gp/help/customer/display.html*nodeId=200879200 Disallow: /gp/help/customer/display.html*nodeId=200879220 Disallow: /gp/help/customer/display.html*nodeId=200879240 Disallow: /gp/help/customer/display.html*nodeId=200879280 Disallow: /gp/help/customer/display.html*nodeId=200879260 Disallow: /gp/help/customer/display.html*nodeId=200877650 Disallow: /gp/help/customer/display.html*nodeId=200879320 Disallow: /gp/help/customer/display.html*nodeId=200879340 Disallow: /gp/help/customer/display.html*nodeId=200879360 Disallow: /gp/help/customer/display.html*nodeId=200879400 Disallow: /gp/help/customer/display.html*nodeId=200879380 Disallow: /gp/help/customer/display.html*nodeId=200877560 Disallow: /gp/help/customer/display.html*nodeId=200843460 Disallow: /gp/help/customer/display.html*nodeId=200843440 Disallow: /gp/help/customer/display.html*nodeId=200899270 Disallow: /gp/help/customer/display.html*nodeId=200879440 Disallow: /gp/help/customer/display.html*nodeId=200899330 Disallow: /gp/help/customer/display.html*nodeId=200899350 Disallow: /gp/help/customer/display.html*nodeId=200899390 Disallow: /gp/help/customer/display.html*nodeId=200899410 Disallow: /gp/help/customer/display.html*nodeId=200899430 Disallow: /gp/help/customer/display.html*nodeId=200899220 Disallow: /gp/help/customer/display.html*nodeId=200899450 Disallow: /gp/help/customer/display.html*nodeId=200899670 Disallow: /gp/help/customer/display.html*nodeId=200899530 Disallow: /gp/help/customer/display.html*nodeId=200899470 Disallow: /gp/help/customer/display.html*nodeId=200899550 Disallow: /gp/help/customer/display.html*nodeId=200899570 Disallow: /gp/help/customer/display.html*nodeId=200899590 Disallow: /gp/help/customer/display.html*nodeId=200899490 Disallow: /gp/help/customer/display.html*nodeId=200899510 Disallow: /gp/help/customer/display.html*nodeId=200899610 Disallow: /gp/help/customer/display.html*nodeId=200899630 Disallow: /gp/help/customer/display.html*nodeId=200899650 Disallow: /gp/help/customer/display.html*nodeId=200879180 Disallow: /gp/help/customer/display.html*nodeId=200879060 Disallow: /gp/help/customer/display.html*nodeId=200879300 Disallow: /gp/help/customer/display.html*nodeId=200879420 Disallow: /gp/help/customer/display.html*nodeId=200899290 Disallow: /gp/help/customer/display.html*nodeId=200899310 Disallow: /gp/help/customer/display.html*nodeId=200843380 Disallow: /gp/help/customer/display.html*nodeId=200843420 Disallow: /gp/help/customer/display.html*nodeId=200899230 Disallow: /gp/help/customer/display.html*nodeId=200899250 Disallow: /gp/help/customer/display.html*nodeId=200899370 Disallow: /gp/help/contact-us/general-questions.html*?type&email&skip=true Disallow: /gp/help/customer/accessibility?ie=UTF8&initialIssue=forgotpw&skip=true Disallow: /gp/registry/search.html Disallow: /gp/orc/rml/ Disallow: /gp/digital/fiona/manage Disallow: /gp/entity-alert/external Disallow: /gp/customer-reviews/dynamic/sims-box Disallow: /review/dynamic/sims-box Disallow: /gp/redirect.html # Sitemap files Sitemap: http://www.amazon.cn/sitemap_feed_index1.xml Sitemap: http://www.amazon.cn/sitemaps.f3053414d236e84.SitemapIndex_0.xml.gz Sitemap: http://www.amazon.cn/sitemaps.1946f6b8171de60.SitemapIndex_0.xml.gz Sitemap: http://www.amazon.cn/sitemaps.c21f969b5f03d33.SitemapIndex_0.xml.gz
我们可以将压缩包，下载下来，打开可以看到是一个xml文件！！
我们再来看一个：
哎？ia_archiver是什么爬虫啊？没见过啊？
百度一下~~
ia_archiver是alexa的一个 爬虫程序，用于检测网站是否做了alexa排名的作弊。
ia_archiver程序会自动在互联网上爬行，刺探每个Web页面的流量信息。尤其是当某个网站的流量超过Alexa设定的阈值时，IA_Archiver就会马上爬到该网站的服务器上，分析此网站的流量是否正常，有没有作弊行为。
邀请ia_arhiver来访
到alexa官网进行登记即可。
禁止ia_archiver访问
ia_archiver是一个中等强度的爬虫。如果你觉得它占用了过多的 服务器资源，同时不关心网站 alexa排名的话，可以屏蔽这个爬虫。方法为在服务器上的网站根目录建立 robots.txt，包含以下内容：
User-agent: ia_archiver Disallow: /
上面在全站之内禁止ia_archiver爬行。或者禁止爬行某个目录：
User-agent: ia_archiver Disallow: /somedir/
基本上就这些了~~~
还有一些好玩的~~大家可以参考：http://lusongsong.com/reed/732.html
关于robots协议就到这里了！！