Nutch中User Agent的问题

本人原创，转载请注明出处（http://blog.csdn.net/panjunbiao/article/details/16960029）。

Apache Nutch 1.7版本抓取某个网站的时候出现错误提示：

2013-11-25 15:23:37,793 INFO  api.HttpRobotRulesParser - Couldn't get robots.txt for http://www.xxx.com/: java.io.EOFException 2013-11-25 15:23:37,893 ERROR http.Http - Failed to get protocol output java.io.EOFException         at org.apache.nutch.protocol.http.HttpResponse.readLine(HttpResponse.java:427)         at org.apache.nutch.protocol.http.HttpResponse.parseStatusLine(HttpResponse.java:319)         at org.apache.nutch.protocol.http.HttpResponse.<init>(HttpResponse.java:154)         at org.apache.nutch.protocol.http.Http.getResponse(Http.java:64)         at org.apache.nutch.protocol.http.api.HttpBase.getProtocolOutput(HttpBase.java:140)         at org.apache.nutch.fetcher.Fetcher$FetcherThread.run(Fetcher.java:703) 2013-11-25 15:23:37,903 INFO  fetcher.Fetcher - fetch of http://www.xxxx.com/ failed with: java.io.EOFException

也就是无法获取robots.txt文件，检查HTTP请求并无明显问题，其他网站是可以抓取的。

用wireshark抓包发现，抓取****网站的GET请求收不到回应，怀疑是User-Agent的问题。

用curl测试之，先用包含Nutch关键字的User-Agent：

Jamess-MacBook-Pro:~ james$ curl -A "Friendly Crawler/Nutch-1.7" -v http://www.****.com/robots.txt * Adding handle: conn: 0x7fe33400fe00 * Adding handle: send: 0 * Adding handle: recv: 0 * Curl_addHandleToPipeline: length: 1 * - Conn 0 (0x7fe33400fe00) send_pipe: 1, recv_pipe: 0 * About to connect() to  www.****.com port 80 (#0) *   Trying 42.121.98.156... * Connected to  www.****.com (42.121.98.156) port 80 (#0) > GET /robots.txt HTTP/1.1 > User-Agent: Friendly Crawler/Nutch-1.7 > Host:  www.****.com > Accept: */* > * Empty reply from server * Connection #0 to host  www.****.com left intact curl: (52) Empty reply from server

再用不包含Nutch关键字的User-Agent：

Jamess-MacBook-Pro:~ james$ curl -A "Chrome" -v  http://www.****.com/robots.txt * Adding handle: conn: 0x7f82f180fe00 * Adding handle: send: 0 * Adding handle: recv: 0 * Curl_addHandleToPipeline: length: 1 * - Conn 0 (0x7f82f180fe00) send_pipe: 1, recv_pipe: 0 * About to connect() to  www.****.com port 80 (#0) *   Trying 42.121.98.156... * Connected to  www. ****.com (42.121.98.156) port 80 (#0) > GET /robots.txt HTTP/1.1 > User-Agent: Chrome > Host:  www. ****.com > Accept: */* > < HTTP/1.1 200 OK * Server nginx/1.5.6 is not blacklisted < Server: nginx/1.5.6 < Date: Mon, 25 Nov 2013 08:51:13 GMT < Content-Type: text/plain; charset=utf-8 < Content-Length: 60 < Last-Modified: Tue, 04 Dec 2012 01:29:47 GMT < Connection: keep-alive < ETag: "50bd520b-3c" < X-UA-Compatible: IE=Edge,chrome=1 < X-XSS-Protection: 1; mode=block < Accept-Ranges: bytes < User-agent: * Disallow: /robot/trap Disallow: /page/6559999 * Connection #0 to host  www. ****.com left intact

果然，目标网站对User-Agent包含Nutch关键字的请求过滤了，然而在哪里修改这个User-Agent字段的内容呢？

搜索apache中包含http.agent.name源代码，在./plugin/lib-http/src/java/org/apache/nutch/protocol/http/api/HttpBase.java中：

// Inherited Javadoc   public void setConf(Configuration conf) {       this.conf = conf;       this.proxyHost = conf.get("http.proxy.host");       this.proxyPort = conf.getInt("http.proxy.port", 8080);       this.useProxy = (proxyHost != null && proxyHost.length() > 0);       this.timeout = conf.getInt("http.timeout", 10000);       this.maxContent = conf.getInt("http.content.limit", 64 * 1024);       this.userAgent = getAgentString(conf.get("http.agent.name"), conf.get("http.agent.version"), conf               .get("http.agent.description"), conf.get("http.agent.url"), conf.get("http.agent.email"));       this.acceptLanguage = conf.get("http.accept.language", acceptLanguage);       this.accept = conf.get("http.accept", accept);       // backward-compatible default setting       this.useHttp11 = conf.getBoolean("http.useHttp11", false);       this.robots.setConf(conf);       logConf();   }

可以发现是http.agent.version设置的，修改之，问题解决。