httpclient+nekohtml 解析HTML[转]
nekohtml.jar,xerces-2.9.0.jar,commons-httpclient-3.1.jar,commons-logging-1.1.jar,commons-codec-1.3.jar
import org.cyberneko.html.parsers.DOMParser;
import org.xml.sax.InputSource;
import org.xml.sax.SAXException;
import org.w3c.dom.Document;
import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.HttpException;
import org.apache.commons.httpclient.HttpStatus;
import org.apache.commons.httpclient.methods.GetMethod;
import java.io.InputStream;
import java.io.IOException;
public class Html2XML {
??? private int connectionTimeout = 5000;
??? private int soTimeout = 12000;
??? private String proxyHost = null;
??? private int proxyPort;
??? public Document getDocument(String url) {
??????? HttpClient client = new HttpClient();
??????? if (proxyHost != null) {
??????????? client.getHostConfiguration().setProxy(proxyHost, proxyPort);
??????? }
??????? client.getHttpConnectionManager().getParams().setConnectionTimeout(connectionTimeout);
??????? client.getHttpConnectionManager().getParams().setSoTimeout(soTimeout);
??????? GetMethod method = new GetMethod(url);
??????? method.addRequestHeader("Content-Type", "text/html; charset=utf-8");
??????? try {
??????????? int statusCode = client.executeMethod(method);
??????????? if (statusCode != HttpStatus.SC_OK) {
??????????????? throw new HttpException("HttpStatusCode : " + statusCode);
??????????? }
??????????? InputStream is = method.getResponseBodyAsStream();
??????????? DOMParser parser = new DOMParser();
??????????? parser.setProperty("http://cyberneko.org/html/properties/default-encoding", "utf-8");
??????????? parser.parse(new InputSource(is));
??????????? return (parser.getDocument());
??????? } catch (HttpException he) {
??????????? he.printStackTrace();
??????? } catch (IOException ie) {
??????????? ie.printStackTrace();
??????? } catch (SAXException se) {
??????????? se.printStackTrace();
??????? }
??????? return null;
??? }
}
设置解析器参数
为了更加精确的控制解析的动作，nekohtml提供了相应的设置函数。如下列：
// settings on HTMLConfiguration
org.apache.xerces.xni.parser.XMLParserConfiguration config =
new org.cyberneko.html.HTMLConfiguration();
config.setFeature("http://cyberneko.org/html/features/augmentations", true);
config.setProperty("http://cyberneko.org/html/properties/names/elems", "lower");
// settings on DOMParser
org.cyberneko.html.parsers.DOMParser parser =
new org.cyberneko.html.parsers.DOMParser();
parser.setFeature("http://cyberneko.org/html/features/augmentations", true);
parser.setProperty("http://cyberneko.org/html/properties/names/elems", "lower");
nekohtml功能(feature)列表
功能 默认值 描述
http://cyberneko.org/html/features/balance-tags True 是否允许增补缺失的标签。如果要以XML方式操作HTML文件，此值必须为真。此处提供设置功能，为了性能的原因。
http://cyberneko.org/html/features/balance-tags/ignore-outside-content False 是否忽略文档根元素以后的数据。如果为false，<html>和<bod>被忽略，所有的内容都被解析。
http://cyberneko.org/html/features/document-fragment False 解析HTML片段时是否作标签增补。此功能不要用在DOMParser上，而要用在DOMFragmentParser上。
http://apache.org/xml/features/scanner/notify-char-refs False 当遇到字符实体引用（如＆#x20;）是否将(#x20)报告给相应地文档处理器。
http://apache.org/xml/features/scanner/notify-builtin-refs False 当遇到XML内建的字符实体引用（如＆amp;）是否将(amp)报告给相应地文档处理器。
http://cyberneko.org/html/features/scanner/notify-builtin-refs False 当遇到HTML内建的字符实体引用（如＆copy;）是否将(copy)报告给相应地文档处理器。
http://cyberneko.org/html/features/scanner/script/strip-comment-delims False 是否剥掉<script>元素中的<!---->等注释符。
http://cyberneko.org/html/features/augmentations False 是否将与HTML事件有关的infoset项包括在解析管道中。
http://cyberneko.org/html/features/report-errors False 是否报告错误。
nekohtml属性列表
属性 默认值 值域 描述
http://cyberneko.org/html/properties/filters null XMLDocumentFilter[] 在解析管道的最后按数组顺序追加自定义的处理组件（过滤器），必须为数组类型。
http://cyberneko.org/html/properties/default-encoding Windows-1252 IANA encoding names 默认的HTML文件编码
http://cyberneko.org/html/properties/names/elems upper upper,lower,match 如果整理识别出的元素名称
http://cyberneko.org/html/properties/names/attrs lower upper,lower,no-change 如果整理识别出的属性名称