htmlparser对html页面处理的算法
主要是如下几种方式
采用Visitor方式访问Html
try {
Parser parser = new Parser();
parser.setURL(”http://www.google.com”);
parser.setEncoding(parser.getEncoding());
NodeVisitor visitor = new NodeVisitor() {
public void visitTag(Tag tag) {
System.out.println (”testVisitorAll() Tag name is :”
+ tag.getTagName() + ” \n Class is :”
+ tag.getClass());
}
};
parser.visitAllNodesWith(visitor);
} catch (ParserException e) {
e.printStackTrace();
}
采用Filter方式访问html
try {
NodeFilter filter = new NodeClassFilter(LinkTag.class);
Parser parser = new Parser();
parser.setURL(”http://www.google.com”);
parser.setEncoding(parser.getEncoding());
NodeList list = parser.extractAllNodesThatMatch(filter);
for (int i = 0; i < list.size(); i++) {
LinkTag node = (LinkTag) list.elementAt(i);
System.out.println(”testLinkTag() Link is :” + node.extractLink());
}
} catch (Exception e) {
e.printStackTrace();
}
采用org.htmlparser.beans方式
另外htmlparser 还在org.htmlparser.beans中对一些常用的方法进行了封装，以简化操作，例如：
Parser parser = new Parser();
LinkBean linkBean = new LinkBean();
linkBean.setURL(”http://www.google.com”);
URL[] urls = linkBean.getLinks();
for (int i = 0; i < urls.length; i++) {
URL url = urls[i];
System.out.println (”testLinkBean() -url is :” + url);
}
Htmlparser关键包结构说明
htmlparser其实核心代码并不多，好好研究一下其代码，弥补文档不足的问题。同时htmlparser的代码注释和单元测试用例还是很齐全的，也有助于了解htmlparser的用法。
3.1、org.htmlparser
定义了htmlparser的一些基础类。其中最为重要的是Parser类。
Parser是htmlparser的最核心的类，其构造函数提供了如下：Parser.createParser (String html, String charset)、 Parser ()、Parser (Lexer lexer, ParserFeedback fb)、Parser (URLConnection connection, ParserFeedback fb)、Parser (String resource, ParserFeedback feedback)、 Parser (String resource)
各构造函数的具体用法及含义可以查看其代码，很容易理解。
Parser常用的几个方法：
• elements获取元素
Parser parser = new Parser (”http://www.google.com”);
for (NodeIterator i = parser.elements (); i.hasMoreElements (); )
processMyNodes (i.nextNode ());
• parse (NodeFilter filter)：通过NodeFilter方式获取
• visitAllNodesWith (NodeVisitor visitor)：通过Nodevisitor方式
• extractAllNodesThatMatch (NodeFilter filter)：通过NodeFilter方式
3.2、org.htmlparser.beans
对Visitor和Filter的方法进行了封装，定义了针对一些常用html元素操作的bean，简化对常用元素的提取操作。
包括：FilterBean、HTMLLinkBean、HTMLTextBean、LinkBean、StringBean、BeanyBaby等。
3.3、org.htmlparser.nodes
定义了基础的node，包括：AbstractNode、RemarkNode、TagNode、TextNode等。
3.4、org.htmlparser.tags
定义了htmlparser的各种tag。
3.5、org.htmlparser.filters
定义了htmlparser所提供的各种filter，主要通过extractAllNodesThatMatch (NodeFilter filter)来对html页面指定类型的元素进行过滤，包括：AndFilter、CssSelectorNodeFilter、HasAttributeFilter、HasChildFilter、HasParentFilter、HasSiblingFilter、IsEqualFilter、LinkRegexFilter、LinkStringFilter、NodeClassFilter、NotFilter、OrFilter、RegexFilter、StringFilter、TagNameFilter、XorFilter
3.6、org.htmlparser.visitors
定义了htmlparser所提供的各种visitor，主要通过visitAllNodesWith (NodeVisitor visitor)来对html页面元素进行遍历，包括：HtmlPage、LinkFindingVisitor、NodeVisitor、ObjectFindingVisitor、StringFindingVisitor、TagFindingVisitor、TextExtractingVisitor、UrlModifyingVisitor
3.7、org.htmlparser.parserapplications
定义了一些实用的工具，包括LinkExtractor、SiteCapturer、StringExtractor、WikiCapturer，这几个类也可以作为htmlparser使用样例。
3.8、org.htmlparser.tests
对各种功能的单元测试用例，也可以作为htmlparser使用的样例。
1 . 逻辑关系：与或非
AndFilter() Creates a new instance of an AndFilter. |
AndFilter(NodeFilter[] predicates) Creates an AndFilter that accepts nodes acceptable to all given filters. |
AndFilter(NodeFilter left, NodeFilter right) Creates an AndFilter that accepts nodes acceptable to both filters. |
OrFilter() Creates a new instance of an OrFilter. |
OrFilter(NodeFilter[] predicates) Creates an OrFilter that accepts nodes acceptable to any of the given filters. |
OrFilter(NodeFilter left, NodeFilter right) Creates an OrFilter that accepts nodes acceptable to either filter. |
OrFilter() Creates a new instance of an OrFilter. |
OrFilter(NodeFilter[] predicates) Creates an OrFilter that accepts nodes acceptable to any of the given filters. |
OrFilter(NodeFilter left, NodeFilter right) Creates an OrFilter that accepts nodes acceptable to either filter. |
2. 内容
StringFilter：功能简单有限；复杂功能可使用RegexFilter (正则表达式)
StringFilter() Creates a new instance of StringFilter that accepts all string nodes. |
StringFilter(String pattern) Creates a StringFilter that accepts text nodes containing a string. |
StringFilter(String pattern, boolean sensitive) Creates a StringFilter that accepts text nodes containing a string. |
StringFilter(String pattern, boolean sensitive, Locale locale) Creates a StringFilter that accepts text nodes containing a string. |
RegexFilter() Creates a new instance of RegexFilter that accepts string nodes matching the regular expression ".*" using the FIND strategy. |
RegexFilter(String pattern) Creates a new instance of RegexFilter that accepts string nodes matching a regular expression using the FIND strategy. |
RegexFilter(String pattern, int strategy) Creates a new instance of RegexFilter that accepts string nodes matching a regular expression. |
3 标签
TagNameFilter()利用标签名过滤 : div ,img , ...
NodeClassFilter()利用标签类别 ：LinkTag.class ...
HasAttributeFilter()利用属性 ：HasAttributeFilter(“class”, “className”)
LinkRegexFilter（）用正则表达式匹配链接
TagNameFilter() Creates a new instance of TagNameFilter. |
TagNameFilter(String name) Creates a TagNameFilter that accepts tags with the given name. |
NodeClassFilter() Creates a NodeClassFilter that accepts Html tags. |
NodeClassFilter(Class cls) Creates a NodeClassFilter that accepts tags of the given class. |
HasAttributeFilter() Creates a new instance of HasAttributeFilter. |
HasAttributeFilter(String attribute) Creates a new instance of HasAttributeFilter that accepts tags with the given attribute. |
HasAttributeFilter(String attribute, String value) Creates a new instance of HasAttributeFilter that accepts tags with the given attribute and value. |
LinkRegexFilter(String regexPattern) Creates a LinkRegexFilter that accepts LinkTag nodes containing a URL that matches the supplied regex pattern. |
LinkRegexFilter(String regexPattern, boolean caseSensitive) Creates a LinkRegexFilter that accepts LinkTag nodes containing a URL that matches the supplied regex pattern. |
LinkStringFilter(String pattern) Creates a LinkStringFilter that accepts LinkTag nodes containing a URL that matches the supplied pattern. |
LinkStringFilter(String pattern, boolean caseSensitive) Creates a LinkStringFilter that accepts LinkTag nodes containing a URL that matches the supplied pattern. |
4 层次关系
HasParentFilter() Creates a new instance of HasParentFilter. |
HasParentFilter(NodeFilter filter) Creates a new instance of HasParentFilter that accepts nodes with the direct parent acceptable to the filter. |
HasParentFilter(NodeFilter filter, boolean recursive) Creates a new instance of HasParentFilter that accepts nodes with a parent acceptable to the filter. |
HasChildFilter() Creates a new instance of a HasChildFilter. |
HasChildFilter(NodeFilter filter) Creates a new instance of HasChildFilter that accepts nodes with a direct child acceptable to the filter. |
HasChildFilter(NodeFilter filter, boolean recursive) Creates a new instance of HasChildFilter that accepts nodes with a child acceptable to the filter. |