lucene-使用htmlparser提取网页普通链接

阅读更多

1、提取普通链接

以http://www.qunar.com/site/zh/Cooperate_4.shtml为例，只能提取普通链接，如EMAIL链接就无法提取

代码:

package extract;

import java.net.URL;

import org.htmlparser.beans.LinkBean;

public class extracthtmllinksimp {

/**
* @param args
*/
public static String getText(String f){
StringBuffer sb=new StringBuffer("");
LinkBean lb=new LinkBean();
lb.setURL(f);
URL[] urls=lb.getLinks();
for (int i=0;isb.append(urls[i]+"\n");
}
return sb.toString();
}
public static void main(String[] args) {
// TODO Auto-generated method stub
String s=getText("./htmls/Qunar_com.htm");
System.out.print(s);
}

}

2、效果

http://www.qunar.com/
http://www.qunar.com/
http://flight.qunar.com/
http://hotel.qunar.com/
http://zhidao.qunar.com/
http://visa.qunar.com/
http://i.qunar.com/
http://deals.qunar.com/
http://www.qunar.com/site/zh/Cooperate_4.shtml
http://www.qunar.com/
http://www.qunar.com/
http://www.qunar.com/site/zh/Qunar.in.China_1.2.shtml
http://www.qunar.com/site/zh/Qunar.in.China_1.2.shtml
http://www.qunar.com/site/zh/zhMilestones.shtml
http://www.qunar.com/site/zh/zhBackground.shtml
http://www.qunar.com/site/zh/Ours.Team_1.4.shtml
http://www.qunar.com/site/zh/News.Room_1.5.shtml
http://www.qunar.com/site/zh/Jobs_2.shtml
http://www.qunar.com/site/zh/ContactUs_3.shtml
http://un.qunar.com/
http://www.qunar.com/site/zh/Duty_5.shtml
http://www.qunar.com/site/zh/Privacy_6.shtml
http://www.qunar.com/site/zh/Question_7.shtml
http://www.qunar.com/site/zh/AboutBooking_10.shtml
http://www.qunar.com/site/zh/Links_8.shtml
http://www.qunar.com/site/zh/Feedback_9.shtml
http://www.qunar.com/site/en/Qunar.in.China_1.1.shtml
http://www.hd315.gov.cn/beian/view.asp?bianhao=010202007112700003
http://www.qunar.com/site/zh/Qunar.in.China_1.2.shtml
http://www.qunar.com/site/zh/ContactUs_3.shtml
http://www.qunar.com/site/zh/Cooperate_4.shtml
http://hot.qunar.com/
http://www.qunar.com/site/zh/Jobs_2.shtml
http://www.qunar.com/site/zh/Duty_5.shtml
http://123.qunar.com/
http://bbs.qunar.com/
http://www.qunar.com/site/zh/Question_7.shtml
http://www.qunar.com/site/zh/Tips_new.shtml
http://www.qunar.com/site/zh/Links_8.shtml
http://www.qunar.com/site/zh/Feedback_9.shtml
http://www.qunar.com/site/en/Qunar.in.China_1.1.shtml
http://www.miibeian.gov.cn/
http://downtmp.qunar.com/qunaricp.pdf
可以看到网页下部有EMAIL链接，但是没有提取