java爬虫系列（二）——爬取动态网页
- 准备工作
- 项目地址
- 网页解析工具地址
- 启动网页解析器
- 根据系统选择所需文件
- 指定端口号启动工具
- 项目配置
- seimi.properties
- SeimiAgentDemo.java
- 分析原网页代码
- Boot.java
- 同系列文章
准备工作
新手的话推荐使用seimiagent+seimicrawler的爬取方式，非常容易上手，轻松爬取动态网页，目测初步上手10分钟以内吧。
项目地址
https://github.com/a252937166/seimicrawler
网页解析工具地址
https://github.com/a252937166/seimiagent
启动网页解析器
根据系统选择所需文件
下载好seimiagent，根据自己的操作系统，如果是windows，就用seimiagent.exe，如果是linux，就选择seimiagent，mac版本暂时还没有，我的话一般会把seimiagent放在自己的linux服务器上。
指定端口号启动工具
以linux为例，进入文件所在目录，指定8000端口，./seimiagent -p 8000，即可启动。
项目配置
seimi.properties
redis.host=127.0.0.1
redis.port=6379
redis.password=
database.driverClassName=com.mysql.jdbc.Driver
database.url=
database.username=
database.password=
seimiAgentHost=127.0.0.1
seimiAgentPort=8000找到此配置文件，修改seimiAgentHost和seimiAgentPort为自己的地址信息。
SeimiAgentDemo.java
package com.ouyang.crawlers;
import cn.wanghaomiao.seimi.annotation.Crawler;
import cn.wanghaomiao.seimi.def.BaseSeimiCrawler;
import cn.wanghaomiao.seimi.struct.Request;
import cn.wanghaomiao.seimi.struct.Response;
import cn.wanghaomiao.xpath.model.JXDocument;
import org.apache.commons.lang3.StringUtils;
import org.springframework.beans.factory.annotation.Value;
/**
* 这个例子演示如何使用SeimiAgent进行复杂动态页面信息抓取
* @author 汪浩淼 [email protected]
* @since 2016/4/14.
*/
@Crawler(name = "seimiagent")
public class SeimiAgentDemo extends BaseSeimiCrawler{
/**
* 在resource/config/seimi.properties中配置方便更换，当然也可以自行根据情况使用自己的统一配置中心等服务
*/
@Value("${seimiAgentHost}")
private String seimiAgentHost;
@Value("${seimiAgentPort}")
private int seimiAgentPort;
@Override
public String[] startUrls() {
return new String[]{"https://www.baidu.com"};
}
@Override
public String seimiAgentHost() {
return this.seimiAgentHost;
}
@Override
public int seimiAgentPort() {
return this.seimiAgentPort;
}
@Override
public void start(Response response) {
Request seimiAgentReq = Request.build("http://manhua.fzdm.com/2/889/","getHtml")
.useSeimiAgent()
// 告诉SeimiAgent针对这个请求是否使用cookie，如果没有设置使用当前Crawler关于cookie使用条件作为默认值。
// .setSeimiAgentUseCookie(true)
//设置全部load完成后给SeimiAgent多少时间用于执行js并渲染页面，单位为毫秒
.setSeimiAgentRenderTime(5000);
push(seimiAgentReq);
}
/**
* 打印网页信息
* @param response
*/
public void getHtml(Response response){
try {
System.out.println(response.getContent());
} catch (Exception e) {
e.printStackTrace();
}
}
}
找到这个demo文件，注意.useSeimiAgent()，这就是表示需要使用seimiagent来解析动态网页了，此外还可以设置cookie，param，meta这些参数。
我们使用getHtml()这个回调函数来打印网页信息，对比一下看看，有没有顺利解析成功。
分析原网页代码
我们使用chrome浏览器的network查看网页原始代码。
<html xmlns="http://www.w3.org/1999/xhtml" lang="utf-8">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta http-equiv="Content-Language" content="utf-8" />
<meta content="all" name="robots" />
<title>海贼王889话 风之动漫title>
<meta name="keywords" content="海贼王889话 " />
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta http-equiv="Cache-Control" content="no-transform" />
<meta http-equiv="Cache-Control" content="no-siteapp" />
<meta name="applicable-device" content="pc,mobile" />
<meta name="HandheldFriendly" content="true" />
<meta property="og:title" content="海贼王889话"/>
<meta property="og:type" content="book"/>
<meta property="og:url" id="readurl" content="http://manhua.fzdm.com/2/889/" />
<link rel="stylesheet" href="//static.fzdm.com/pure/pure-min.css">
<link rel="stylesheet" href="//static.fzdm.com/pure/grids-responsive-min.css">
<link rel="stylesheet" href="//static.fzdm.com/pure/fzdm.css">
<link rel="icon" href="//static.fzdm.com/favicon.ico" mce_href="//static.fzdm.com/favicon.ico" type="image/x-icon">
<meta name="renderer" content="webkit">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<link rel="apple-touch-icon" href="//static.fzdm.com/apple-touch-icon-144x144.png" />
<style>
.logo {
top: -2px;
height: 70px;
overflow: hidden;}
.logo img{height:77px}
#header {
height: 70px;
}
#header ul {
top: 8px;}
.pure-menu.pure-menu-open, .pure-menu.pure-menu-horizontal li .pure-menu-children {
text-align: left;
height: 70px;
background: none;
/* border: 1px solid #b7b7b7; */
}
style>
<script>
var _hmt = _hmt || [];
(function() {
var hm = document.createElement("script");
hm.src = "//hm.baidu.com/hm.js?cb51090e9c10cda176f81a7fa92c3dfc";
var s = document.getElementsByTagName("script")[0];
s.parentNode.insertBefore(hm, s);
})();
script>
head>
<body>
<script src="//static.fzdm.com/jquery-1.9.1.min.js?v=1">script>
<script src="//static.fzdm.com/fzdm.js?v=1">script>
<script src="//static.fzdm.com/u.js">script>
<script src="//dup.baidustatic.com/js/dm.js">script>
<div id="header">
<div class="pure-g">
<div class="pure-menu pure-menu-open pure-menu-horizontal">
<div class="logo">
<a href="//www.fzdm.com"><img src="//static.fzdm.com/css/logo.png" alt="风之动漫" />a>
div>
<ul>
<li><a href="//www.fzdm.com/"> 首页a>li>
<li><a href="//news.fzdm.com/">动漫新闻a>li>
<li><a href="//manhua.fzdm.com/">在线漫画a>li>
<li><a href="//flash.fzdm.com/">动漫flasha>li>
ul>
div>
div>
div>
div>
<center>
center>
<br>
<br>
<div id="weizhi">位置：<a href="//www.fzdm.com">首页a> - <a href="../../">在线漫画a> - <a href="../">海贼王a> - 海贼王889话<h4 style="float:right;margin-right: 100px;"><a href="#comments">海贼王889话讨论区a>h4>div>
<div id="mh">
<h1>海贼王889话h1>
<div id="mhimg0"><h2><a href="//manhua.fzdm.com/2/889/">《无法观看》请点击此处～a>h2>div>
<center>
<div id="share">
<div class="bdsharebuttonbox"><a href="#" class="bds_more" data-cmd="more">分享<strong>海贼王889话漫画strong>到：a><a href="#" class="bds_qzone" data-cmd="qzone" title="分享到QQ空间">QQ空间a><a href="#" class="bds_weixin" data-cmd="weixin" title="分享到微信">微信a><a href="#" class="bds_sqq" data-cmd="sqq" title="分享到QQ好友">QQ好友a><a href="#" class="bds_tsina" data-cmd="tsina" title="分享到新浪微博">微博a><a href="#" class="bds_tqq" data-cmd="tqq" title="分享到腾讯微博">腾讯a><a href="#" class="bds_renren" data-cmd="renren" title="分享到人人网">人人网a><a href="#" class="bds_fbook" data-cmd="fbook" title="分享到Facebook">Facebooka><a href="#" class="bds_baidu" data-cmd="baidu" title="分享到百度搜藏">百度搜藏a><a href="#" class="bds_bdhome" data-cmd="bdhome" title="分享到百度新首页">百度首页a><a class="bds_count" data-cmd="count">a>div>
div>
<div id="ad">
<script src='//m.xmshqh.com/fz2.js'>script>
div>
center>
<div class="navigation">
<a href="index_0.html" id="mhona">第1页a><a href="index_1.html">2a><a href="index_2.html">3a><a href="index_3.html">4a><a href="index_4.html">5a><a href="index_5.html">6a><a href="index_6.html">7a><a href="index_7.html">8a><a href="index_8.html">9a><a href="index_9.html">10a><a href="index_10.html">11a><a href="index_11.html">12a><a href="index_12.html">13a><a href="index_13.html">14a><a href="index_14.html">15a><a href="index_15.html">16a><a href="index_16.html">17a><a href='index_1.html' id="mhona">下一页a>div>
<br />
<br />
<script type="text/javascript">
document.write('');
tanx_s = document.createElement("script");
tanx_s.type = "text/javascript";
tanx_s.charset = "gbk";
tanx_s.id = "tanx-s-mm_10028503_120355_28042038";
tanx_s.async = true;
tanx_s.src = "//p.tanx.com/ex?i=mm_10028503_120355_28042038";
tanx_h = document.getElementsByTagName("head")[0];
if(tanx_h)tanx_h.insertBefore(tanx_s,tanx_h.firstChild);
script><script type="text/javascript">
document.write('');
tanx_s = document.createElement("script");
tanx_s.type = "text/javascript";
tanx_s.charset = "gbk";
tanx_s.id = "tanx-s-mm_10028503_120355_28058018";
tanx_s.async = true;
tanx_s.src = "//p.tanx.com/ex?i=mm_10028503_120355_28058018";
tanx_h = document.getElementsByTagName("head")[0];
if(tanx_h)tanx_h.insertBefore(tanx_s,tanx_h.firstChild);
script>
<script type="text/javascript">
document.write('');
tanx_s = document.createElement("script");
tanx_s.type = "text/javascript";
tanx_s.charset = "gbk";
tanx_s.id = "tanx-s-mm_10028503_120355_28066012";
tanx_s.async = true;
tanx_s.src = "//p.tanx.com/ex?i=mm_10028503_120355_28066012";
tanx_h = document.getElementsByTagName("head")[0];
if(tanx_h)tanx_h.insertBefore(tanx_s,tanx_h.firstChild);
script>
<br />
<br><br>
<script charset="gbk" src="//p.tanx.com/ex?i=mm_10028503_120355_41360495">script><br /><br />
<br />
<br /><div id="weizhi">热门漫画导航： <a href='//manhua.fzdm.com/91/' target=_blank>美食的俘虏漫画a> - <a href='//manhua.fzdm.com/7/ 'target=_blank>死神漫画a> - <a href='//manhua.fzdm.com/39/' target=_blank>进击的巨人漫画a> - <a href='//manhua.fzdm.com/35/' target=_blank>家庭教师漫画a> - <a href="//manhua.fzdm.com/27/" target=_blank>妖精的尾巴漫画a> - <a href="//manhua.fzdm.com/1/" target=_blank>火影忍者漫画a> - <a href='//manhua.fzdm.com/53/' target=_blank>黑子的篮球漫画a> - <a href='//manhua.fzdm.com/45/' target=_blank>恶魔奶爸漫画a> - <a href='//manhua.fzdm.com/51/' target=_blank>史上最强弟子兼一漫画a> - <a href='//manhua.fzdm.com/74/' target=_blank>王者天下漫画a> - <a href='//manhua.fzdm.com/56/' target=_blank>七原罪漫画a> - <a href='//manhua.fzdm.com/141/' target=_blank>暗杀教室漫画a>div>
<div id="mhimg1">div>
<script type="text/javascript">
var mhurl = "2017/12/22064917941533.jpg";
var mhss = getCookie("picHost");
if (mhss == "") {
mhss = "p1.xiaoshidi.net";
}
if (mhurl.indexOf("2015") != -1 || mhurl.indexOf("2016") != -1|| mhurl.indexOf("2017") != -1 || mhurl.indexOf("2018") != -1){
}else{
mhss = mhss.replace(/p1/,"p0");
};
var mhpicurl = mhss+"/"+mhurl;
if (mhurl.indexOf("http") != -1){
mhpicurl = mhurl;
};
function nofind(){
var img=event.srcElement;
img.src="http://p1.xiaoshidi.net/"+mhurl;
var exp = new Date();
exp.setTime(exp.getTime() - 1);
document.cookie = "picHost=0;path=/;domain=fzdm.com;expires="+exp.toGMTString();
img.onerror=null;
};
$("#mhimg0").html('
'" id="mhpic" alt="海贼王889话" onerror="nofind();" />');
var mhurl1 = "2017/12/22064917942026.jpg";
mhpicurl = mhss+"/"+mhurl1;
$("#mhimg1").html(''" width="0" height="0" id="mhpic1" />');
script>
<br />
<br />
<br />
<script>
if (document.location.protocol == "http:"){
window._bd_share_config={"common":{"bdSnsKey":{},"bdText":"海贼王889话 风之动漫","bdUrl":"//manhua.fzdm.com/2/889/","bdDesc":"海贼王889话","bdMini":"2","bdMiniList":false,"bdSign":"","bdPic":"","bdStyle":"0","bdSize":"16"},"share":{"bdSize":16}};with(document)0[(getElementsByTagName('head')[0]||body).appendChild(createElement('script')).src='http://bdimg.share.baidu.com/static/api/js/share.js?v=89860593.js?cdnversion='+~(-new Date()/36e5)];
};
script>
div>
<div class="clear">div>
<div id="footer">
<div id="hd">
<div class="bg">div>
<br><a href="//www.fzdm.com/about">关于我们a> | <a href="//www.fzdm.com/lianxi">联系我们a> | <a href="//www.fzdm.com/map">网站地图a><br />
Copyright ⓒ 2014-2015 风之动漫 版本beta 0.3<br />
div>
div>
<div style="display:none;" ><script src="//static.fzdm.com/stat.js">script>div>
body>
html>
请注意这段代码
<div id="mhimg0"><h2><a href="//manhua.fzdm.com/2/889/">《无法观看》请点击此处～a>h2>div>
如果直接爬取原网页，肯定没法获取图片的，那么图片从哪来呢？
var mhurl = "2017/12/22064917941533.jpg";
var mhss = getCookie("picHost");
if (mhss == "") {
mhss = "p1.xiaoshidi.net";
}
if (mhurl.indexOf("2015") != -1 || mhurl.indexOf("2016") != -1|| mhurl.indexOf("2017") != -1 || mhurl.indexOf("2018") != -1){
}else{
mhss = mhss.replace(/p1/,"p0");
};
var mhpicurl = mhss+"/"+mhurl;
if (mhurl.indexOf("http") != -1){
mhpicurl = mhurl;
};
function nofind(){
var img=event.srcElement;
img.src="http://p1.xiaoshidi.net/"+mhurl;
var exp = new Date();
exp.setTime(exp.getTime() - 1);
document.cookie = "picHost=0;path=/;domain=fzdm.com;expires="+exp.toGMTString();
img.onerror=null;
};
$("#mhimg0").html(''" id="mhpic" alt="海贼王889话" onerror="nofind();" />');这一段js代码在网页加载完后自动运行，修改了的内容，才有了图片。
js没有多余请求，只是修改了网页内容，这种情况如果想直接java解析，只能用正则，而且如果js代码稍有变化，正则解析就不行了。所以针对这种网页，直接用SeimiAgent把js渲染之后的网页返回给我们是最好的处理方式。
Boot.java
package com.ouyang.main;
import cn.wanghaomiao.seimi.core.Seimi;
/**
* @author 汪浩淼 [[email protected]]
* @since 2015/10/21.
*/
public class Boot {
public static void main(String[] args){
Seimi s = new Seimi();
s.goRun("seimiagent");
}
}
goRun("seimiagent");填写对用爬虫名就行了。
启动main函数：
seimiagent的解析信息，
windows版本是后台运行，没有解析信息。
控制台信息：
<div id="mhimg0"><a href="index_1.html"><img src="http://p1.xiaoshidi.net/2017/12/22064917941533.jpg" id="mhpic" alt="海贼王889话" onerror="nofind();">a>div>这段代码，很明显表示，网页信息已经是成功解析后的了。
总的来说这套爬虫框架还是很简单的，想要深入了解框架的同学，可以看看下一篇关于实战的文章。
同系列文章
java爬虫系列（一）——爬虫入门
java爬虫系列（三）——漫画网站爬取实战
java爬虫系列（四）——动态网页爬虫升级版
java爬虫系列（五）——今日头条文章爬虫实战