HtmlUnit爬取js动态生成的网页

 private Document doHttp(String url) throws IOException {

         //构造一个webClient 模拟Chrome 浏览器
        WebClient webClient = new WebClient(BrowserVersion.CHROME);
        //支持JavaScript
        webClient.getOptions().setJavaScriptEnabled(true);
        webClient.getOptions().setCssEnabled(false);
        webClient.getOptions().setActiveXNative(false);
        webClient.getOptions().setCssEnabled(false);
        webClient.getOptions().setThrowExceptionOnScriptError(false);
        webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
        webClient.getOptions().setTimeout(8000);
        HtmlPage rootPage = webClient.getPage(url);
        //设置一个运行JavaScript的时间
        webClient.waitForBackgroundJavaScript(6000);
        String html = rootPage.asXml();
        Document doc = Jsoup.parse(html);
        webClient.close();
        return doc;
    }

pom部分 

        
            org.jsoup
            jsoup
            1.11.3
            test
        
        
            com.alibaba
            fastjson
            1.2.47
        
        
            net.sourceforge.htmlunit
            htmlunit
            2.29
            test
        
        
            org.apache.httpcomponents
            httpclient
            4.5.2
        
        
            xml-apis
            xml-apis
            1.4.01
        

        
            xerces
            xercesImpl
            2.11.0
        

 

参考文章：【Jsoup】配合 htmlunit 爬取异步加载的网页

导包报错处理

httpUnit模拟点击事件