【JavaWeb】动态网页抓取

Jsoup无法获取Js及Ajax执行后的网页内容,用HtmlUnit抓取动态网页:

private String getPage(String url,boolean enabledJs,boolean ignoreSSL,boolean enabledCss,boolean enabledAjax) throws IOException {
        WebClient webClient = new WebClient(BrowserVersion.CHROME); //创建一个webclient
        webClient.getOptions().setJavaScriptEnabled(enabledJs); // 启动JS
        webClient.getOptions().setUseInsecureSSL(ignoreSSL);//忽略ssl认证
        webClient.getOptions().setCssEnabled(enabledCss);//禁用Css,可避免自动二次请求CSS进行渲染
        webClient.getOptions().setThrowExceptionOnScriptError(false);//运行错误时,不抛出异常
        if(enabledAjax)
            webClient.setAjaxController(new NicelyResynchronizingAjaxController());// 设置Ajax异步

        HtmlPage page = webClient.getPage(url);
        webClient.waitForBackgroundJavaScript(10000);
        return page.asXml();
    }

你可能感兴趣的:(JavaWeb,HtmlUnit)