连接池技术作为创建和管理连接的缓冲池技术,目前已广泛用于诸如数据库连接等长连接的维护和管理中,能够有效减少系统的响应时间,节省服务器资源开销。其优势主要有两个:其一是减少创建连接的资源开销,其二是资源的访问控制。连接池管理的对象是长连接,对于HTTP连接是否适用,我们需要首先回顾一下长连接和短连接。
所谓长连接是指客户端与服务器端一旦建立连接以后,可以进行多次数据传输而不需要重新建立连接,而短连接则每次数据传输都需要客户端和服务器端建立一次连接。长连接的优势在于省去了每次数据传输连接建立的时间开销,能够大幅度提高数据传输的速度,对于P2P应用十分适合,但是对于诸如Web网站之类的B2C应用,并发请求量大,每一个用户又不需频繁操作的场景下,维护大量的长连接对服务器无疑是一个巨大的考验。而此时,短连接可能更加适用。但是短连接每次数据传输都需要建立连接,我们知道HTTP协议的传输层协议是TCP协议,TCP连接的建立和释放分别需要进行3次握手和4次挥手,频繁的建立连接即增加了时间开销,同时频繁的创建和销毁Socket同样是对服务器端资源的浪费。所以对于需要频繁发送HTTP请求的应用,需要在客户端使用HTTP长连接。
HTTP连接是无状态的,这样很容易给我们造成HTTP连接是短连接的错觉,实际上HTTP1.1默认即是持久连接,HTTP1.0也可以通过在请求头中设置Connection:keep-alive使得连接为长连接。既然HTTP协议支持长连接,我们就有理由相信HTTP连接同样需要连接池技术来管理和维护连接的建立和销毁。HTTP Client4.0的ThreadSafeClientConnManager实现了HTTP连接的池化管理,其管理连接的基本单位是Route(路由),每个路由上都会维护一定数量的HTTP连接。这里的Route的概念可以理解为客户端机器到目标机器的一条线路,例如使用HttpClient的实现来分别请求 www.163.com 的资源和 www.sina.com 的资源就会产生两个route。缺省条件下对于每个Route,HttpClient仅维护2个连接,总数不超过20个连接,显然对于大多数应用来讲,都是不够用的,可以通过设置HTTP参数进行调整。
两个主机建立连接的过程是很复杂的一个过程,涉及到多个数据包的交换,并且也很耗时间。Http连接需要的三次握手开销很大,这一开销对于比较小的http消息来说更大。但是如果我们直接使用已经建立好的http连接,这样花费就比较小,吞吐率更大。 传统的HttpURLConnection并不支持连接池,如果要实现连接池的机制,还需要自己来管理连接对象。对于网络请求这种底层相对复杂的操作,个人以为如果有可用的其他方案,也没有必要自己去管理连接对象。除了HttpURLConnection,大家肯定还知道HttpClient。一般情况下,普通使用HttpClient已经能满足我们的需求,不过有时候,在我们需要高并发大量的请求网络的时候,还是用“连接池”这样的概念能提升吞吐量。
我们来看下怎么使用 org.apache.httpcomponents.httpclient(版本4.4)提供的连接池来实现我们的高并发网络请求。
使用到的jar包:
org\apache\httpcomponents\httpclient\4.4-beta1\httpclient-4.4-beta1.jar
org\apache\httpcomponents\httpclient-cache\4.4-beta1\httpclient-cache-4.4-beta1.jar
org\apache\httpcomponents\httpcore\4.4-beta1\httpcore-4.4-beta1.jar
下面代码实例中主要使用到 PoolingHttpClientConnectionManager
package com.hvgroup.zhuhai10086.jms.utils;
import java.io.IOException;
import java.io.InterruptedIOException;
import java.io.UnsupportedEncodingException;
import java.net.UnknownHostException;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import javax.net.ssl.SSLException;
import javax.net.ssl.SSLHandshakeException;
import org.apache.http.HttpEntity;
import org.apache.http.HttpEntityEnclosingRequest;
import org.apache.http.HttpHost;
import org.apache.http.HttpRequest;
import org.apache.http.NameValuePair;
import org.apache.http.NoHttpResponseException;
import org.apache.http.client.HttpRequestRetryHandler;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.client.entity.UrlEncodedFormEntity;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.client.methods.HttpRequestBase;
import org.apache.http.client.protocol.HttpClientContext;
import org.apache.http.config.Registry;
import org.apache.http.config.RegistryBuilder;
import org.apache.http.conn.ConnectTimeoutException;
import org.apache.http.conn.routing.HttpRoute;
import org.apache.http.conn.socket.ConnectionSocketFactory;
import org.apache.http.conn.socket.LayeredConnectionSocketFactory;
import org.apache.http.conn.socket.PlainConnectionSocketFactory;
import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.impl.conn.PoolingHttpClientConnectionManager;
import org.apache.http.message.BasicNameValuePair;
import org.apache.http.protocol.HttpContext;
import org.apache.http.util.EntityUtils;
/**
* HttpClient工具类
*
* @return
* @author SHANHY
* @create 2015年12月18日
*/
public class HttpClientUtil {
static final int timeOut = 10 * 1000;
private static CloseableHttpClient httpClient = null;
private final static Object syncLock = new Object();
private static void config(HttpRequestBase httpRequestBase) {
// 设置Header等
// httpRequestBase.setHeader("User-Agent", "Mozilla/5.0");
// httpRequestBase
// .setHeader("Accept",
// "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
// httpRequestBase.setHeader("Accept-Language",
// "zh-CN,zh;q=0.8,en-US;q=0.5,en;q=0.3");// "en-US,en;q=0.5");
// httpRequestBase.setHeader("Accept-Charset",
// "ISO-8859-1,utf-8,gbk,gb2312;q=0.7,*;q=0.7");
// 配置请求的超时设置
RequestConfig requestConfig = RequestConfig.custom()
.setConnectionRequestTimeout(timeOut)
.setConnectTimeout(timeOut).setSocketTimeout(timeOut).build();
httpRequestBase.setConfig(requestConfig);
}
/**
* 获取HttpClient对象
*
* @return
* @author SHANHY
* @create 2015年12月18日
*/
public static CloseableHttpClient getHttpClient(String url) {
String hostname = url.split("/")[2];
int port = 80;
if (hostname.contains(":")) {
String[] arr = hostname.split(":");
hostname = arr[0];
port = Integer.parseInt(arr[1]);
}
if (httpClient == null) {
synchronized (syncLock) {
if (httpClient == null) {
httpClient = createHttpClient(200, 40, 100, hostname, port);
}
}
}
return httpClient;
}
/**
* 创建HttpClient对象
*
* @return
* @author SHANHY
* @create 2015年12月18日
*/
public static CloseableHttpClient createHttpClient(int maxTotal,
int maxPerRoute, int maxRoute, String hostname, int port) {
ConnectionSocketFactory plainsf = PlainConnectionSocketFactory
.getSocketFactory();
LayeredConnectionSocketFactory sslsf = SSLConnectionSocketFactory
.getSocketFactory();
Registry<ConnectionSocketFactory> registry = RegistryBuilder
.<ConnectionSocketFactory> create().register("http", plainsf)
.register("https", sslsf).build();
PoolingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager(
registry);
// 将最大连接数增加
cm.setMaxTotal(maxTotal);
// 将每个路由基础的连接增加
cm.setDefaultMaxPerRoute(maxPerRoute);
HttpHost httpHost = new HttpHost(hostname, port);
// 将目标主机的最大连接数增加
cm.setMaxPerRoute(new HttpRoute(httpHost), maxRoute);
// 请求重试处理
HttpRequestRetryHandler httpRequestRetryHandler = new HttpRequestRetryHandler() {
public boolean retryRequest(IOException exception,
int executionCount, HttpContext context) {
if (executionCount >= 5) {// 如果已经重试了5次,就放弃
return false;
}
if (exception instanceof NoHttpResponseException) {// 如果服务器丢掉了连接,那么就重试
return true;
}
if (exception instanceof SSLHandshakeException) {// 不要重试SSL握手异常
return false;
}
if (exception instanceof InterruptedIOException) {// 超时
return false;
}
if (exception instanceof UnknownHostException) {// 目标服务器不可达
return false;
}
if (exception instanceof ConnectTimeoutException) {// 连接被拒绝
return false;
}
if (exception instanceof SSLException) {// SSL握手异常
return false;
}
HttpClientContext clientContext = HttpClientContext
.adapt(context);
HttpRequest request = clientContext.getRequest();
// 如果请求是幂等的,就再次尝试
if (!(request instanceof HttpEntityEnclosingRequest)) {
return true;
}
return false;
}
};
CloseableHttpClient httpClient = HttpClients.custom()
.setConnectionManager(cm)
.setRetryHandler(httpRequestRetryHandler).build();
return httpClient;
}
private static void setPostParams(HttpPost httpost,
Map<String, Object> params) {
List<NameValuePair> nvps = new ArrayList<NameValuePair>();
Set<String> keySet = params.keySet();
for (String key : keySet) {
nvps.add(new BasicNameValuePair(key, params.get(key).toString()));
}
try {
httpost.setEntity(new UrlEncodedFormEntity(nvps, "UTF-8"));
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
}
}
/**
* GET请求URL获取内容
*
* @param url
* @return
* @author SHANHY
* @throws IOException
* @create 2015年12月18日
*/
public static String post(String url, Map<String, Object> params) throws IOException {
HttpPost httppost = new HttpPost(url);
config(httppost);
setPostParams(httppost, params);
CloseableHttpResponse response = null;
try {
response = getHttpClient(url).execute(httppost,
HttpClientContext.create());
HttpEntity entity = response.getEntity();
String result = EntityUtils.toString(entity, "utf-8");
EntityUtils.consume(entity);
return result;
} catch (Exception e) {
// e.printStackTrace();
throw e;
} finally {
try {
if (response != null)
response.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
/**
* GET请求URL获取内容
*
* @param url
* @return
* @author SHANHY
* @create 2015年12月18日
*/
public static String get(String url) {
HttpGet httpget = new HttpGet(url);
config(httpget);
CloseableHttpResponse response = null;
try {
response = getHttpClient(url).execute(httpget,
HttpClientContext.create());
HttpEntity entity = response.getEntity();
String result = EntityUtils.toString(entity, "utf-8");
EntityUtils.consume(entity); //关闭HttpEntity是的流,如果手动关闭了InputStream instream = entity.getContent();这个流,也可以不调用这个方法
return result;
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
if (response != null)
response.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return null;
}
public static void main(String[] args) {
// URL列表数组
String[] urisToGet = {
"http://blog.csdn.net/catoop/article/details/38849497",
"http://blog.csdn.net/catoop/article/details/38849497",
"http://blog.csdn.net/catoop/article/details/38849497",
"http://blog.csdn.net/catoop/article/details/38849497",
"http://blog.csdn.net/catoop/article/details/38849497",
"http://blog.csdn.net/catoop/article/details/38849497",
"http://blog.csdn.net/catoop/article/details/38849497",
"http://blog.csdn.net/catoop/article/details/38849497",
"http://blog.csdn.net/catoop/article/details/38849497",
"http://blog.csdn.net/catoop/article/details/38849497",
"http://blog.csdn.net/catoop/article/details/38849497",
"http://blog.csdn.net/catoop/article/details/38849497",
"http://blog.csdn.net/catoop/article/details/38849497",
"http://blog.csdn.net/catoop/article/details/38849497",
"http://blog.csdn.net/catoop/article/details/38849497",
"http://blog.csdn.net/catoop/article/details/38849497",
"http://blog.csdn.net/catoop/article/details/38849497",
"http://blog.csdn.net/catoop/article/details/38849497",
"http://blog.csdn.net/catoop/article/details/38849497",
"http://blog.csdn.net/catoop/article/details/38849497",
"http://blog.csdn.net/catoop/article/details/38849497",
"http://blog.csdn.net/catoop/article/details/38849497",
"http://blog.csdn.net/catoop/article/details/38849497",
"http://blog.csdn.net/catoop/article/details/38849497" };
long start = System.currentTimeMillis();
try {
int pagecount = urisToGet.length;
ExecutorService executors = Executors.newFixedThreadPool(pagecount);
CountDownLatch countDownLatch = new CountDownLatch(pagecount);
for (int i = 0; i < pagecount; i++) {
HttpGet httpget = new HttpGet(urisToGet[i]);
config(httpget);
// 启动线程抓取
executors
.execute(new GetRunnable(urisToGet[i], countDownLatch));
}
countDownLatch.await();
executors.shutdown();
} catch (InterruptedException e) {
e.printStackTrace();
} finally {
System.out.println("线程" + Thread.currentThread().getName() + ","
+ System.currentTimeMillis() + ", 所有线程已完成,开始进入下一步!");
}
long end = System.currentTimeMillis();
System.out.println("consume -> " + (end - start));
}
static class GetRunnable implements Runnable {
private CountDownLatch countDownLatch;
private String url;
public GetRunnable(String url, CountDownLatch countDownLatch) {
this.url = url;
this.countDownLatch = countDownLatch;
}
@Override
public void run() {
try {
System.out.println(HttpClientUtil.get(url));
} finally {
countDownLatch.countDown();
}
}
}
}
Post使用方法
// 其中 params 为 Map params
String ret = HttpClientUtil.post(url, params);
jsonRet = new JSONObject(ret);
一开始我是使用传统的 HttpURLConnection 来做网络请求的,查了很多资料,有不少说 HttpURLConnection 效率高的。可是经过我修改实现方法后,HttpClient 连接池版本的网络请求相对比较稳定。这也说明,我们并不能尽信他人解说,凡事还是要寻找适合自己的方法,真正的解决自己的问题,才是王道。
在使用 HttpURLConnection 的时候,大并发对外做网络请求的时候,前期请求耗时还好,后面耗时越来越高。下面是我之前的实现代码:
@Deprecated
protected JSONObject callRestfulOld(String url, Map<String, Object> params)
{
String temp;
String ret="";
JSONObject jsonRet=null;
String sign = generateSign("POST", url, params);// 对参数进行加密签名
if(sign.isEmpty()) return new JSONObject("{\"ret_code\":-1,\"err_msg\":\"generateSign error\"}");
params.put("sign", sign);
try{
URL u = new URL(url);
HttpURLConnection conn = (HttpURLConnection)u.openConnection();
conn.setRequestMethod("POST");
conn.setConnectTimeout(10000);
conn.setDoOutput(true);
conn.setDoInput(true);
conn.setUseCaches(false);
StringBuffer param = new StringBuffer();
for (String key: params.keySet())
{
param.append(key).append("=").append(URLEncoder.encode(params.get(key).toString(), "UTF-8")).append("&");
}
conn.getOutputStream().write(param.toString().getBytes("UTF-8"));
//System.out.println(param);
conn.getOutputStream().flush();
conn.getOutputStream().close();
InputStreamReader isr = new InputStreamReader(conn.getInputStream());
BufferedReader br = new BufferedReader(isr);
while((temp = br.readLine()) != null){
ret += temp;
}
br.close();
isr.close();
conn.disconnect();
//System.out.println(ret);
jsonRet = new JSONObject(ret);
} catch(java.net.SocketTimeoutException e) {
//e.printStackTrace();
jsonRet = new JSONObject("{\"ret_code\":-1,\"err_msg\":\"call restful timeout\"}");
} catch(Exception e) {
//e.printStackTrace();
jsonRet = new JSONObject("{\"ret_code\":-1,\"err_msg\":\"call restful error\"}");
}
return jsonRet;
}
只有一个httpclient的实例,你可以看看CloseableHttpClient和PoolingHttpClientConnectionManager的源码,你会发现httpclient实例通过execute执行get或post获取连接的时候,会通过实例关联的connectionManager的connect()来建立连接,这里的connectionManager是一个连接池的实现, 因此connect方法调用的是PoolingHttpClientConnectionManager重写的实现,这就是一个连接池建立连接的实现,具体还是看源码吧
HttpClient 3.x,4.x都提供了http连接池管理器,当使用了请求连接池管理器(比如PoolingHttpClientConnectionManager)后,HttpClient就可以同时执行多个线程的请求了。
hc3.x和4.x的早期版本,提供了PoolingClientConnectionManager,DefaultHttpClient等类来实现http连接池,但这些类在4.3.x版本之后大部分就已经过时,本文使用4.3.x提供的最新的PoolingHttpClientConnectionManager等类进行http连接池的实现.
废话不多说,下面是全部代码:
public class PoolTest {
private static void config(HttpRequestBase httpRequestBase) {
httpRequestBase.setHeader("User-Agent", "Mozilla/5.0");
httpRequestBase.setHeader("Accept","text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
httpRequestBase.setHeader("Accept-Language", "zh-CN,zh;q=0.8,en-US;q=0.5,en;q=0.3");//"en-US,en;q=0.5");
httpRequestBase.setHeader("Accept-Charset", "ISO-8859-1,utf-8,gbk,gb2312;q=0.7,*;q=0.7");
// 配置请求的超时设置
RequestConfig requestConfig = RequestConfig.custom()
.setConnectionRequestTimeout(3000)
.setConnectTimeout(3000)
.setSocketTimeout(3000)
.build();
httpRequestBase.setConfig(requestConfig);
}
public static void main(String[] args) {
ConnectionSocketFactory plainsf = PlainConnectionSocketFactory.getSocketFactory();
LayeredConnectionSocketFactory sslsf = SSLConnectionSocketFactory.getSocketFactory();
Registry<ConnectionSocketFactory> registry = RegistryBuilder.<ConnectionSocketFactory>create()
.register("http", plainsf)
.register("https", sslsf)
.build();
PoolingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager(registry);
// 将最大连接数增加到200
cm.setMaxTotal(200);
// 将每个路由基础的连接增加到20
cm.setDefaultMaxPerRoute(20);
// 将目标主机的最大连接数增加到50
HttpHost localhost = new HttpHost("http://blog.csdn.net/gaolu",80);
cm.setMaxPerRoute(new HttpRoute(localhost), 50);
//请求重试处理
HttpRequestRetryHandler httpRequestRetryHandler = new HttpRequestRetryHandler() {
public boolean retryRequest(IOException exception,int executionCount, HttpContext context) {
if (executionCount >= 5) {// 如果已经重试了5次,就放弃
return false;
}
if (exception instanceof NoHttpResponseException) {// 如果服务器丢掉了连接,那么就重试
return true;
}
if (exception instanceof SSLHandshakeException) {// 不要重试SSL握手异常
return false;
}
if (exception instanceof InterruptedIOException) {// 超时
return false;
}
if (exception instanceof UnknownHostException) {// 目标服务器不可达
return false;
}
if (exception instanceof ConnectTimeoutException) {// 连接被拒绝
return false;
}
if (exception instanceof SSLException) {// ssl握手异常
return false;
}
HttpClientContext clientContext = HttpClientContext.adapt(context);
HttpRequest request = clientContext.getRequest();
// 如果请求是幂等的,就再次尝试
if (!(request instanceof HttpEntityEnclosingRequest)) {
return true;
}
return false;
}
};
CloseableHttpClient httpClient = HttpClients.custom()
.setConnectionManager(cm)
.setRetryHandler(httpRequestRetryHandler)
.build();
// URL列表数组
String[] urisToGet = {
"http://blog.csdn.net/gaolu/article/details/48466059",
"http://blog.csdn.net/gaolu/article/details/48243103",
"http://blog.csdn.net/gaolu/article/details/47656987",
"http://blog.csdn.net/gaolu/article/details/47055029",
"http://blog.csdn.net/gaolu/article/details/46400883",
"http://blog.csdn.net/gaolu/article/details/46359127",
"http://blog.csdn.net/gaolu/article/details/46224821",
"http://blog.csdn.net/gaolu/article/details/45305769",
"http://blog.csdn.net/gaolu/article/details/43701763",
"http://blog.csdn.net/gaolu/article/details/43195449",
"http://blog.csdn.net/gaolu/article/details/42915521",
"http://blog.csdn.net/gaolu/article/details/41802319",
"http://blog.csdn.net/gaolu/article/details/41045233",
"http://blog.csdn.net/gaolu/article/details/40395425",
"http://blog.csdn.net/gaolu/article/details/40047065",
"http://blog.csdn.net/gaolu/article/details/39891877",
"http://blog.csdn.net/gaolu/article/details/39499073",
"http://blog.csdn.net/gaolu/article/details/39314327",
"http://blog.csdn.net/gaolu/article/details/38820809",
"http://blog.csdn.net/gaolu/article/details/38439375",
};
long start = System.currentTimeMillis();
try {
int pagecount = urisToGet.length;
ExecutorService executors = Executors.newFixedThreadPool(pagecount);
CountDownLatch countDownLatch = new CountDownLatch(pagecount);
for(int i = 0; i< pagecount;i++){
HttpGet httpget = new HttpGet(urisToGet[i]);
config(httpget);
//启动线程抓取
executors.execute(new GetRunnable(httpClient,httpget,countDownLatch));
}
countDownLatch.await();
executors.shutdown();
} catch (InterruptedException e) {
e.printStackTrace();
} finally {
System.out.println("线程" + Thread.currentThread().getName() + "," + System.currentTimeMillis() + ", 所有线程已完成,开始进入下一步!");
}
long end = System.currentTimeMillis();
System.out.println("consume -> " + (end - start));
}
static class GetRunnable implements Runnable {
private CountDownLatch countDownLatch;
private final CloseableHttpClient httpClient;
private final HttpGet httpget;
public GetRunnable(CloseableHttpClient httpClient, HttpGet httpget, CountDownLatch countDownLatch){
this.httpClient = httpClient;
this.httpget = httpget;
this.countDownLatch = countDownLatch;
}
@Override
public void run() {
CloseableHttpResponse response = null;
try {
response = httpClient.execute(httpget,HttpClientContext.create());
HttpEntity entity = response.getEntity();
System.out.println(EntityUtils.toString(entity, "utf-8")) ;
EntityUtils.consume(entity);
} catch (IOException e) {
e.printStackTrace();
} finally {
countDownLatch.countDown();
try {
if(response != null)
response.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
}