HttpUrlConnection底层实现和关于java host绑定ip即时生效的设置及分析
最近有个需求需要对于获取URL页面进行host绑定并且立即生效，在java里面实现可以用代理服务器来实现：因为在测试环境下可能需要通过绑定来访问测试环境的应用
实现代码如下：
public staticString getResponseText(String queryUrl,String host,String ip) { //queryUrl，完整的url，host和ip需要绑定的host和ip
InputStream is = null;
BufferedReader br = null;
StringBuffer res = new StringBuffer();
try {
HttpURLConnection httpUrlConn= null;
URL url = new URL(queryUrl);
if(ip!=null){
String str[] = ip.split("\\.");
byte[] b =new byte[str.length];
for(inti=0,len=str.length;i b[i] = (byte)(Integer.parseInt(str[i],10)); } Proxy proxy = new Proxy(Proxy.Type.HTTP, newInetSocketAddress(InetAddress.getByAddress(b), 80)); //b是绑定的ip，生成proxy代理对象，因为http底层是socket实现， httpUrlConn = (HttpURLConnection) url .openConnection(proxy); }else{ httpUrlConn = (HttpURLConnection) url .openConnection(); } httpUrlConn.setRequestMethod("GET"); httpUrlConn.setDoOutput(true); httpUrlConn.setConnectTimeout(2000); httpUrlConn.setReadTimeout(2000); httpUrlConn.setDefaultUseCaches(false); httpUrlConn.setUseCaches(false); is = httpUrlConn.getInputStream(); 那么底层对于proxy对象到底是怎么处理，底层的socket实现到底怎么样，带着这个疑惑看了下jdk的rt.jar对于这块的处理 httpUrlConn = (HttpURLConnection) url.openConnection(proxy) java.net.URL类里面的openConnection方法： public URLConnection openConnection(Proxy proxy){ … return handler.openConnection(this, proxy); Handler是sun.net.www.protocol.http.Handler.java类，继承java.net. URLStreamHandler.java类，用来处理http连接请求响应的。 } Handler的方法： protectedjava.net.URLConnection openConnection(URL u, Proxy p) throws IOException { return newHttpURLConnection(u, p, this); } 只是简单的生成sun.net.www.protocl.http.HttpURLConnection对象，并进行初始化 protectedHttpURLConnection(URL u, Proxy p, Handler handler) { super(u); requests = newMessageHeader(); 请求头信息生成类 responses = newMessageHeader(); 响应头信息解析类 this.handler = handler; instProxy = p; 代理服务器对象 cookieHandler = (CookieHandler)java.security.AccessController.doPrivileged( new java.security.PrivilegedAction() { public Object run() { return CookieHandler.getDefault(); } }); cacheHandler = (ResponseCache)java.security.AccessController.doPrivileged( new java.security.PrivilegedAction() { public Object run() { return ResponseCache.getDefault(); } }); } 最终在httpUrlConn.getInputStream();才进行socket连接，发送http请求，解析http响应信息。具体过程如下： sun.net.www.protocl.http.HttpURLConnection.java的getInputStream方法： public synchronized InputStream getInputStream() throws IOException { ...socket连接 connect(); ... ps = (PrintStream)http.getOutputStream(); 获得输出流，打开连接之后已经生成。 if (!streaming()) { writeRequests(); 输出http请求头信息 } ... http.parseHTTP(responses, pi, this); 解析响应信息 if(logger.isLoggable(Level.FINEST)) { logger.fine(responses.toString()); } inputStream = http.getInputStream(); 获得输入流 } 其中connect()调用方法链： plainConnect(){ ... Proxy p = null; if (sel != null) { URI uri = sun.net.www.ParseUtil.toURI(url); Iterator while (it.hasNext()) { p = it.next(); try { if (!failedOnce) { http = getNewHttpClient(url, p, connectTimeout); ... } getNewHttpClient(){ ... return HttpClient.New(url, p, connectTimeout, useCache); ... } 下面跟进去最终建立socket连接的代码： sun.net.www.http.HttpClient.java的openServer()方法建立socket连接： protected synchronized void openServer() throwsIOException { ... if ((proxy != null)&& (proxy.type() == Proxy.Type.HTTP)) { sun.net.www.URLConnection.setProxiedHost(host); if (security != null) { security.checkConnect(host, port); } privilegedOpenServer((InetSocketAddress) proxy.address());最终socket连接的是设置的代理服务器的地址， ... } private synchronized voidprivilegedOpenServer(finalInetSocketAddress server) throws IOException { try { java.security.AccessController.doPrivileged( newjava.security.PrivilegedExceptionAction() { public Object run() throwsIOException { openServer(server.getHostName(), server.getPort()); 注意openserver函数 这里的server的getHostName是设置的代理服务器，（ip或者hostname，如果是host绑定设置的代理服务器的ip，那么这里getHostName出来的就是ip地址，可以去查看InetSocketAddress类的getHostName方法） return null; } }); } catch(java.security.PrivilegedActionException pae) { throw (IOException) pae.getException(); } } public void openServer(String server, int port) throwsIOException { serverSocket = doConnect(server, port); 生成的Socket连接对象 try { serverOutput = newPrintStream( new BufferedOutputStream(serverSocket.getOutputStream()), false, encoding); 生成输出流， } catch (UnsupportedEncodingException e) { throw newInternalError(encoding+" encoding not found"); } serverSocket.setTcpNoDelay(true); } protectedSocket doConnect (String server, int port) throws IOException, UnknownHostException { Socket s; if (proxy != null) { if (proxy.type() == Proxy.Type.SOCKS) { s = (Socket) AccessController.doPrivileged( new PrivilegedAction() { public Object run() { return newSocket(proxy); }}); } else s = new Socket(Proxy.NO_PROXY); } else s = new Socket(); // Instance specific timeouts do have priority, that means // connectTimeout & readTimeout (-1 means not set) // Then global default timeouts // Then no timeout. if (connectTimeout> = 0) { s.connect(new InetSocketAddress(server, port), connectTimeout); } else { if (defaultConnectTimeout > 0) { s.connect(new InetSocketAddress(server, port), defaultConnectTimeout);//连接到代理服务器，看下面Socket类的connect方法代码 } else { s.connect(new InetSocketAddress(server, port)); } } if (readTimeout> = 0) s.setSoTimeout(readTimeout); else if (defaultSoTimeout > 0) { s.setSoTimeout(defaultSoTimeout); } return s; } 上面的new InetSocketAddress(server, port)这里会涉及到java DNS cache的处理， public InetSocketAddress(String hostname, int port) { if (port < 0 || port > 0xFFFF) { throw newIllegalArgumentException("port out of range:" + port); } if (hostname == null) { throw newIllegalArgumentException("hostname can't be null"); } try { addr = InetAddress.getByName(hostname); //这里会有java DNS缓存的处理，先从缓存取hostname绑定的ip地址，如果取不到再通过OS的DNS cache机制去取，取不到再从DNS服务器上取。 } catch(UnknownHostException e) { this.hostname = hostname; addr = null; } this.port = port; } 当然最终的Socket.java的connect方法 java.net.socket public voidconnect(SocketAddress endpoint, int timeout) throwsIOException { if (endpoint == null) if(timeout < 0) throw newIllegalArgumentException("connect: timeout can't be negative"); if (isClosed()) throw newSocketException("Socket is closed"); if (!oldImpl&& isConnected()) throw newSocketException("already connected"); if (!(endpoint instanceofInetSocketAddress)) throw newIllegalArgumentException("Unsupported address type"); InetSocketAddress epoint = (InetSocketAddress) endpoint; SecurityManager security = System.getSecurityManager(); if (security != null) { if (epoint.isUnresolved()) security.checkConnect(epoint.getHostName(), epoint.getPort()); else security.checkConnect(epoint.getAddress().getHostAddress(), epoint.getPort()); } if (!created) createImpl(true); if (!oldImpl) impl.connect(epoint, timeout); else if(timeout == 0) { if (epoint.isUnresolved()) //如果没有设置SocketAddress的ip地址，则用域名去访问 impl.connect(epoint.getAddress().getHostName(), epoint.getPort()); else impl.connect(epoint.getAddress(), epoint.getPort()); 最终socket连接的是设置的SocketAddress的ip地址， } else throw newUnsupportedOperationException("SocketImpl.connect(addr, timeout)"); connected = true; /* * If the socket was not bound before the connect, it is now because * the kernel will have picked an ephemeral port & a local address */ bound = true; } 我们再看下通过socket来发送HTTP请求的处理代码，也就是sun.net.www.protocl.http.HttpURLConnection.java的getInputStream方法中调用的writeRequests()方法： private voidwriteRequests() throws IOException { 这段代码就是封装http请求的头请求信息，通过socket发送出去 /* print all message headers in the MessageHeader * onto the wire - all the ones we've set and any * others that have been set */ // send any pre-emptive authentication if (http.usingProxy) { setPreemptiveProxyAuthentication(requests); } if (!setRequests) { /* We're very particular about the order in which we * set the request headers here. The order should not * matter, but some careless CGI programs have been * written to expect a very particular order of the * standard headers. To name names, the order in which * Navigator3.0 sends them. In particular, we make *sure* * to send Content-type: <> and Content-length:<> second * to last and last, respectively, in the case of a POST * request. */ if (!failedOnce) requests.prepend(method + " " + http.getURLFile()+" " + httpVersion, null); if (!getUseCaches()) { requests.setIfNotSet ("Cache-Control", "no-cache"); requests.setIfNotSet ("Pragma", "no-cache"); } requests.setIfNotSet("User-Agent", userAgent); int port = url.getPort(); String host = url.getHost(); if (port != -1 && port != url.getDefaultPort()) { host += ":" + String.valueOf(port); } requests.setIfNotSet("Host", host); requests.setIfNotSet("Accept", acceptString); /* * For HTTP/1.1 the default behavior is to keep connections alive. * However, we may be talking to a 1.0 server so we should set * keep-alive just in case, except if we have encountered an error * or if keep alive is disabled via a system property */ // Try keep-alive only on first attempt if (!failedOnce&& http.getHttpKeepAliveSet()) { if (http.usingProxy) { requests.setIfNotSet("Proxy-Connection", "keep-alive"); } else { requests.setIfNotSet("Connection", "keep-alive"); } } else { /* * RFC 2616 HTTP/1.1 section 14.10 says: * HTTP/1.1 applications that do not support persistent * connections MUST include the "close" connection option * in every message */ requests.setIfNotSet("Connection", "close"); } // Set modified since if necessary long modTime = getIfModifiedSince(); if (modTime != 0 ) { Date date = new Date(modTime); //use the preferred date format according to RFC 2068(HTTP1.1), // RFC 822 and RFC 1123 SimpleDateFormat fo = new SimpleDateFormat ("EEE, dd MMM yyyy HH:mm:ss 'GMT'", Locale.US); fo.setTimeZone(TimeZone.getTimeZone("GMT")); requests.setIfNotSet("If-Modified-Since", fo.format(date)); } // check for preemptive authorization AuthenticationInfo sauth = AuthenticationInfo.getServerAuth(url); if (sauth != null&& sauth.supportsPreemptiveAuthorization() ) { // Sets "Authorization" requests.setIfNotSet(sauth.getHeaderName(), sauth.getHeaderValue(url,method)); currentServerCredentials = sauth; } if (!method.equals("PUT")&& (poster != null || streaming())) { requests.setIfNotSet ("Content-type", "application/x-www-form-urlencoded"); } if (streaming()) { if (chunkLength !=-1) { requests.set ("Transfer-Encoding", "chunked"); } else { requests.set ("Content-Length", String.valueOf(fixedContentLength)); } } else if (poster != null) { /* add Content-Length & POST/PUT data */ synchronized (poster) { /* close it, so no more data can be added */ poster.close(); requests.set("Content-Length", String.valueOf(poster.size())); } } // get applicable cookies based on the uri and request headers // add them to the existing request headers setCookieHeader(); … } 再来看看把socket响应信息解析为http的响应信息的代码： sun.net.www.http.HttpClient.java的parseHTTP方法： private boolean parseHTTPHeader(MessageHeader responses, ProgressSource pi, HttpURLConnection httpuc) throws IOException { /* If "HTTP/*" is found in the beginning, return true. Let * HttpURLConnection parse the mime header itself. * * If this isn't valid HTTP, then we don't try to parse a header * out of the beginning of the response into the responses, * and instead just queue up the output stream to it's very beginning. * This seems most reasonable, and is what the NN browser does. */ keepAliveConnections = -1; keepAliveTimeout = 0; boolean ret = false; byte[] b = new byte[8]; try { int nread = 0; serverInput.mark(10); while (nread < 8) { int r = serverInput.read(b, nread, 8 - nread); if (r < 0) { break; } nread += r; } String keep=null; ret = b[0] == 'H' && b[1] == 'T' && b[2] == 'T' && b[3] == 'P'&& b[4] == '/' && b[5] == '1' && b[6] == '.'; serverInput.reset(); if (ret) { // is valid HTTP - response started w/ "HTTP/1." responses.parseHeader(serverInput); // we've finished parsing http headers // check if there are any applicable cookies to set (in cache) if (cookieHandler != null) { URI uri = ParseUtil.toURI(url); // NOTE: That cast from Map shouldn't be necessary but // a bug in javac is triggered under certain circumstances // So we do put the cast in as a workaround until // it is resolved. if (uri != null) cookieHandler.put(uri, (Map } /* decide if we're keeping alive: * This is a bit tricky. There's a spec, but most current * servers (10/1/96) that support this differ in dialects. * If the server/client misunderstand each other, the * protocol should fall back onto HTTP/1.0, no keep-alive. */ if (usingProxy) { // not likely a proxy will return this keep = responses.findValue("Proxy-Connection"); } if (keep == null) { keep = responses.findValue("Connection"); } if (keep != null&& keep.toLowerCase().equals("keep-alive")) { /* some servers, notably Apache1.1, send something like: * "Keep-Alive: timeout=15, max=1" which we should respect. */ HeaderParser p = new HeaderParser( responses.findValue("Keep-Alive")); if (p != null) { /* default should be larger in case of proxy */ keepAliveConnections = p.findInt("max", usingProxy?50:5); keepAliveTimeout = p.findInt("timeout", usingProxy?60:5); } } else if(b[7] != '0') { /* * We're talking 1.1 or later. Keep persistent until * the server says to close. */ if (keep != null) { /* * The only Connection token we understand is close. * Paranoia: if there is any Connection header then * treat as non-persistent. */ keepAliveConnections = 1; } else { keepAliveConnections = 5; } } …… } 对于java.net包的http，ftp等各种协议的底层实现，可以参考rt.jar下面的几个包的代码： sun.net.www.protocl下的几个包。 在http client中也可以设置代理： HostConfiguration conf = newHostConfiguration(); conf.setHost(host); conf.setProxy(ip, 80); statusCode = httpclient.executeMethod(conf,getMethod); httpclient自己也是基于socket封装的http处理的库。底层代理的实现是一样的。 另外一种不设置代理，通过反射修改InetAddress的cache也是ok的。但是这种方法非常不推荐，不要使用，因为对于proxy代理服务器概念了解不清楚，最开始还使用这种方法， public static voidjdkDnsNoCache(final String host, finalString ip) throws SecurityException, NoSuchFieldException, IllegalArgumentException, IllegalAccessException { if (StringUtils.isBlank(host)) { return; } final Class clazz = java.net.InetAddress.class; final Field cacheField = clazz.getDeclaredField("addressCache"); cacheField.setAccessible(true); final Object o = cacheField.get(clazz); Class clazz2 = o.getClass(); final Field cacheMapField = clazz2.getDeclaredField("cache"); cacheMapField.setAccessible(true); final Map cacheMap = (Map) cacheMapField.get(o); AccessController.doPrivileged(newPrivilegedAction() { public Object run() { try { synchronized (o) {// 同步是必须的,因为o可能会有多个线程同时访问修改。 // cacheMap.clear();//这步比较关键，用于清除原来的缓存 // cacheMap.remove(host); if (!StringUtils.isBlank(ip)) { InetAddress inet = InetAddress.getByAddress(host,IPUtil.int2byte(ip)); InetAddress addressstart = InetAddress.getByName(host); Object cacheEntry = cacheMap.get(host); cacheMap.put(host,newCacheEntry(inet,cacheEntry)); // cacheMap.put(host,newCacheEntry(newInetAddress(host, ip))); }else{ cacheMap.remove(host); } // System.out.println(getStaticProperty( // "java.net.InetAddress", "addressCacheInit")); // System.out.println(invokeStaticMethod("java.net.InetAddress","getCachedAddress",new // Object[]{host})); } } catch (Throwable te) { throw newRuntimeException(te); } return null; } }); final Map cacheMapafter = (Map) cacheMapField.get(o); System.out.println(cacheMapafter); } 关于java中对于DNS的缓存设置可以参考： 1.在${java_home}/jre/lib/secuiry/java.secuiry文件，修改下面为 networkaddress.cache.negative.ttl=0 DNS解析不成功的缓存时间 networkaddress.cache.ttl=0 DNS解析成功的缓存的时间 2.jvm启动时增加下面两个启动环境变量 -Dsun.net.inetaddr.ttl=0 -Dsun.net.inetaddr.negative.ttl=0 如果在java程序中使用，可以这么设置设置： java.security.Security.setProperty("networkaddress.cache.ttl", "0"); java.security.Security.setProperty("networkaddress.cache.negative.ttl" , "0"); 还有几篇文档链接可以查看： http://www.rgagnon.com/javadetails/java-0445.html http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6247501
linux下关于ＯＳ ＤＮＳ设置的几个文件是 /etc/resolve.conf /etc/nscd.conf /etc/nsswitch.conf http://www.linuxfly.org/post/543/ http://linux.die.net/man/5/nscd.conf http://www.linuxhomenetworking.com/wiki/index.php/Quick_HOWTO_:_Ch18_:_Configuring_DNS http://linux.die.net/man/5/nscd.conf 转载自：http://zhwj184.iteye.com/blog/1240765