爬网程序(一)：文件下载功能

1. 预备知识：

 

主要就是HttpWebRequest和HttpWebResponse这两个类的用法，请参考MSDN学习，这里不赘述。

http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.aspx

http://msdn.microsoft.com/zh-cn/library/0h6wsc8k(v=VS.80).aspx

 

2. 废话不多说，直接上程序，详情见注释：

 

(1)Win32.cs文件。里面主要放需要调用的Win32 API.

using System; using System.Collections.Generic; using System.Linq; using System.Text; using System.Runtime.InteropServices; namespace KQ.CrawlRobert.Robert { public class Win32 { //获取url的cookie. [DllImport("wininet.dll", SetLastError = true, CharSet = CharSet.Auto)] public static extern bool InternetGetCookie(string lpszUrlName, string lpszCookieName, StringBuilder lpszCookieData, ref int lpdwSize); } }

 

(2)RobertUtility.cs。放爬网程序需要的一些静态辅助函数。

using System; using System.Collections.Generic; using System.Linq; using System.Text; using System.Net; namespace KQ.CrawlRobert.Robert { public class RobertUtility { //获取CookieContainer. public static CookieContainer GetCookieContainer(Uri url) { CookieContainer container = new CookieContainer(); string cookieHeaders = GetCookie(url.AbsoluteUri); if (cookieHeaders.Length > 0) { try { container.SetCookies(url, cookieHeaders); } catch (CookieException) { } } return container; } //获取cookie private static string GetCookie(string url) { StringBuilder cookieHeader = new StringBuilder(new String(' ', 256), 256); int datasize = cookieHeader.Length; if (!Win32.InternetGetCookie(url, null, cookieHeader, ref datasize)) { if (datasize < 0) { return String.Empty; } cookieHeader = new StringBuilder(datasize); Win32.InternetGetCookie(url, null, cookieHeader, ref datasize); } return cookieHeader.ToString(); } } }

(2)RoberConsts。