agent代理解决网站屏蔽

视频地址 https://www.bilibili.com/video/av31369828/?p=10
agent参考链接 http://tools.jb51.net/table/useragent

浏览器agent

1. PC：

• safari5.1–MAC
  User-Agent:Mozilla/5.0(Macintosh;U;IntelMacOSX10_6_8;en-us)AppleWebKit/534.50(KHTML,likeGecko)Version/5.1Safari/534.50
• safari5.1–Windows
  User-Agent:Mozilla/5.0(Windows;U;WindowsNT6.1;en-us)AppleWebKit/534.50(KHTML,likeGecko)Version/5.1Safari/534.50

2. IE9.0
   User-Agent:Mozilla/5.0(compatible;MSIE9.0;WindowsNT6.1;Trident/5.0
3. IE8.0
   User-Agent:Mozilla/4.0(compatible;MSIE8.0;WindowsNT6.0;Trident/4.0)
4. IE7.0
   User-Agent:Mozilla/4.0(compatible;MSIE7.0;WindowsNT6.0)
5. IE6.0
   User-Agent:Mozilla/4.0(compatible;MSIE6.0;WindowsNT5.1)
6. Firefox4.0.1–MAC
   User-Agent:Mozilla/5.0(Macintosh;IntelMacOSX10.6;rv:2.0.1)Gecko/20100101Firefox/4.0.1
7. Firefox4.0.1–Windows
   User-Agent:Mozilla/5.0(WindowsNT6.1;rv:2.0.1)Gecko/20100101Firefox/4.0.1
8. Opera11.11–MAC
   User-Agent:Opera/9.80(Macintosh;IntelMacOSX10.6.8;U;en)Presto/2.8.131Version/11.11
9. Opera11.11–Windows
   User-Agent:Opera/9.80(WindowsNT6.1;U;en)Presto/2.8.131Version/11.11
10. Chrome17.0–MAC
    User-Agent:Mozilla/5.0(Macintosh;IntelMacOSX10_7_0)AppleWebKit/535.11(KHTML,likeGecko)Chrome/17.0.963.56Safari/535.11
11. 傲游（Maxthon）
    User-Agent:Mozilla/4.0(compatible;MSIE7.0;WindowsNT5.1;Maxthon2.0)
12. 腾讯TT
    User-Agent:Mozilla/4.0(compatible;MSIE7.0;WindowsNT5.1;TencentTraveler4.0)
13. 世界之窗（TheWorld）2.x
    User-Agent:Mozilla/4.0(compatible;MSIE7.0;WindowsNT5.1)
14. 世界之窗（TheWorld）3.x
    User-Agent:Mozilla/4.0(compatible;MSIE7.0;WindowsNT5.1;TheWorld)
15. 搜狗浏览器1.x
    User-Agent:Mozilla/4.0(compatible;MSIE7.0;WindowsNT5.1;Trident/4.0;SE2.XMetaSr1.0;SE2.XMetaSr1.0;.NETCLR2.0.50727;SE2.XMetaSr1.0)
16. 360浏览器
    User-Agent:Mozilla/4.0(compatible;MSIE7.0;WindowsNT5.1;360SE)
17. Avant
    User-Agent:Mozilla/4.0(compatible;MSIE7.0;WindowsNT5.1;AvantBrowser)
18. GreenBrowser
    User-Agent:Mozilla/4.0(compatible;MSIE7.0;WindowsNT5.1)

手机agent

1. safariiOS4.33–iPhone
   User-Agent:Mozilla/5.0(iPhone;U;CPUiPhoneOS4_3_3likeMacOSX;en-us)AppleWebKit/533.17.9(KHTML,likeGecko)Version/5.0.2Mobile/8J2Safari/6533.18.5
2. safariiOS4.33–iPodTouch
   User-Agent:Mozilla/5.0(iPod;U;CPUiPhoneOS4_3_3likeMacOSX;en-us)AppleWebKit/533.17.9(KHTML,likeGecko)Version/5.0.2Mobile/8J2Safari/6533.18.5
3. safariiOS4.33–iPad
   User-Agent:Mozilla/5.0(iPad;U;CPUOS4_3_3likeMacOSX;en-us)AppleWebKit/533.17.9(KHTML,likeGecko)Version/5.0.2Mobile/8J2Safari/6533.18.5
4. AndroidN1
   User-Agent:Mozilla/5.0(Linux;U;Android2.3.7;en-us;NexusOneBuild/FRF91)AppleWebKit/533.1(KHTML,likeGecko)Version/4.0MobileSafari/533.1
5. AndroidQQ浏览器Forandroid
   User-Agent:MQQBrowser/26Mozilla/5.0(Linux;U;Android2.3.7;zh-cn;MB200Build/GRJ22;CyanogenMod-7)AppleWebKit/533.1(KHTML,likeGecko)Version/4.0MobileSafari/533.1
6. AndroidOperaMobile
   User-Agent:Opera/9.80(Android2.3.4;Linux;OperaMobi/build-1107180945;U;en-GB)Presto/2.8.149Version/11.10
7. AndroidPadMotoXoom
   User-Agent:Mozilla/5.0(Linux;U;Android3.0;en-us;XoomBuild/HRI39)AppleWebKit/534.13(KHTML,likeGecko)Version/4.0Safari/534.13
8. BlackBerry
   User-Agent:Mozilla/5.0(BlackBerry;U;BlackBerry9800;en)AppleWebKit/534.1+(KHTML,likeGecko)Version/6.0.0.337MobileSafari/534.1+
9. WebOSHPTouchpad
   User-Agent:Mozilla/5.0(hp-tablet;Linux;hpwOS/3.0.0;U;en-US)AppleWebKit/534.6(KHTML,likeGecko)wOSBrowser/233.70Safari/534.6TouchPad/1.0
10. NokiaN97
    User-Agent:Mozilla/5.0(SymbianOS/9.4;Series60/5.0NokiaN97-1/20.0.019;Profile/MIDP-2.1Configuration/CLDC-1.1)AppleWebKit/525(KHTML,likeGecko)BrowserNG/7.1.18124
11. WindowsPhoneMango
    User-Agent:Mozilla/5.0(compatible;MSIE9.0;WindowsPhoneOS7.5;Trident/5.0;IEMobile/9.0;HTC;Titan)
12. UC无
    User-Agent:UCWEB7.0.2.37/28/999
13. UC标准
    User-Agent:NOKIA5700/UCWEB7.0.2.37/28/999
14. UCOpenwave
    User-Agent:Openwave/UCWEB7.0.2.37/28/999
15. UCOpera
    User-Agent:Mozilla/4.0(compatible;MSIE6.0;)Opera/UCWEB7.0.2.37/28/999

主要代码

#encoding: utf-8
import urllib2

def download(url):
  headers = {"User Agent : "}
  request = urllib2.Request(url, headers = headers) #发起请求
  data = urllib2.urlopen(request).read() #打开请求，抓取数据
  return data

url = "https://sou.zhaopin.com/?jl=613&kw=" + searchname + "&kt=3"
print download(url)