目录
- python爬虫学习-day1
- python爬虫学习-day2正则表达式
- python爬虫学习-day3-BeautifulSoup
- python爬虫学习-day4-使用lxml+xpath提取内容
- python爬虫学习-day5-selenium
- python爬虫学习-day6-ip池
- python爬虫学习-day7-实战
学习IP相关知识
学习什么是IP，为什么会出现IP被封，如何应对IP被封的问题。
抓取西刺代理，并构建自己的代理池。
西刺直通点：https://www.xicidaili.com/ 。
1. 为什么会出现IP被封，如何应对IP被封的问题。
网站为了防止被爬取，会有反爬机制，对于同一个IP地址的大量同类型的访问，会封锁IP，过一段时间后，才能继续访问
现有的反扒策略：
0. 检测浏览器header， User-Agent
1. ip 封禁
2. 图片验证码
3. 滑块
4. JS轨迹
5. 证书加密
6. AI识别
2. 如何应对IP被封
1. 建立代理IP, 轮换访问
2. 设置访问时间间隔
3. 可动态设置user agent
4. 禁用cookies
5. 设置延迟下载
6. 使用Google Cache
7. 使用IP地址池（代理IP、VPN等）
8. 使用Crawlera
参考： https://desmonday.github.io/2019/03/06/python%E7%88%AC%E8%99%AB%E5%AD%A6%E4%B9%A0-day6-IP%E4%BB%A3%E7%90%86/
3. 获取代理IP地址
网站： https://www.xicidaili.com/
示例
import requests, re
from bs4 import BeautifulSoup as bs
import json
def get_html(url):
user_agent = 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.108 Safari/537.36'
headers = {'User-Agent': user_agent}
try:
# html = requests.get(url=url, headers=headers)
r = requests.get(url, headers=headers, timeout=10)
r.raise_for_status()
r.encoding = r.apparent_encoding
return r.text
except:
print('error , not open page by url:' + url)
def get_proxy_ip(html):
html = bs(html, 'html.parser')
proxy_ips = html.find(id='ip_list').find_all('tr')
for proxy_ip in proxy_ips:
if len(proxy_ip.select('td')) > 0:
ip = proxy_ip.select('td')[1].text
port = proxy_ip.select('td')[2].text
protocol = proxy_ip.select('td')[5].text
if protocol in protocollists:
proxy_ip_list.append(f'{protocol}://{ip}:{port}')
return proxy_ip_list
def check_proxy_avaliability(ip):
url = 'https://www.baidu.com'
user_agent = 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.108 Safari/537.36'
headers = {'User-Agent': user_agent}
try:
proxies = {}
if ip.startswith(('HTTPS', 'https')):
proxies['HTTPS'] = ip
else:
proxies['HTTP'] = ip
r = requests.get(url=url, headers=headers, proxies=proxies, timeout=10)
r.raise_for_status()
r.encoding = r.apparent_encoding
text, status_code = r.text, r.status_code
if status_code == 200:
print('有效IP, %s', ip)
return True
else:
print('无效IP, %s', ip)
return False
except:
print('error ', url)
return False
if __name__ == '__main__':
proxy_ip_list = []
url = 'https://www.xicidaili.com/'
protocollists = ['http', 'https', 'HTTP', 'HTTPS']
html = get_html(url)
ips = get_proxy_ip(html)
print(ips)
use_ip_list = []
for ip in ips:
if check_proxy_avaliability(ip):
use_ip_list.append(ip)
print('有效代理ip')
print(use_ip_list)
结果
['HTTP://120.83.98.192:9999', 'HTTP://115.239.25.244:9999', 'HTTP://120.83.111.221:9999', 'HTTP://1.198.72.153:9999', 'HTTP://180.126.169.6:8118', 'HTTPS://122.137.4.230:8118', 'HTTP://163.204.244.150:9999', 'HTTPS://112.85.170.172:9999', 'HTTPS://112.85.131.34:9999', 'HTTP://112.85.171.8:9999', 'HTTPS://112.85.164.213:9999', 'HTTPS://117.95.98.120:42704', 'HTTP://115.225.49.188:8118', 'HTTPS://119.162.37.165:8118', 'HTTP://123.139.28.36:8118', 'HTTPS://171.80.3.169:9999', 'HTTPS://112.85.164.161:9999', 'HTTPS://115.53.16.222:9999', 'HTTPS://36.99.212.233:9999', 'HTTPS://113.121.46.78:9999', 'HTTPS://221.225.147.58:8118', 'HTTP://222.186.45.145:57273', 'HTTP://121.238.82.201:8118', 'HTTP://114.221.20.254:8118', 'HTTPS://58.250.23.210:1080', 'HTTPS://180.160.139.246:8118', 'HTTPS://112.80.41.86:8888', 'HTTP://218.64.69.79:8080', 'HTTPS://59.38.61.164:9797', 'HTTPS://182.18.13.149:53281', 'HTTP://113.251.221.143:8118', 'HTTP://117.90.5.64:9000', 'HTTPS://125.32.80.52:8080', 'HTTPS://58.247.127.145:53281', 'HTTPS://175.23.40.250:8080', 'HTTP://211.162.70.229:3128', 'HTTPS://182.149.157.168:8118', 'HTTPS://218.22.7.62:53281', 'HTTP://121.79.131.58:8080', 'HTTP://14.115.106.178:808', 'HTTPS://122.137.4.230:8118', 'HTTPS://112.85.170.172:9999', 'HTTPS://112.85.131.34:9999', 'HTTPS://221.225.147.58:8118', 'HTTPS://112.85.164.213:9999', 'HTTPS://117.95.98.120:42704', 'HTTPS://119.162.37.165:8118', 'HTTPS://171.80.3.169:9999', 'HTTPS://112.85.164.161:9999', 'HTTPS://115.53.16.222:9999', 'HTTPS://36.99.212.233:9999', 'HTTPS://113.121.46.78:9999', 'HTTPS://123.163.96.141:9999', 'HTTPS://222.137.4.96:8118', 'HTTPS://115.53.20.2:9999', 'HTTPS://124.94.199.204:9999', 'HTTPS://112.87.71.206:9999', 'HTTPS://120.83.105.77:9999', 'HTTPS://112.85.151.97:9999', 'HTTPS://58.250.23.210:1080', 'HTTP://120.83.98.192:9999', 'HTTP://115.239.25.244:9999', 'HTTP://120.83.111.221:9999', 'HTTP://1.198.72.153:9999', 'HTTP://180.126.169.6:8118', 'HTTP://163.204.244.150:9999', 'HTTP://112.85.171.8:9999', 'HTTP://115.225.49.188:8118', 'HTTP://222.186.45.145:57273', 'HTTP://123.139.28.36:8118', 'HTTP://112.85.169.44:9999', 'HTTP://121.238.82.201:8118', 'HTTP://1.198.72.48:9999', 'HTTP://112.85.129.140:9999', 'HTTP://171.37.157.61:8123', 'HTTP://119.162.150.192:8118', 'HTTP://114.221.20.254:8118', 'HTTP://49.86.176.110:9999', 'HTTP://114.230.69.201:9999', 'HTTP://112.255.118.2:8118']
有效IP, %s HTTP://120.83.98.192:9999
有效IP, %s HTTP://115.239.25.244:9999
有效IP, %s HTTP://120.83.111.221:9999
有效IP, %s HTTP://1.198.72.153:9999
有效IP, %s HTTP://180.126.169.6:8118
有效IP, %s HTTPS://122.137.4.230:8118
有效IP, %s HTTP://163.204.244.150:9999
有效IP, %s HTTPS://112.85.170.172:9999
有效IP, %s HTTPS://112.85.131.34:9999
有效IP, %s HTTP://112.85.171.8:9999
有效IP, %s HTTPS://112.85.164.213:9999
有效IP, %s HTTPS://117.95.98.120:42704
有效IP, %s HTTP://115.225.49.188:8118
有效IP, %s HTTPS://119.162.37.165:8118
有效IP, %s HTTP://123.139.28.36:8118
有效IP, %s HTTPS://171.80.3.169:9999
有效IP, %s HTTPS://112.85.164.161:9999
有效IP, %s HTTPS://115.53.16.222:9999
有效IP, %s HTTPS://36.99.212.233:9999
有效IP, %s HTTPS://113.121.46.78:9999
有效IP, %s HTTPS://221.225.147.58:8118
有效IP, %s HTTP://222.186.45.145:57273
有效IP, %s HTTP://121.238.82.201:8118
有效IP, %s HTTP://114.221.20.254:8118
有效IP, %s HTTPS://58.250.23.210:1080
有效IP, %s HTTPS://180.160.139.246:8118
有效IP, %s HTTPS://112.80.41.86:8888
有效IP, %s HTTP://218.64.69.79:8080
有效IP, %s HTTPS://59.38.61.164:9797
有效IP, %s HTTPS://182.18.13.149:53281
有效IP, %s HTTP://113.251.221.143:8118
有效IP, %s HTTP://117.90.5.64:9000
有效IP, %s HTTPS://125.32.80.52:8080
有效IP, %s HTTPS://58.247.127.145:53281
有效IP, %s HTTPS://175.23.40.250:8080
有效IP, %s HTTP://211.162.70.229:3128
有效IP, %s HTTPS://182.149.157.168:8118
有效IP, %s HTTPS://218.22.7.62:53281
有效IP, %s HTTP://121.79.131.58:8080
有效IP, %s HTTP://14.115.106.178:808
有效IP, %s HTTPS://122.137.4.230:8118
有效IP, %s HTTPS://112.85.170.172:9999
有效IP, %s HTTPS://112.85.131.34:9999
有效IP, %s HTTPS://221.225.147.58:8118
有效IP, %s HTTPS://112.85.164.213:9999
有效IP, %s HTTPS://117.95.98.120:42704
有效IP, %s HTTPS://119.162.37.165:8118
有效IP, %s HTTPS://171.80.3.169:9999
有效IP, %s HTTPS://112.85.164.161:9999
有效IP, %s HTTPS://115.53.16.222:9999
有效IP, %s HTTPS://36.99.212.233:9999
有效IP, %s HTTPS://113.121.46.78:9999
有效IP, %s HTTPS://123.163.96.141:9999
有效IP, %s HTTPS://222.137.4.96:8118
有效IP, %s HTTPS://115.53.20.2:9999
有效IP, %s HTTPS://124.94.199.204:9999
有效IP, %s HTTPS://112.87.71.206:9999
有效IP, %s HTTPS://120.83.105.77:9999
有效IP, %s HTTPS://112.85.151.97:9999
有效IP, %s HTTPS://58.250.23.210:1080
有效IP, %s HTTP://120.83.98.192:9999
有效IP, %s HTTP://115.239.25.244:9999
有效IP, %s HTTP://120.83.111.221:9999
有效IP, %s HTTP://1.198.72.153:9999
有效IP, %s HTTP://180.126.169.6:8118
有效IP, %s HTTP://163.204.244.150:9999
有效IP, %s HTTP://112.85.171.8:9999
有效IP, %s HTTP://115.225.49.188:8118
有效IP, %s HTTP://222.186.45.145:57273
有效IP, %s HTTP://123.139.28.36:8118
有效IP, %s HTTP://112.85.169.44:9999
有效IP, %s HTTP://121.238.82.201:8118
有效IP, %s HTTP://1.198.72.48:9999
有效IP, %s HTTP://112.85.129.140:9999
有效IP, %s HTTP://171.37.157.61:8123
有效IP, %s HTTP://119.162.150.192:8118
有效IP, %s HTTP://114.221.20.254:8118
有效IP, %s HTTP://49.86.176.110:9999
有效IP, %s HTTP://114.230.69.201:9999
有效IP, %s HTTP://112.255.118.2:8118
有效代理ip
['HTTP://120.83.98.192:9999', 'HTTP://115.239.25.244:9999', 'HTTP://120.83.111.221:9999', 'HTTP://1.198.72.153:9999', 'HTTP://180.126.169.6:8118', 'HTTPS://122.137.4.230:8118', 'HTTP://163.204.244.150:9999', 'HTTPS://112.85.170.172:9999', 'HTTPS://112.85.131.34:9999', 'HTTP://112.85.171.8:9999', 'HTTPS://112.85.164.213:9999', 'HTTPS://117.95.98.120:42704', 'HTTP://115.225.49.188:8118', 'HTTPS://119.162.37.165:8118', 'HTTP://123.139.28.36:8118', 'HTTPS://171.80.3.169:9999', 'HTTPS://112.85.164.161:9999', 'HTTPS://115.53.16.222:9999', 'HTTPS://36.99.212.233:9999', 'HTTPS://113.121.46.78:9999', 'HTTPS://221.225.147.58:8118', 'HTTP://222.186.45.145:57273', 'HTTP://121.238.82.201:8118', 'HTTP://114.221.20.254:8118', 'HTTPS://58.250.23.210:1080', 'HTTPS://180.160.139.246:8118', 'HTTPS://112.80.41.86:8888', 'HTTP://218.64.69.79:8080', 'HTTPS://59.38.61.164:9797', 'HTTPS://182.18.13.149:53281', 'HTTP://113.251.221.143:8118', 'HTTP://117.90.5.64:9000', 'HTTPS://125.32.80.52:8080', 'HTTPS://58.247.127.145:53281', 'HTTPS://175.23.40.250:8080', 'HTTP://211.162.70.229:3128', 'HTTPS://182.149.157.168:8118', 'HTTPS://218.22.7.62:53281', 'HTTP://121.79.131.58:8080', 'HTTP://14.115.106.178:808', 'HTTPS://122.137.4.230:8118', 'HTTPS://112.85.170.172:9999', 'HTTPS://112.85.131.34:9999', 'HTTPS://221.225.147.58:8118', 'HTTPS://112.85.164.213:9999', 'HTTPS://117.95.98.120:42704', 'HTTPS://119.162.37.165:8118', 'HTTPS://171.80.3.169:9999', 'HTTPS://112.85.164.161:9999', 'HTTPS://115.53.16.222:9999', 'HTTPS://36.99.212.233:9999', 'HTTPS://113.121.46.78:9999', 'HTTPS://123.163.96.141:9999', 'HTTPS://222.137.4.96:8118', 'HTTPS://115.53.20.2:9999', 'HTTPS://124.94.199.204:9999', 'HTTPS://112.87.71.206:9999', 'HTTPS://120.83.105.77:9999', 'HTTPS://112.85.151.97:9999', 'HTTPS://58.250.23.210:1080', 'HTTP://120.83.98.192:9999', 'HTTP://115.239.25.244:9999', 'HTTP://120.83.111.221:9999', 'HTTP://1.198.72.153:9999', 'HTTP://180.126.169.6:8118', 'HTTP://163.204.244.150:9999', 'HTTP://112.85.171.8:9999', 'HTTP://115.225.49.188:8118', 'HTTP://222.186.45.145:57273', 'HTTP://123.139.28.36:8118', 'HTTP://112.85.169.44:9999', 'HTTP://121.238.82.201:8118', 'HTTP://1.198.72.48:9999', 'HTTP://112.85.129.140:9999', 'HTTP://171.37.157.61:8123', 'HTTP://119.162.150.192:8118', 'HTTP://114.221.20.254:8118', 'HTTP://49.86.176.110:9999', 'HTTP://114.230.69.201:9999', 'HTTP://112.255.118.2:8118']
参考资料：
- https://blog.csdn.net/weixin_43720396/article/details/88218204
- https://desmonday.github.io/2019/03/06/python%E7%88%AC%E8%99%AB%E5%AD%A6%E4%B9%A0-day6-IP%E4%BB%A3%E7%90%86/
PS: 若你觉得可以、还行、过得去、甚至不太差的话，可以“关注”一下，就此谢过!