python爬虫错误:http.client.HTTPException: got more than 100 headers的解决方法

python爬虫错误:http.client.HTTPException: got more than 100 headers的解决方法

Traceback (most recent call last):
  File "C:\Users\Jonariguez\AppData\Roaming\Python\Python36\site-packages\urllib3\connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "C:\Users\Jonariguez\AppData\Roaming\Python\Python36\site-packages\urllib3\connectionpool.py", line 387, in _make_request
    six.raise_from(e, None)
  File "", line 2, in raise_from
  File "C:\Users\Jonariguez\AppData\Roaming\Python\Python36\site-packages\urllib3\connectionpool.py", line 383, in _make_request
    httplib_response = conn.getresponse()
  File "D:\Software\Python3\lib\http\client.py", line 1331, in getresponse
    response.begin()
  File "D:\Software\Python3\lib\http\client.py", line 321, in begin
    self.headers = self.msg = parse_headers(self.fp)
  File "D:\Software\Python3\lib\http\client.py", line 211, in parse_headers
    raise HTTPException("got more than %d headers" % _MAXHEADERS)
http.client.HTTPException: got more than 100 headers

错误

返回的请求头大多，超过了自定义的最大请求头个数

解决方法：

修改最大请求头个数即可。

python3

import http.client

http.client._MAXHEADERS = 1000

python2

import httplib

httplib._MAXHEADERS = 1000