fake_useragent 实例化对象时报错

import fake_useragent

# 实例化对象时报错
ua = fake_useragent.UserAgent()
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "d:\programdata\anaconda3\lib\site-packages\fake_useragent\utils.py", lin
e 166, in load
    verify_ssl=verify_ssl,
  File "d:\programdata\anaconda3\lib\site-packages\fake_useragent\utils.py", lin
e 122, in get_browser_versions
    verify_ssl=verify_ssl,
  File "d:\programdata\anaconda3\lib\site-packages\fake_useragent\utils.py", lin
e 84, in get
    raise FakeUserAgentError('Maximum amount of retries reached')
fake_useragent.errors.FakeUserAgentError: Maximum amount of retries reached

依据报错,初步推断是网络超时,从网查阅资料得知,这个库会引用在线资源,经排查,相关配置如下

fake_useragent\settings.py

# -*- coding: utf-8 -*-
from __future__ import absolute_import, unicode_literals

import os
import tempfile

__version__ = '0.1.11'

DB = os.path.join(
    tempfile.gettempdir(),
    'fake_useragent_{version}.json'.format(
        version=__version__,
    ),
)

CACHE_SERVER = 'https://fake-useragent.herokuapp.com/browsers/{version}'.format(
    version=__version__,
)

BROWSERS_STATS_PAGE = 'https://www.w3schools.com/browsers/default.asp'

BROWSER_BASE_PAGE = 'http://useragentstring.com/pages/useragentstring.php?name={browser}'  # noqa

其中,这个网站https://www.w3schools.com/browsers/default.asp打不开,导致超时报错

依次查看其它url,发现都可以正常打开,其中CACHE_SERVER是存储了所有UserAgent的json数据,再次观察其中DB这个变量,结合fake_useragent\fake.py中的逻辑,判断这个变量应该是存储json数据的,所以大体逻辑应该是,首次初始化时,会自动爬取CACHE_SERVER中的json数据,然后将其存储到本地,所以我们直接将json存到指定路径下,再次初始化时,应该就不会报错

In [11]: import tempfile

In [13]: print(tempfile.gettempdir())
C:\Users\ADMINI~1.-20\AppData\Local\Temp\1

CACHE_SERVER的json数据保存为fake_useragent_0.1.11.json,并放到目录C:\Users\ADMINI~1.-20\AppData\Local\Temp\1

再次调用,可以正常调用

In [24]: import fake_useragent

In [25]: ua = fake_useragent.UserAgent()	# 这个过程没有再次去获取在线数据

In [26]: ua.data_browsers['chrome'][0]
Out[26]: 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Ch
rome/41.0.2228.0 Safari/537.36'

注:如果CACHE_SERVER不是https://fake-useragent.herokuapp.com/browsers/0.1.11,请更新一下库
pip install --upgrade fake_useragent

你可能感兴趣的:(python库,爬虫)