安装pyspider
pip install pyspider
直接报错
Please specify --curl-dir=/path/to/built/libcurl
于是从PythonLibs官网 中获取依赖并自行下载到本地
下载与3.9对应的
执行安装
pip install D:\下载\ad3-2.2.1-cp39-cp39-win32.whl
安装成功之后 执行 pip install pyspider ,提示安装成功,运行 pyspider 报错。
Traceback (most recent call last):
File "D:\Python39\lib\runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "D:\Python39\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "D:\Python39\Scripts\pyspider.exe\__main__.py", line 4, in
File "D:\Python39\lib\site-packages\pyspider\run.py", line 231
async=True, get_object=False, no_input=False):
^
SyntaxError: invalid syntax
打开run.py 把async 替换成async1
继续执行pyspider ,还是报错
aceback (most recent call last):
File "D:\Python39\lib\threading.py", line 980, in _bootstrap_inner
self.run()
File "D:\Python39\lib\threading.py", line 917, in run
self._target(*self._args, **self._kwargs)
File "D:\Python39\lib\site-packages\click\core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "D:\Python39\lib\site-packages\click\decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "D:\Python39\lib\site-packages\pyspider\run.py", line 236, in fetcher
Fetcher = load_cls(None, None, fetcher_cls)
File "D:\Python39\lib\site-packages\pyspider\run.py", line 48, in load_cls
return utils.load_object(value)
File "D:\Python39\lib\site-packages\pyspider\libs\utils.py", line 369, in load_object
module = __import__(module_name, globals(), locals(), [object_name])
File "D:\Python39\lib\site-packages\pyspider\fetcher\__init__.py", line 1, in
from .tornado_fetcher import Fetcher
File "D:\Python39\lib\site-packages\pyspider\fetcher\tornado_fetcher.py", line 81
def __init__(self, inqueue, outqueue, poolsize=100, proxy=None, async=True):
打开 tornado_fetcher.py 把async 替换成async1 ,在执行pyspider,成功了,不再报错如下:
遇到如下错误:
File "\python39\lib\site-packages
\wsgidav\wsgidav_app.py", line 135, in __init__
_check_config(config)
File "python39\lib\site-packages
\wsgidav\wsgidav_app.py", line 119, in _check_config
raise ValueError("Invalid configuration:\n - " + "\n - ".join(errors))
ValueError: Invalid configuration:
- Deprecated option 'domaincontroller': use 'http_authenticator.domain_control
解决方案:
找到 文件 找到domaincontroller,这一行按照提示修改,但是依然报错,我就直接删除了,删除之后启动就报如下错误。
TypeError: Can't instantiate abstract class ScriptProvider with abstract methods get_resource_inst
我看到有的帖子说是wsgidav
版本太高,是的就是版本高了,高版本中的方法变了,于是乎复制 getResourceInst 改名为get_resource_inst 增加一下实现
于是重新执行 pyspider all 报错如下:
File "D:\Python39\lib\runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "D:\Python39\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "D:\Python39\Scripts\pyspider.exe\__main__.py", line 7, in
File "D:\Python39\lib\site-packages\pyspider\run.py", line 754, in main
cli()
File "D:\Python39\lib\site-packages\click\core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "D:\Python39\lib\site-packages\click\core.py", line 1055, in main
rv = self.invoke(ctx)
File "D:\Python39\lib\site-packages\click\core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "D:\Python39\lib\site-packages\click\core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "D:\Python39\lib\site-packages\click\core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "D:\Python39\lib\site-packages\click\decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "D:\Python39\lib\site-packages\pyspider\run.py", line 497, in all
ctx.invoke(webui, **webui_config)
File "D:\Python39\lib\site-packages\click\core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "D:\Python39\lib\site-packages\click\decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "D:\Python39\lib\site-packages\pyspider\run.py", line 384, in webui
app.run(host=host, port=port)
File "D:\Python39\lib\site-packages\pyspider\webui\app.py", line 64, in run
from werkzeug.dispatcher import DispatcherMiddleware
ModuleNotFoundError: No module named 'werkzeug.dispatcher'
依然是安装了新版本之后依赖包找不到了,于是找到 site-packages\werkzeug 安装目录发现werkzeug\middleware 下有个dispatcher.py 中有想要的类,于是就把webui 的引用包改了一下
if dav_app:
from werkzeug.middleware.dispatcher import DispatcherMiddleware
application = DispatcherMiddleware(application, {
'/dav': dav_app
})
container = tornado.wsgi.WSGIContainer(application)
于是乎就愉快的跑起来了。
经历两天奋战终于搞定了。
本次安装思路就是:安装新版本,遇到依赖报错就去找新版本的包中类库所在的路径,修改对应的依赖文件中的依赖。但是安装之后修改过的痕迹被覆盖的话就得重新搞了。这个安装方式适合有一定的开发经验的朋友。