安装和使用pyspider框架时遇到的问题

安装pyspider, 直接cmd中输入 pip install pyspider, 而后提示报错

 

 从网上找到解决方法,pycurl需要根据python版本采用wheel方法安装.

下载链接为https://www.lfd.uci.edu/~gohlke/pythonlibs/, 打开网页搜索pycurl,找到对应版本.

pip install wheel安装好直接, cmd切换到wheel文件目录, 输入 pip install 安装包名称.whl 即可

由于pycuel链接较慢,可用如下地址下载

https://pan.baidu.com/s/1jjf-CD6W0kJ8tDlj88C-QQ

-----------------------------------------------------------------------------

然后启动又遇到问题如图

 

 问题产生原因为python3.7中async被纳为关键词,需要将pyspider中的async替换掉, 建议改为async_,

分别在run.py、fetcher->tornado_fetcher.py、webui->app.py , ctrl+f查找async替换掉即可.

---------------------------------------------------------------------------

接下来启动,再次报错如下:

 

 百度得到解决方法如下:

产生原因为 WsgiDAV发布了版本 pre-release 3.x导致，需要降低版本,
将wsgidav替换为2.4.1

cmd输入如下命令
•# python -m pip install wsgidav==2.4.1 即可解决.

---------------------------------------------------------------

测试雪球网,

{
  "fetch": {},
  "process": {
    "callback": "index_page"
  },
  "project": "xueqiu",
  "schedule": {
    "age": 864000
  },
  "taskid": "6d1a833f721781353912c169bdaace49",
  "url": "https://xueqiu.com/"
}

提示报错

[E 191219 22:37:51 base_handler:203] HTTP 599: SSL certificate problem: unable to get local issuer certificate
    Traceback (most recent call last):
      File "d:\program files\python36\lib\site-packages\pyspider\libs\base_handler.py", line 196, in run_task
        result = self._run_task(task, response)
      File "d:\program files\python36\lib\site-packages\pyspider\libs\base_handler.py", line 175, in _run_task
        response.raise_for_status()
      File "d:\program files\python36\lib\site-packages\pyspider\libs\response.py", line 172, in raise_for_status
        six.reraise(Exception, Exception(self.error), Traceback.from_string(self.traceback).as_traceback())
      File "d:\program files\python36\lib\site-packages\six.py", line 695, in reraise
        raise value.with_traceback(tb)
      File "d:\program files\python36\lib\site-packages\pyspider\fetcher\tornado_fetcher.py", line 378, in http_fetch
        response = yield gen.maybe_future(self.http_client.fetch(request))
      File "d:\program files\python36\lib\site-packages\tornado\httpclient.py", line 102, in fetch
        self._async_client.fetch, request, **kwargs))
      File "d:\program files\python36\lib\site-packages\tornado\ioloop.py", line 458, in run_sync
        return future_cell[0].result()
      File "d:\program files\python36\lib\site-packages\tornado\concurrent.py", line 238, in result
        raise_exc_info(self._exc_info)
      File "", line 4, in raise_exc_info
    Exception: HTTP 599: SSL certificate problem: unable to get local issuer certificate

测试了百度到的各种解决办法,均无果.

去他喵的.

最后把
"url": "https://xueqiu.com/"
改成
"url": "http://xueqiu.com/"
测试通过,收工