一直想爬虫，前一段时间终于吧python大概的学了一下，想着终于可以上手爬虫了，不成想一路坎坷，各种坑，先大概的总结下目前遇到的问题以及处理方案（也有没找到解决方案的）。
-
ImportError: pycurl: libcurl link-time ssl backend (openssl)- 找到的解决方案1：说来奇怪这个方法至今不能解决我的问题 ，我的电脑是Mac 10.13.1的系统
-
pip uninstall pycurl#卸载掉之前安装的但是,有的需要在这个命令前加上sudo -H用来增加权限 - 继续执行
export PYCURL_SSL_LIBRARY=openssl - 再次安装
pip install pycurl
- 方案二：貌似对我有用实际没有
就是在方案一中的第二步后边改为下边三步
export LDFLAGS=-L/usr/local/opt/openssl/lib
export CPPFLAGS=-I/usr/local/opt/openssl/include
pip install pycurl --compile --no-cache-dir - 方案三：
pip3 uninstall pycurl
pip3 install --compile --install-option="--with-openssl" pycurl
有的人说以上操作完成后重启电脑就会好，我重启电脑，再次
pyspider，依旧有问题
~ ᐅ pyspider
[I 171228 12:59:16 result_worker:49] result_worker starting...
Process Process-4:
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "/Users/belief/Library/Python/2.7/lib/python/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/Users/belief/Library/Python/2.7/lib/python/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/Library/Python/2.7/site-packages/pyspider/run.py", line 236, in fetcher
Fetcher = load_cls(None, None, fetcher_cls)
File "/Library/Python/2.7/site-packages/pyspider/run.py", line 48, in load_cls
return utils.load_object(value)
File "/Library/Python/2.7/site-packages/pyspider/libs/utils.py", line 357, in load_object
module = __import__(module_name, globals(), locals(), [utf8(object_name)], -1)
File "/Library/Python/2.7/site-packages/pyspider/fetcher/__init__.py", line 1, in
from .tornado_fetcher import Fetcher
File "/Library/Python/2.7/site-packages/pyspider/fetcher/tornado_fetcher.py", line 30, in
from tornado.curl_httpclient import CurlAsyncHTTPClient
File "/Users/belief/Library/Python/2.7/lib/python/site-packages/tornado/curl_httpclient.py", line 24, in
import pycurl # type: ignore
ImportError: pycurl: libcurl link-time ssl backend (openssl) is different from compile-time ssl backend (none/other)
[I 171228 12:59:16 processor:211] processor starting...
[I 171228 12:59:17 scheduler:647] scheduler starting...
[I 171228 12:59:17 scheduler:782] scheduler.xmlrpc listening on 127.0.0.1:23333
Traceback (most recent call last):
File "/usr/local/bin/pyspider", line 9, in
load_entry_point('pyspider==0.3.9', 'console_scripts', 'pyspider')()
File "/Library/Python/2.7/site-packages/pyspider/run.py", line 754, in main
cli()
File "/Users/belief/Library/Python/2.7/lib/python/site-packages/click/core.py", line 722, in __call__
return self.main(*args, **kwargs)
File "/Users/belief/Library/Python/2.7/lib/python/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/Users/belief/Library/Python/2.7/lib/python/site-packages/click/core.py", line 1043, in invoke
return Command.invoke(self, ctx)
File "/Users/belief/Library/Python/2.7/lib/python/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/belief/Library/Python/2.7/lib/python/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/Users/belief/Library/Python/2.7/lib/python/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/Library/Python/2.7/site-packages/pyspider/run.py", line 165, in cli
ctx.invoke(all)
File "/Users/belief/Library/Python/2.7/lib/python/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/Users/belief/Library/Python/2.7/lib/python/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/Library/Python/2.7/site-packages/pyspider/run.py", line 497, in all
ctx.invoke(webui, **webui_config)
File "/Users/belief/Library/Python/2.7/lib/python/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/Users/belief/Library/Python/2.7/lib/python/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/Library/Python/2.7/site-packages/pyspider/run.py", line 333, in webui
app = load_cls(None, None, webui_instance)
File "/Library/Python/2.7/site-packages/pyspider/run.py", line 48, in load_cls
return utils.load_object(value)
File "/Library/Python/2.7/site-packages/pyspider/libs/utils.py", line 357, in load_object
module = __import__(module_name, globals(), locals(), [utf8(object_name)], -1)
File "/Library/Python/2.7/site-packages/pyspider/webui/__init__.py", line 8, in
from . import app, index, debug, task, result, login
File "/Library/Python/2.7/site-packages/pyspider/webui/app.py", line 17, in
from pyspider.fetcher import tornado_fetcher
File "/Library/Python/2.7/site-packages/pyspider/fetcher/__init__.py", line 1, in
from .tornado_fetcher import Fetcher
File "/Library/Python/2.7/site-packages/pyspider/fetcher/tornado_fetcher.py", line 30, in
from tornado.curl_httpclient import CurlAsyncHTTPClient
File "/Users/belief/Library/Python/2.7/lib/python/site-packages/tornado/curl_httpclient.py", line 24, in
import pycurl # type: ignore
ImportError: pycurl: libcurl link-time ssl backend (openssl) is different from compile-time ssl backend (none/other)
~ ᐅ phantomjs fetcher running on port 25555
有点要崩溃，为啥各种问题，pyspider是没有办法绕开的，谁有解决方法和我说一下
这是回家在家里电脑再一次安装出现的问题
In file included from src/docstrings.c:4:
src/pycurl.h:185:5: warning: "libcurl was compiled with SSL support, but configure could not determine which " "library was used; thus no SSL crypto locking callbacks will be set, which may " "cause random crashes on SSL requests" [-W#warnings]
# warning \
^
1 warning generated.
In file included from src/docstrings.c:4:
src/pycurl.h:185:5: warning: "libcurl was compiled with SSL support, but configure could not determine which " "library was used; thus no SSL crypto locking callbacks will be set, which may " "cause random crashes on SSL requests" [-W#warnings]
# warning \
^
1 warning generated.
cc -fno-strict-aliasing -fno-common -dynamic -arch x86_64 -arch i386 -g -Os -pipe -fno-common -fno-strict-aliasing -fwrapv -DENABLE_DTRACE -DMACOSX -DNDEBUG -Wall -Wstrict-prototypes -Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch i386 -arch x86_64 -pipe -DPYCURL_VERSION="7.43.0.1" -DHAVE_CURL_SSL=1 -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c src/easy.c -o build/temp.macosx-10.13-intel-2.7/src/easy.o
In file included from src/easy.c:1:
src/pycurl.h:185:5: warning: "libcurl was compiled with SSL support, but configure could not determine which " "library was used; thus no SSL crypto locking callbacks will be set, which may " "cause random crashes on SSL requests" [-W#warnings]
# warning \
^
src/easy.c:813:26: warning: implicit conversion loses integer precision: 'long' to 'int' [-Wshorten-64-to-32]
int sockfd = PyInt_AsLong(fileno_result);
~~~~~~ ^~~~~~~~~~~~~~~~~~~~~~~~~~~
src/easy.c:1097:24: warning: implicit conversion loses integer precision: 'long' to 'int' [-Wshorten-64-to-32]
int ret_code = PyInt_AsLong(result);
~~~~~~~~ ^~~~~~~~~~~~~~~~~~~~
src/easy.c:2062:23: warning: implicit conversion loses integer precision: 'long' to 'int' [-Wshorten-64-to-32]
val = PyLong_AsLong(PyListOrTuple_GetItem(httppost_option, j, which_httppost_option));
~ ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4 warnings generated.
In file included from src/easy.c:1:
src/pycurl.h:185:5: warning: "libcurl was compiled with SSL support, but configure could not determine which " "library was used; thus no SSL crypto locking callbacks will be set, which may " "cause random crashes on SSL requests" [-W#warnings]
# warning \
^
1 warning generated.
cc -fno-strict-aliasing -fno-common -dynamic -arch x86_64 -arch i386 -g -Os -pipe -fno-common -fno-strict-aliasing -fwrapv -DENABLE_DTRACE -DMACOSX -DNDEBUG -Wall -Wstrict-prototypes -Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch i386 -arch x86_64 -pipe -DPYCURL_VERSION="7.43.0.1" -DHAVE_CURL_SSL=1 -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c src/module.c -o build/temp.macosx-10.13-intel-2.7/src/module.o
In file included from src/module.c:1:
src/pycurl.h:185:5: warning: "libcurl was compiled with SSL support, but configure could not determine which " "library was used; thus no SSL crypto locking callbacks will be set, which may " "cause random crashes on SSL requests" [-W#warnings]
# warning \
^
1 warning generated.
In file included from src/module.c:1:
src/pycurl.h:185:5: warning: "libcurl was compiled with SSL support, but configure could not determine which " "library was used; thus no SSL crypto locking callbacks will be set, which may " "cause random crashes on SSL requests" [-W#warnings]
# warning \
^
1 warning generated.
cc -fno-strict-aliasing -fno-common -dynamic -arch x86_64 -arch i386 -g -Os -pipe -fno-common -fno-strict-aliasing -fwrapv -DENABLE_DTRACE -DMACOSX -DNDEBUG -Wall -Wstrict-prototypes -Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch i386 -arch x86_64 -pipe -DPYCURL_VERSION="7.43.0.1" -DHAVE_CURL_SSL=1 -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c src/multi.c -o build/temp.macosx-10.13-intel-2.7/src/multi.o
In file included from src/multi.c:1:
src/pycurl.h:185:5: warning: "libcurl was compiled with SSL support, but configure could not determine which " "library was used; thus no SSL crypto locking callbacks will be set, which may " "cause random crashes on SSL requests" [-W#warnings]
# warning \
^
src/multi.c:815:22: warning: implicit conversion loses integer precision: 'long' to '__darwin_suseconds_t' (aka 'int') [-Wshorten-64-to-32]
tv.tv_usec = (long)(timeout*1000000.0);
~ ^~~~~~~~~~~~~~~~~~~~~~~~~
2 warnings generated.
In file included from src/multi.c:1:
src/pycurl.h:185:5: warning: "libcurl was compiled with SSL support, but configure could not determine which " "library was used; thus no SSL crypto locking callbacks will be set, which may " "cause random crashes on SSL requests" [-W#warnings]
# warning \
^
1 warning generated.
cc -fno-strict-aliasing -fno-common -dynamic -arch x86_64 -arch i386 -g -Os -pipe -fno-common -fno-strict-aliasing -fwrapv -DENABLE_DTRACE -DMACOSX -DNDEBUG -Wall -Wstrict-prototypes -Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch i386 -arch x86_64 -pipe -DPYCURL_VERSION="7.43.0.1" -DHAVE_CURL_SSL=1 -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c src/oscompat.c -o build/temp.macosx-10.13-intel-2.7/src/oscompat.o
In file included from src/oscompat.c:1:
src/pycurl.h:185:5: warning: "libcurl was compiled with SSL support, but configure could not determine which " "library was used; thus no SSL crypto locking callbacks will be set, which may " "cause random crashes on SSL requests" [-W#warnings]
# warning \
^
1 warning generated.
In file included from src/oscompat.c:1:
src/pycurl.h:185:5: warning: "libcurl was compiled with SSL support, but configure could not determine which " "library was used; thus no SSL crypto locking callbacks will be set, which may " "cause random crashes on SSL requests" [-W#warnings]
# warning \
^
1 warning generated.
cc -fno-strict-aliasing -fno-common -dynamic -arch x86_64 -arch i386 -g -Os -pipe -fno-common -fno-strict-aliasing -fwrapv -DENABLE_DTRACE -DMACOSX -DNDEBUG -Wall -Wstrict-prototypes -Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch i386 -arch x86_64 -pipe -DPYCURL_VERSION="7.43.0.1" -DHAVE_CURL_SSL=1 -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c src/pythoncompat.c -o build/temp.macosx-10.13-intel-2.7/src/pythoncompat.o
In file included from src/pythoncompat.c:1:
src/pycurl.h:185:5: warning: "libcurl was compiled with SSL support, but configure could not determine which " "library was used; thus no SSL crypto locking callbacks will be set, which may " "cause random crashes on SSL requests" [-W#warnings]
# warning \
^
1 warning generated.
In file included from src/pythoncompat.c:1:
src/pycurl.h:185:5: warning: "libcurl was compiled with SSL support, but configure could not determine which " "library was used; thus no SSL crypto locking callbacks will be set, which may " "cause random crashes on SSL requests" [-W#warnings]
# warning \
^
1 warning generated.
cc -fno-strict-aliasing -fno-common -dynamic -arch x86_64 -arch i386 -g -Os -pipe -fno-common -fno-strict-aliasing -fwrapv -DENABLE_DTRACE -DMACOSX -DNDEBUG -Wall -Wstrict-prototypes -Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch i386 -arch x86_64 -pipe -DPYCURL_VERSION="7.43.0.1" -DHAVE_CURL_SSL=1 -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c src/share.c -o build/temp.macosx-10.13-intel-2.7/src/share.o
In file included from src/share.c:1:
src/pycurl.h:185:5: warning: "libcurl was compiled with SSL support, but configure could not determine which " "library was used; thus no SSL crypto locking callbacks will be set, which may " "cause random crashes on SSL requests" [-W#warnings]
# warning \
^
1 warning generated.
In file included from src/share.c:1:
src/pycurl.h:185:5: warning: "libcurl was compiled with SSL support, but configure could not determine which " "library was used; thus no SSL crypto locking callbacks will be set, which may " "cause random crashes on SSL requests" [-W#warnings]
# warning \
^
1 warning generated.
cc -fno-strict-aliasing -fno-common -dynamic -arch x86_64 -arch i386 -g -Os -pipe -fno-common -fno-strict-aliasing -fwrapv -DENABLE_DTRACE -DMACOSX -DNDEBUG -Wall -Wstrict-prototypes -Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch i386 -arch x86_64 -pipe -DPYCURL_VERSION="7.43.0.1" -DHAVE_CURL_SSL=1 -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c src/stringcompat.c -o build/temp.macosx-10.13-intel-2.7/src/stringcompat.o
In file included from src/stringcompat.c:1:
src/pycurl.h:185:5: warning: "libcurl was compiled with SSL support, but configure could not determine which " "library was used; thus no SSL crypto locking callbacks will be set, which may " "cause random crashes on SSL requests" [-W#warnings]
# warning \
^
1 warning generated.
In file included from src/stringcompat.c:1:
src/pycurl.h:185:5: warning: "libcurl was compiled with SSL support, but configure could not determine which " "library was used; thus no SSL crypto locking callbacks will be set, which may " "cause random crashes on SSL requests" [-W#warnings]
# warning \
^
1 warning generated.
cc -fno-strict-aliasing -fno-common -dynamic -arch x86_64 -arch i386 -g -Os -pipe -fno-common -fno-strict-aliasing -fwrapv -DENABLE_DTRACE -DMACOSX -DNDEBUG -Wall -Wstrict-prototypes -Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch i386 -arch x86_64 -pipe -DPYCURL_VERSION="7.43.0.1" -DHAVE_CURL_SSL=1 -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c src/threadsupport.c -o build/temp.macosx-10.13-intel-2.7/src/threadsupport.o
In file included from src/threadsupport.c:1:
src/pycurl.h:185:5: warning: "libcurl was compiled with SSL support, but configure could not determine which " "library was used; thus no SSL crypto locking callbacks will be set, which may " "cause random crashes on SSL requests" [-W#warnings]
# warning \
^
1 warning generated.
In file included from src/threadsupport.c:1:
src/pycurl.h:185:5: warning: "libcurl was compiled with SSL support, but configure could not determine which " "library was used; thus no SSL crypto locking callbacks will be set, which may " "cause random crashes on SSL requests" [-W#warnings]
# warning \
^
1 warning generated.
cc -bundle -undefined dynamic_lookup -arch x86_64 -arch i386 -Wl,-F. build/temp.macosx-10.13-intel-2.7/src/docstrings.o build/temp.macosx-10.13-intel-2.7/src/easy.o build/temp.macosx-10.13-intel-2.7/src/module.o build/temp.macosx-10.13-intel-2.7/src/multi.o build/temp.macosx-10.13-intel-2.7/src/oscompat.o build/temp.macosx-10.13-intel-2.7/src/pythoncompat.o build/temp.macosx-10.13-intel-2.7/src/share.o build/temp.macosx-10.13-intel-2.7/src/stringcompat.o build/temp.macosx-10.13-intel-2.7/src/threadsupport.o -lcurl -o build/lib.macosx-10.13-intel-2.7/pycurl.so
running install_lib
copying build/lib.macosx-10.13-intel-2.7/pycurl.so -> /Library/Python/2.7/site-packages
creating /Library/Python/2.7/site-packages/curl
copying build/lib.macosx-10.13-intel-2.7/curl/__init__.py -> /Library/Python/2.7/site-packages/curl
byte-compiling /Library/Python/2.7/site-packages/curl/__init__.py to __init__.pyc
running install_data
creating /System/Library/Frameworks/Python.framework/Versions/2.7/share
error: could not create '/System/Library/Frameworks/Python.framework/Versions/2.7/share': Operation not permitted
----------------------------------------
Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/private/tmp/pip-build-FOzCHC/pycurl/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-arRBfJ-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/tmp/pip-build-FOzCHC/pycurl/