weixin_33736832

【转】CentOS 6.5安装pyspider过程记录

原文地址：http://blog.sina.com.cn/s/blog_48c95a190102wczx.html

1、根据pyspider官方推荐的安装方法，使用pip命令直接安装pyspider

# sudo pip install pyspider

Installing collected packages: click, itsdangerous, Werkzeug, Flask, chardet, cssselect, lxml, pyquery, ordereddict, backports.ssl-match-hostname, singledispatch, certifi, backports-abc, tornado, Flask-Login, u-msgpack-python, wsgidav, pyspider

Running setup.py install for click

Running setup.py install for itsdangerous

Running setup.py install for chardet

Running setup.py install for lxml

Complete output from command /usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build-rpK3h8/lxml/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-mmscE4-record/install-record.txt --single-version-externally-managed --compile:

Building lxml version 3.6.0.

Building without Cython.

ERROR: /bin/sh: xslt-config: command not found

** make sure the development packages of libxml2 and libxslt are installed **

Using build configuration of libxslt

running install

running build

running build_py

creating build

creating build/lib.linux-x86_64-2.6

creating build/lib.linux-x86_64-2.6/lxml

copying src/lxml/_elementpath.py -> build/lib.linux-x86_64-2.6/lxml

copying src/lxml/doctestcompare.py -> build/lib.linux-x86_64-2.6/lxml

copying src/lxml/cssselect.py -> build/lib.linux-x86_64-2.6/lxml

copying src/lxml/ElementInclude.py -> build/lib.linux-x86_64-2.6/lxml

copying src/lxml/__init__.py -> build/lib.linux-x86_64-2.6/lxml

copying src/lxml/builder.py -> build/lib.linux-x86_64-2.6/lxml

copying src/lxml/pyclasslookup.py -> build/lib.linux-x86_64-2.6/lxml

copying src/lxml/usedoctest.py -> build/lib.linux-x86_64-2.6/lxml

copying src/lxml/sax.py -> build/lib.linux-x86_64-2.6/lxml

creating build/lib.linux-x86_64-2.6/lxml/includes

copying src/lxml/includes/__init__.py -> build/lib.linux-x86_64-2.6/lxml/includes

creating build/lib.linux-x86_64-2.6/lxml/html

copying src/lxml/html/defs.py -> build/lib.linux-x86_64-2.6/lxml/html

copying src/lxml/html/html5parser.py -> build/lib.linux-x86_64-2.6/lxml/html

copying src/lxml/html/_html5builder.py -> build/lib.linux-x86_64-2.6/lxml/html

copying src/lxml/html/_setmixin.py -> build/lib.linux-x86_64-2.6/lxml/html

copying src/lxml/html/_diffcommand.py -> build/lib.linux-x86_64-2.6/lxml/html

copying src/lxml/html/__init__.py -> build/lib.linux-x86_64-2.6/lxml/html

copying src/lxml/html/builder.py -> build/lib.linux-x86_64-2.6/lxml/html

copying src/lxml/html/clean.py -> build/lib.linux-x86_64-2.6/lxml/html

copying src/lxml/html/ElementSoup.py -> build/lib.linux-x86_64-2.6/lxml/html

copying src/lxml/html/usedoctest.py -> build/lib.linux-x86_64-2.6/lxml/html

copying src/lxml/html/soupparser.py -> build/lib.linux-x86_64-2.6/lxml/html

copying src/lxml/html/formfill.py -> build/lib.linux-x86_64-2.6/lxml/html

copying src/lxml/html/diff.py -> build/lib.linux-x86_64-2.6/lxml/html

creating build/lib.linux-x86_64-2.6/lxml/isoschematron

copying src/lxml/isoschematron/__init__.py -> build/lib.linux-x86_64-2.6/lxml/isoschematron

copying src/lxml/lxml.etree.h -> build/lib.linux-x86_64-2.6/lxml

copying src/lxml/lxml.etree_api.h -> build/lib.linux-x86_64-2.6/lxml

copying src/lxml/includes/config.pxd -> build/lib.linux-x86_64-2.6/lxml/includes

copying src/lxml/includes/schematron.pxd -> build/lib.linux-x86_64-2.6/lxml/includes

copying src/lxml/includes/xmlschema.pxd -> build/lib.linux-x86_64-2.6/lxml/includes

copying src/lxml/includes/relaxng.pxd -> build/lib.linux-x86_64-2.6/lxml/includes

copying src/lxml/includes/c14n.pxd -> build/lib.linux-x86_64-2.6/lxml/includes

copying src/lxml/includes/xslt.pxd -> build/lib.linux-x86_64-2.6/lxml/includes

copying src/lxml/includes/xpath.pxd -> build/lib.linux-x86_64-2.6/lxml/includes

copying src/lxml/includes/uri.pxd -> build/lib.linux-x86_64-2.6/lxml/includes

copying src/lxml/includes/etreepublic.pxd -> build/lib.linux-x86_64-2.6/lxml/includes

copying src/lxml/includes/tree.pxd -> build/lib.linux-x86_64-2.6/lxml/includes

copying src/lxml/includes/xmlerror.pxd -> build/lib.linux-x86_64-2.6/lxml/includes

copying src/lxml/includes/htmlparser.pxd -> build/lib.linux-x86_64-2.6/lxml/includes

copying src/lxml/includes/dtdvalid.pxd -> build/lib.linux-x86_64-2.6/lxml/includes

copying src/lxml/includes/xmlparser.pxd -> build/lib.linux-x86_64-2.6/lxml/includes

copying src/lxml/includes/xinclude.pxd -> build/lib.linux-x86_64-2.6/lxml/includes

copying src/lxml/includes/etree_defs.h -> build/lib.linux-x86_64-2.6/lxml/includes

copying src/lxml/includes/lxml-version.h -> build/lib.linux-x86_64-2.6/lxml/includes

creating build/lib.linux-x86_64-2.6/lxml/isoschematron/resources

creating build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/rng

copying src/lxml/isoschematron/resources/rng/iso-schematron.rng -> build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/rng

creating build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl

copying src/lxml/isoschematron/resources/xsl/RNG2Schtrn.xsl -> build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl

copying src/lxml/isoschematron/resources/xsl/XSD2Schtrn.xsl -> build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl

creating build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl/iso-schematron-xslt1

copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_message.xsl -> build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl/iso-schematron-xslt1

copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_abstract_expand.xsl -> build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl/iso-schematron-xslt1

copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_svrl_for_xslt1.xsl -> build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl/iso-schematron-xslt1

copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_dsdl_include.xsl -> build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl/iso-schematron-xslt1

copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_skeleton_for_xslt1.xsl -> build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl/iso-schematron-xslt1

copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/readme.txt -> build/lib.linux-x86_64-2.6/lxml/isoschematron/resources/xsl/iso-schematron-xslt1

running build_ext

building 'lxml.etree' extension

creating build/temp.linux-x86_64-2.6

creating build/temp.linux-x86_64-2.6/src

creating build/temp.linux-x86_64-2.6/src/lxml

gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -Isrc/lxml/includes -I/usr/include/python2.6 -c src/lxml/lxml.etree.c -o build/temp.linux-x86_64-2.6/src/lxml/lxml.etree.o -w

src/lxml/lxml.etree.c:82:20: error: Python.h: No such file or directory

src/lxml/lxml.etree.c:84:6: error: #error Python headers needed to compile C extensions, please install development version of Python.

Compile failed: command 'gcc' failed with exit status 1

creating tmp

cc -I/usr/include/libxml2 -c /tmp/xmlXPathInitlSjq1o.c -o tmp/xmlXPathInitlSjq1o.o

/tmp/xmlXPathInitlSjq1o.c:1:26: error: libxml/xpath.h: No such file or directory

*********************************************************************************

Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?

*********************************************************************************

error: command 'gcc' failed with exit status 1

根据错误提示可知，导致安装出错的原因是libxml2和libxslt这两个库没有安装，而且找不到xslt-config这个可执行程序，解决方法如下：

# sudo yum install libxml2

# sudo yum install libxslt

# sudo yum install libxslt-devel

2、继续使用pip命令安装pyspider

# sudo pip install pyspider

...

Running setup.py install for lxml

Complete output from command /usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build-EQanqI/lxml/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-WzvLSh-record/install-record.txt --single-version-externally-managed --compile:

Building lxml version 3.6.0.

Building without Cython.

Using build configuration of libxslt 1.1.26

Building against libxml2/libxslt in the following directory: /usr/lib64