scrapy 安装出错 centos6 requirement Twisted>=13.1.0

Wednesday night： redhat/centos6.7 系统 安装scrapy时，提示缺少 twisted。然后用pip install twisted也提示找不到合适版本。

最终下载twisted源码安装，再pip install scrapy。问题就解决了。

$ vim make_scrapy.sh
$ cat make_scrapy.sh

# virtualenv --python=python3.6 ENV
# source ENV/bin/activate
wget https://twistedmatrix.com/Releases/Twisted/17.1/Twisted-17.1.0.tar.bz2
tar -jxvf Twisted-17.1.0.tar.bz2
cd Twisted-17.1.0
python setup.py install
cd ..
pip install scrapy

$ chmod +x make_scrapy.sh
$ ./make_scrapy.sh

详情如下：

[willie@bogon spiders]$ django-admin startproject beautygirls

[willie@bogon spiders]$ cd beautygirls

[willie@bogon beautygirls]$ vvv   #在~/.bashrc文件中 alias vvv=‘virtualenv --python=python3 venv'
Running virtualenv with interpreter /usr/local/bin/python3
Using base prefix '/opt/Python352'
New python executable in /home/willie/projects/spiders/beautygirls/venv/bin/python3
Also creating executable in /home/willie/projects/spiders/beautygirls/venv/bin/python
Installing setuptools, pip, wheel...done.
[willie@bogon beautygirls]$ ve   #在~/.bashrc文件中 alias ve="source venv/bin/activate"
(venv) [willie@bogon beautygirls]$ pip install scrapy
Collecting scrapy
  Using cached Scrapy-1.3.3-py2.py3-none-any.whl
Collecting PyDispatcher>=2.0.5 (from scrapy)
Collecting cssselect>=0.9 (from scrapy)
  Using cached cssselect-1.0.1-py2.py3-none-any.whl
Collecting parsel>=1.1 (from scrapy)
  Using cached parsel-1.1.0-py2.py3-none-any.whl
Collecting pyOpenSSL (from scrapy)
  Using cached pyOpenSSL-16.2.0-py2.py3-none-any.whl
Collecting queuelib (from scrapy)
  Using cached queuelib-1.4.2-py2.py3-none-any.whl
Collecting w3lib>=1.15.0 (from scrapy)
  Using cached w3lib-1.17.0-py2.py3-none-any.whl
Collecting lxml (from scrapy)
  Using cached lxml-3.7.3-cp35-cp35m-manylinux1_x86_64.whl
Collecting service-identity (from scrapy)
  Using cached service_identity-16.0.0-py2.py3-none-any.whl
Collecting Twisted>=13.1.0 (from scrapy)
  Could not find a version that satisfies the requirement Twisted>=13.1.0 (from scrapy) (from versions: )

No matching distribution found for Twisted>=13.1.0 (from scrapy)

-------------------------------------------------------------------------------------------------

(venv) [willie@bogon beautygirls]$ pip install Twisted
Collecting Twisted
  Could not find a version that satisfies the requirement Twisted (from versions: )
No matching distribution found for Twisted
(venv) [willie@bogon beautygirls]$ pip install twisted
Collecting twisted
  Could not find a version that satisfies the requirement twisted (from versions: )
No matching distribution found for twisted

--------------------------------------------------------------------------------------------------

(venv) [willie@bogon beautygirls]$ pip install --verbose twisted

...

Traceback (most recent call last):
  File "/home/willie/projects/spiders/beautygirls/venv/lib/python3.5/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/home/willie/projects/spiders/beautygirls/venv/lib/python3.5/site-packages/pip/commands/install.py", line 335, in run
    wb.build(autobuilding=True)
  File "/home/willie/projects/spiders/beautygirls/venv/lib/python3.5/site-packages/pip/wheel.py", line 749, in build
    self.requirement_set.prepare_files(self.finder)
  File "/home/willie/projects/spiders/beautygirls/venv/lib/python3.5/site-packages/pip/req/req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/home/willie/projects/spiders/beautygirls/venv/lib/python3.5/site-packages/pip/req/req_set.py", line 554, in _prepare_file
    require_hashes
  File "/home/willie/projects/spiders/beautygirls/venv/lib/python3.5/site-packages/pip/req/req_install.py", line 278, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/home/willie/projects/spiders/beautygirls/venv/lib/python3.5/site-packages/pip/index.py", line 514, in find_requirement
    'No matching distribution found for %s' % req
pip.exceptions.DistributionNotFound: No matching distribution found for twisted

------------------------------------------------------------------------------------------------------

the way I fix it（写了个小脚本自动下载和安装twisted及scrapy）:

(venv) [willie@bogon beautygirls]$ vim make_scrapy.sh
(venv) [willie@bogon beautygirls]$ cat make_scrapy.sh 
#virtualenv --python=python3.5 venv
#source venv/bin/activate
wget https://twistedmatrix.com/Releases/Twisted/17.1/Twisted-17.1.0.tar.bz2
tar -jxvf Twisted-17.1.0.tar.bz2
cd Twisted-17.1.0
python setup.py install
cd ..
pip install scrapy

(venv) [willie@bogon beautygirls]$ source make_scrapy.sh

.....wait for a few minutes 几分钟过后就自动安装完成了

。。。

  Failed cleaning build dir for cryptography
Failed to build cryptography
Installing collected packages: cssselect, PyDispatcher, w3lib, idna, asn1crypto, pycparser, cffi, cryptography, pyOpenSSL, pyasn1, pyasn1-modules, service-identity, lxml, queuelib, parsel, scrapy
  Running setup.py install for cryptography ... done
Successfully installed PyDispatcher-2.0.5 asn1crypto-0.22.0 cffi-1.10.0 cryptography-1.8.1 cssselect-1.0.1 idna-2.5 lxml-3.7.3 parsel-1.1.0 pyOpenSSL-16.2.0 pyasn1-0.2.3 pyasn1-modules-0.0.8 pycparser-2.17 queuelib-1.4.2 scrapy-1.3.3 service-identity-16.0.0 w3lib-1.17.0
(venv) [willie@bogon beautygirls]$ scrapy version
Scrapy 1.3.3

That's all,well done!

安装完成，爬个美女图片庆祝一下吧。