Python3环境安装scrapy库发生的问题解决方法

昨天安装scrapy库发生许多错误，历时一天终于解决，将解决过程记录下来，供大家参考。

我昨天首先在cmd中用pip install scrapy的方法安装scrapy库，结果出现了错误：

Exception:
Traceback (most recent call last):
  File "D:\ProgramData\Anaconda3\lib\site-packages\pip\compat\__init__.py", line 73, in console_to_str
    return s.decode(sys.__stdout__.encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbd in position 19: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\ProgramData\Anaconda3\lib\site-packages\pip\commands\install.py", line 335, in run
    wb.build(autobuilding=True)
  File "D:\ProgramData\Anaconda3\lib\site-packages\pip\wheel.py", line 749, in build
    self.requirement_set.prepare_files(self.finder)
  File "D:\ProgramData\Anaconda3\lib\site-packages\pip\req\req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "D:\ProgramData\Anaconda3\lib\site-packages\pip\req\req_set.py", line 634, in _prepare_file
    abstract_dist.prep_for_dist()
  File "D:\ProgramData\Anaconda3\lib\site-packages\pip\req\req_set.py", line 129, in prep_for_dist
    self.req_to_install.run_egg_info()
  File "D:\ProgramData\Anaconda3\lib\site-packages\pip\req\req_install.py", line 439, in run_egg_info
    command_desc='python setup.py egg_info')
  File "D:\ProgramData\Anaconda3\lib\site-packages\pip\utils\__init__.py", line 676, in call_subprocess
    line = console_to_str(proc.stdout.readline())
  File "D:\ProgramData\Anaconda3\lib\site-packages\pip\compat\__init__.py", line 75, in console_to_str
    return s.decode('utf_8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbd in position 19: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\ProgramData\Anaconda3\lib\site-packages\pip\basecommand.py", line 215, in main
    status = self.run(options, args)
  File "D:\ProgramData\Anaconda3\lib\site-packages\pip\commands\install.py", line 385, in run
    requirement_set.cleanup_files()
  File "D:\ProgramData\Anaconda3\lib\site-packages\pip\utils\build.py", line 38, in __exit__
    self.cleanup()
  File "D:\ProgramData\Anaconda3\lib\site-packages\pip\utils\build.py", line 42, in cleanup
    rmtree(self.name)
  File "D:\ProgramData\Anaconda3\lib\site-packages\pip\_vendor\retrying.py", line 49, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
  File "D:\ProgramData\Anaconda3\lib\site-packages\pip\_vendor\retrying.py", line 212, in call
    raise attempt.get()
  File "D:\ProgramData\Anaconda3\lib\site-packages\pip\_vendor\retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "D:\ProgramData\Anaconda3\lib\site-packages\six.py", line 693, in reraise
    raise value
  File "D:\ProgramData\Anaconda3\lib\site-packages\pip\_vendor\retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "D:\ProgramData\Anaconda3\lib\site-packages\pip\utils\__init__.py", line 102, in rmtree
    οnerrοr=rmtree_errorhandler)
  File "D:\ProgramData\Anaconda3\lib\shutil.py", line 494, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "D:\ProgramData\Anaconda3\lib\shutil.py", line 384, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "D:\ProgramData\Anaconda3\lib\shutil.py", line 389, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "D:\ProgramData\Anaconda3\lib\site-packages\pip\utils\__init__.py", line 114, in rmtree_errorhandler
    func(path)
PermissionError: [WinError 32] 另一个程序正在使用此文件，进程无法访问。: 'C:\\Users\\****\\AppData\\Local\\Temp\\pip-build-z2svhfkl\\Twisted\\MANIFEST.in'

重点是UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbd in position 19: invalid start byte.

翻译：unicode编码:“utf-8”编解码器不能解码字节0xbd在第19个位置:无效的开始字节

我用的是python编译平台是Anaconda，它的默认编码是‘utf-8’编码，而解析scrapy库需要的编码是‘gbk’。我需要更改我的默认编码。在网上查到sys库可以设置python2的默认编码，但是python3取消了设置默认编码的函数sys.setdefaultencoding()，无法修改默认编码。

后来，我有找到一种解决方案，就是将D:\ProgramData\Anaconda3\Lib\site-packages\pip\compat\__init__.py中所有'utf-8'改为'gbk'。如此便解决了编码的问题。

之后，又出现了错误：

Command "D:\ProgramData\Anaconda3\python.exe -u -c "import setuptools, tokenize;__file__='C:\\Users\\****\\AppData\\Local\\Temp\\pip-build-d8xmaagh\\Twisted\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record C:\Users\****\AppData\Local\Temp\pip-2lpyjh3t-record\install-record.txt --single-version-externally-managed --compile" failed with error code 1 in C:\Users\****\AppData\Local\Temp\pip-build-d8xmaagh\Twisted\

没看懂错误提示，但在网上发现安装scrapy需要许多支持库，Scrapy依赖的库比较多，至少需要依赖库有Twisted 14.0，lxml 3.4，pyOpenSSL 0.14。而在不同平台环境又各不相同，所以在安装之前最好确保把一些基本库安装好，尤其是Windows。

于是我又分别安装了以pip方法安装了lxml，pyOpenSSL，zope.interface，其中pywin32是我平台自带的库。在我安装Twisted时出现了与以上代码类似的错误提醒（安装srapy时的错误代码找不到了，与上述代码类似）。我尝试在网上下载了Twisted的最新版本，并安装，出现错误FileNotFoundError: [Errno 2] No such file or directory: 'C:\\WINDOWS\\system32\\Twisted-17.5.0-cp37-cp37m-win_amd64.whl'。后我将下载后的whl文件移动至system32文件夹下。安装，出现错误Twisted-17.9.0-cp37-cp37m-win_amd64.whl is not a supported wheel on this platform.因为下载的版本不对，我又下载了Twisted-17.9.0-cp36-cp36m-win_amd64.whl，这次总算安装成功了，之后便可成功安装scrapy库了。