网络爬虫-pyspider环境搭建（standalone）

1、安装依赖：apt-get install python python-dev python-distribute python-pip libcurl4-openssl-dev libxml2-dev libxslt1-dev python-lxml （注意apt-get的升级，apt-get update）

2、安装pip：

#wget "https://pypi.python.org/packages/source/p/pip/pip-1.5.4.tar.gz#md5=834b2904f92d46aaa333267fb1c922bb" --no-check-certificate
#tar -xzvf pip
#cd pip
#python setup.py install

此时可能会报错，需要安装一个setuptools，方法

# wget "https://bitbucket.org/pypa/setuptools/get/default.tar.gz#egg=setuptools-dev" --no-check-certificate
# tar -xzvf default.tar.gz
# cd pypa-setuptools-eb92fc5071bf 
# python setup.py install

3、使用pip安装pyspider

#pip install pyspider

4、执行pyspider

#pyspider -c config.json

如果没有配置文件，直接：

#pyspider

此处可能会报错：

Traceback (most recent call last):
File “/usr/local/bin/pyspider”, line 5, in
from pkg_resources import load_entry_point
File “build/bdist.linux-x86_64/egg/pkg_resources.py”, line 2603, in
)
File “build/bdist.linux-x86_64/egg/pkg_resources.py”, line 666, in require
to resolve their dependencies. error_info is a dictionary mapping
File “build/bdist.linux-x86_64/egg/pkg_resources.py”, line 565, in resolve
keys2 = self.entry_keys.setdefault(dist.location,[])
pkg_resources.DistributionNotFound: click>=3.3
需要升级下setuptools，从经验看来，很多安装问题都和升级有关系

#pip install -U setuptools

下面来介绍通过pyspider源码来运行：
安装依赖，无需安装pip

#python setup.py install
#./run.py