redhat系统安装scrapy爬虫框架步骤Python版

1、配置本地yum
把Redhat6.5系统ISO镜像文件复制到/root
mkdir /yum


cd /etc/yum.repos.d
cp rhel-source.repo iso.repo
mv rhel-source.repo rhel-source.repo.bak
vim  /etc/yum.repos.d/iso.repo
更新yum原地址:
点i,进入vim编辑模式,
[rhel-iso]
name=Red Hat Enterprise Linux $releasever - $basearch - Source
baseurl=file:///yum
enabled=1
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release


点esc,输入:号,wq退出vim


然后执行:
[root@localhost yum.repos.d]# yum clean all
然后检查:
[root@localhost yum.repos.d]# yum list


vim /etc/rc.local
点i,进入vim编辑模式,添加一行语句
mount -o loop /root/RHEL_6.5_x86_64.iso /yum
(把你的ISO文件,挂载到/yum下)
点esc,输入:号,wq退出vim
reboot




vim /etc/profile
http_proxy=10.0.251.94:8080
https_proxy=10.0.251.94:8080
ftp_proxy=10.0.251.94:8080
no_proxy=10.0.*
export http_proxy https_proxy ftp_proxy no_proxy
source /etc/profile


2、添加依赖
yum install -y python-devel libxml2 zlib-devel bzip2-devel xz-libs xz wget git tar gcc gcc-c++ openssl openssl-devel pcre-devel python-devel libevent automake autoconf libtool make git sqlite-devel


3、升级python
#wget https://www.python.org/ftp/python/2.7.12/Python-2.7.12.tgz
#xz -d Python-2.7.12.tar.xz
#tar -xvf Python-2.7.12.tar
cd Python-2.7.12
./configure --prefix=/usr/local/python2.7.12
make && make install
mv /usr/bin/python /usr/bin/python2.6.6
ln -s /usr/local/python2.7.12/bin/python2.7 /usr/bin/python


vim /usr/bin/yum  
修改为
#!/usr/bin/python2.6.6 


4、安装setuptools
#wget --no-check-certificate https://pypi.python.org/packages/source/s/setuptools/setuptools-1.4.2.tar.gz
#tar -xvf setuptools-1.4.2.tar.gz
cd setuptools-26.0.0
sudo python setup.py install
 
 #ln -s /usr/python/Python-2.7.12/bin/easy_install /usr/bin/easy_install
使用这个:  ln -s /usr/local/python2.7.12/bin/easy_install /usr/bin/easy_install






5、安装pip
#wget --no-check-certificate https://pypi.python.org/packages/source/p/pip/pip-6.0.7.tar.gz
#tar zxvf pip-6.0.7.tar.gz
cd pip-8.1.2
python setup.py install 


#ln -s /usr/python/Python-2.7.12/bin/pip /usr/bin/pip
使用这个:       /usr/local/python2.7.12/bin/python 


vim /etc/profile


export PATH=$HOME/bin:/usr/local/python2.7.12/bin:/usr/local/bin:$PATH


source /etc/profile


6、安装Scrapy
pip install Scrapy  -d /home/source
pip install scrapy-redis  -d /home/source
此处需要更新openssl


7、创建scrapy工程
在工程目录中执行
scrapy startproject 工程名


工程目录下 创建爬虫spider
scrapy genspider mydomain mydomain.com




新增文件夹需要创建,加入package列表
_init_.py






# coding=utf-8 ##以utf-8编码储存中文字符



如果您觉得本文确实帮助了您,可以微信扫一扫,进行小额的打赏和鼓励,谢谢 ^_^


redhat系统安装scrapy爬虫框架步骤Python版_第1张图片





你可能感兴趣的:(网络爬虫,linux)