2019-04-17 分布式爬虫

服务端 scrapyd

1 环境安装

sudo apt update -y 
sudo apt install  -y build-essential libssl-dev libffi-dev libxml2 libxml2-dev libxslt1-dev zlib1g-dev python-dev
sudo pip install -y scrapy redis python_redis pymongo scrapyd 

2 配置

find / -name scrapyd

bind 0.0.0.0

3 运行

scrapyd

客户端 scrapyd-client

1 web端查看

http://localhost:6800

2 打包上传

 scrapyd-deploy

3 调度

curl http://localhost:6800/schedule.json -d project=zhihuuser -d spider=zhihu

4 取消

curl http:/localhost:6800/cancel.json -d project=zhihuuser -d job=51e8b2c0602011e9b97c000c2989fdcb