探究
想法是，手头上有2台服务器。如果有爬抓数据任务，这两台机器同时工作并且最大程度上利用网络和多核CPU资源我又通过multiprocess开启多进程，提高速度。我在之前抓京东数据时做了一份统计
1进程 147秒
2进程 67秒
3进程 58秒
4进程 56秒
我本身的机器是4核，用的是公司的百兆光纤网。当然分配给每一个IP肯定做了速度限制。
但是从上面的数据可以看出，多核还是非常有用的。假设你的任务queue，那么多核抓取也非常合适的代码类似于这样，可以参考我的做法。
p = Pool(3)
for item in items:
p.apply_async(crwal, args=(item,))
print('Waiting for all subprocesses done...')
p.close()
p.join()
上面的也不算废话，想象一下如果使用2台机器去跑，那么抓取时间理论上减半。3台、4台....N台 ，抓取时间就是 time/N 那么我们实现分布式部署一个可拓展，稳定，容错的可靠系统还是非常有必要的。
那么分布式到底如何去做呢？我在网上查阅了一些资料，查到Python有个分布式模型Celery，看下它的架构图。
Celery架构图
User 用户即使用者
broker 消息中间人Celery 需要一个发送和接收消息的解决方案，其通常以独立服务形式出现, 仔细观察实习AMQP，这是高级消息队列协议的缩写。实现这个协议的MQ非常多，官网主推RabbitMQ和Redis ，而前者更是推荐，因为后者容易受到突然断电会丢失数据。请看 中文官网解释
workers 工作机
task result store 结果存储。这样的好处是完全分离。互不依赖。
因此Celery+RabbitMQ的结合架构是这样子的(图片来源于网络)
image.png
RabbitMQ安装
系统环境：CentOS7.4
yum -y install erlang
下载rpm 若链接不上，建议直接复制地址到迅雷下载，会很快。然后再rz导入到系统里即可。
wget http://www.rabbitmq.com/releases/rabbitmq-server/v3.6.6/rabbitmq-server-3.6.6-1.el7.noarch.rpm
安装
yum install rabbitmq-server-3.6.6-1.el7.noarch.rpm
启动
sudo systemctl start rabbitmq-server
报错类似于
-- Subject: Unit rabbitmq-server.service has begun start-up
-- Defined-By: systemd
Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
▽-
-- Unit rabbitmq-server.service has begun starting up.
Jul 10 14:21:20 l22-30-91 rabbitmq-server[6275]: ERROR: epmd error for host l22-30-91: address (cannot connect to host/port)
Jul 10 14:21:20 l22-30-91 systemd[1]: rabbitmq-server.service: main process exited, code=exited, status=1/FAILURE
Jul 10 14:21:20 l22-30-91 systemd[1]: Cannot find unit for notify message of PID 6556.
Jul 10 14:21:21 l22-30-91 rabbitmqctl[6447]: Stopping and halting node 'rabbit@l22-30-91' ...
Jul 10 14:21:24 l22-30-91 rabbitmqctl[6447]: Error: unable to connect to node 'rabbit@l22-30-91': nodedown
Jul 10 14:21:27 l22-30-91 rabbitmqctl[6447]: DIAGNOSTICS
Jul 10 14:21:27 l22-30-91 rabbitmqctl[6447]: ===========
Jul 10 14:21:27 l22-30-91 rabbitmqctl[6447]: attempted to contact: ['rabbit@l22-30-91']
Jul 10 14:21:27 l22-30-91 rabbitmqctl[6447]: rabbit@l22-30-91:
Jul 10 14:21:27 l22-30-91 rabbitmqctl[6447]: * unable to connect to epmd (port 4369) on l22-30-91: address (cannot connect to host/port)
Jul 10 14:21:27 l22-30-91 rabbitmqctl[6447]: current node details:
Jul 10 14:21:27 l22-30-91 rabbitmqctl[6447]: - node name: 'rabbitmq-cli-09@l22-30-91'
Jul 10 14:21:27 l22-30-91 rabbitmqctl[6447]: - home dir: .
Jul 10 14:21:27 l22-30-91 rabbitmqctl[6447]: - cookie hash: xTHTDK3u5qC0QMJxWs38NQ==
Jul 10 14:21:27 l22-30-91 systemd[1]: Failed to start RabbitMQ broker.
-- Subject: Unit rabbitmq-server.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit rabbitmq-server.service has failed.
--
-- The result is failed.
Jul 10 14:21:27 l22-30-91 systemd[1]: Unit rabbitmq-server.service entered failed state.
Jul 10 14:21:27 l22-30-91 systemd[1]: rabbitmq-server.service failed.
Jul 10 14:21:27 l22-30-91 polkitd[477]: Unregistered Authentication Agent for unix-process:6259:33349831 (system bus name :1.12504, object path /org/freedesktop/Polic
这段异常有一个地方很重要。
Jul 10 14:21:20 l22-30-91 rabbitmq-server[6275]: ERROR: epmd error for host l22-30-91: address (cannot connect to host/port)
记住 l22-30-91 这是你的hostname
vim /etc/hosts
在里面新加一行
127.0.0.1 your-hostname
设置开机启动，并启动即可，查看状态
sudo systemctl enable rabbitmq-server
sudo systemctl start rabbitmq-server
sudo systemctl stop rabbitmq-server
打印出
# sudo systemctl status rabbitmq-server
● rabbitmq-server.service - RabbitMQ broker
Loaded: loaded (/usr/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2018-07-10 14:22:51 CST; 39min ago
Process: 6447 ExecStop=/usr/sbin/rabbitmqctl stop (code=exited, status=0/SUCCESS)
Main PID: 6744 (beam.smp)
Status: "Initialized"
Memory: 45.9M
CGroup: /system.slice/rabbitmq-server.service
├─6744 /usr/lib64/erlang/erts-5.10.4/bin/beam.smp -W w -A 128 -P 1048576 -t 5000000 -stbt db -zdbbl 32000 -K true -- -root /usr/lib64/erlang -progname e...
├─6889 /usr/lib64/erlang/erts-5.10.4/bin/epmd -daemon
├─7097 inet_gethost 4
└─7098 inet_gethost 4
Jul 10 14:22:50 l22-30-91 rabbitmq-server[6744]: RabbitMQ 3.6.6. Copyright (C) 2007-2016 Pivotal Software, Inc.
Jul 10 14:22:50 l22-30-91 rabbitmq-server[6744]: ## ## Licensed under the MPL. See http://www.rabbitmq.com/
Jul 10 14:22:50 l22-30-91 rabbitmq-server[6744]: ## ##
Jul 10 14:22:50 l22-30-91 rabbitmq-server[6744]: ########## Logs: /var/log/rabbitmq/[email protected]
Jul 10 14:22:50 l22-30-91 rabbitmq-server[6744]: ###### ## /var/log/rabbitmq/[email protected]
Jul 10 14:22:50 l22-30-91 rabbitmq-server[6744]: ##########
Jul 10 14:22:50 l22-30-91 rabbitmq-server[6744]: Starting broker...
Jul 10 14:22:51 l22-30-91 rabbitmq-server[6744]: systemd unit for activation check: "rabbitmq-server.service"
Jul 10 14:22:51 l22-30-91 systemd[1]: Started RabbitMQ broker.
Jul 10 14:22:51 l22-30-91 rabbitmq-server[6744]: completed with 0 plugins.
启用rabbitmq-management插件：
sudo rabbitmq-plugins enable rabbitmq_management
重启生效
sudo systemctl restart rabbitmq-server