【Scrapy学习】 使用supervisor对scrapyd进程进行监控和管理

2019独角兽企业重金招聘Python工程师标准>>> hot3.png

CentOS 6.5 64 位 scrapy 1.3.3 python 2.7 scrapyd 1.1.1

首先使用pip安装supervisor

# pip install supervisor

输出配置文件到 /etc/supervisord.conf
# echo_supervisord_conf > /etc/supervisord.conf

修改 supervisord.conf配置文件:

[unix_http_server]
file=/tmp/supervisor.sock   ; (the path to the socket file)
chmod=0775                 ; socket file mode (default 0700)
[inet_http_server]         ; inet (TCP) server disabled by default
port=127.0.0.1:9001        ; (ip_address:port specifier, *:port for all iface) # web服务的IP和端口号
username=user              ; (default is no username (open server))  # 登录web界面的用户名
password=123               ; (default is no password (open server)) # 登录web界面的密码
[supervisord]
logfile=/tmp/supervisord.log ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=50MB        ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10           ; (num of main logfile rotation backups;default 10)
loglevel=info                ; (log level;default info; others: debug,warn,trace)
pidfile=/tmp/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=false               ; (start in foreground if true;default false)
minfds=1024                  ; (min. avail startup file descriptors;default 1024)
minprocs=200                 ; (min. avail process descriptors;default 200)
[supervisorctl]
serverurl=http://127.0.0.1:9001 ; use an http:// url to specify an inet socket
[include]
files = /etc/supervisor/*.conf

在 /etc/文件夹下创建 supervisor 文件夹,并在文件夹下创建 scrapyd.conf 配置文件 scrapyd.conf 文件:

[program:scrapyd]
command=scrapyd
autostart=true
autorestart=true

设置supervisor 自启动

在 /etc/init.d/ 下创建文件夹 supervisord

/etc/init.d/supervisord 内容如下:

#! /bin/sh
PATH=/usr/local/bin:/usr/local/sbin:/sbin:/bin:/usr/sbin:/usr/bin
PROGNAME=supervisord 
DAEMON=/usr/local/bin/$PROGNAME # supervisord 文件执行路径
CONFIG=/etc/$PROGNAME.conf # supervisord.conf 文件的存储路径
PIDFILE=/tmp/$PROGNAME.pid # supervisord.pid 的存储路径 在supervisord.conf 文件中设置
DESC="supervisord daemon"
SCRIPTNAME=/etc/init.d/$PROGNAME
# Gracefully exit if the package has been removed.
test -x $DAEMON || exit 0
start() # start 函数开始 
{
        echo -n "Starting $DESC: $PROGNAME"
        $DAEMON -c $CONFIG
        echo "..."
}
stop()
{
        echo -n "Stopping $DESC: $PROGNAME"
        supervisor_pid=$(cat $PIDFILE)
        kill -15 $supervisor_pid
        echo "..."
}
case "$1" in
  start)
        start
        ;;
  stop)
        stop
        ;;
  restart)
        stop
        start
        ;;
  *)
        echo "Usage: $SCRIPTNAME {start|stop|restart}" >&2
        exit 1
        ;;
esac
exit 0

保存后,执行以下命令修改文件权限:```

# chmod 777 /etc/init.d/supervisord

#  /etc/init.d/supervisord start
或者
# service supervisord start

配置开机启动:

chkconfig supervisord on

查看是否成功:

chkconfig --list | grep supervisord

在设置好 supervisor 自启动之后,发现supervisor文件找不到scrapyd 的执行路径,更改 scrapyd.conf 的配置文件:

[program:scrapyd]
command=/usr/local/bin/scrapyd # scrapyd 的执行路径
autostart=true
autorestart=true

转载于:https://my.oschina.net/whitejavadog/blog/906665

你可能感兴趣的:(【Scrapy学习】 使用supervisor对scrapyd进程进行监控和管理)