赵健乔

Docker部署redis-sentinel高可用性(HA)解决方案

Redis-Sentinel是Redis官方推荐的高可用性(HA)解决方案，当用Redis做Master-slave的高可用方案时，假如master宕机了，Redis本身(包括它的很多客户端)都没有实现自动进行主备切换，而Redis-sentinel本身也是一个独立运行的进程，它能监控多个master-slave集群，发现master宕机后能进行自动切换。

一、封装Docker镜像

1、创建redis Dockerfile

操作系统镜像采用debian:jessie-slim瘦身版
redis源码采用最新的stable版本, http://download.redis.io/releases/ ，端口采用6379

FROM debian:jessie-slim

# add our user and group first to make sure their IDs get assigned consistently, regardless of whatever dependencies get added
RUN groupadd -r redis && useradd -r -g redis redis

# grab gosu for easy step-down from root
# https://github.com/tianon/gosu/releases
ENV GOSU_VERSION 1.11
RUN set -ex; \
	\
	fetchDeps='ca-certificates wget'; \
	apt-get update; \
	apt-get install -y --no-install-recommends $fetchDeps; \
	rm -rf /var/lib/apt/lists/*; \
	\
	dpkgArch="$(dpkg --print-architecture | awk -F- '{ print $NF }')"; \
	wget -O /usr/local/bin/gosu "https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$dpkgArch"; \
	wget -O /usr/local/bin/gosu.asc "https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$dpkgArch.asc"; \
	export GNUPGHOME="$(mktemp -d)"; \
	gpg --keyserver ha.pool.sks-keyservers.net --recv-keys B42F6819007F00F88E364FD4036A9C25BF357DD4; \
	gpg --batch --verify /usr/local/bin/gosu.asc /usr/local/bin/gosu; \
	rm -r "$GNUPGHOME" /usr/local/bin/gosu.asc; \
	chmod +x /usr/local/bin/gosu; \
	gosu nobody true; \
	\
	apt-get purge -y --auto-remove $fetchDeps

ENV REDIS_VERSION stable
ENV REDIS_DOWNLOAD_URL http://download.redis.io/releases/redis-stable.tar.gz

# for redis-sentinel see: http://redis.io/topics/sentinel
RUN set -ex; \
	\
	buildDeps=' \
		wget \
		\
		gcc \
		libc6-dev \
		make \
	'; \
	apt-get update; \
	apt-get install -y $buildDeps --no-install-recommends; \
	rm -rf /var/lib/apt/lists/*; \
	\
	wget -O redis.tar.gz "$REDIS_DOWNLOAD_URL"; \
	mkdir -p /usr/src/redis; \
	tar -xzf redis.tar.gz -C /usr/src/redis --strip-components=1; \
	rm redis.tar.gz; \
	\
# disable Redis protected mode [1] as it is unnecessary in context of Docker
# (ports are not automatically exposed when running inside Docker, but rather explicitly by specifying -p / -P)
# [1]: https://github.com/antirez/redis/commit/edd4d555df57dc84265fdfb4ef59a4678832f6da
	grep -q '^#define CONFIG_DEFAULT_PROTECTED_MODE 1$' /usr/src/redis/src/server.h; \
	sed -ri 's!^(#define CONFIG_DEFAULT_PROTECTED_MODE) 1$!\1 0!' /usr/src/redis/src/server.h; \
	grep -q '^#define CONFIG_DEFAULT_PROTECTED_MODE 0$' /usr/src/redis/src/server.h; \
# for future reference, we modify this directly in the source instead of just supplying a default configuration flag because apparently "if you specify any argument to redis-server, [it assumes] you are going to specify everything"
# see also https://github.com/docker-library/redis/issues/4#issuecomment-50780840
# (more exactly, this makes sure the default behavior of "save on SIGTERM" stays functional by default)
	\
	make -C /usr/src/redis -j "$(nproc)"; \
	make -C /usr/src/redis install; \
	\
	rm -r /usr/src/redis; \
	\
	apt-get purge -y --auto-remove $buildDeps

RUN apt-get update && apt-get install -y vim

RUN mkdir /data && chown redis:redis /data
VOLUME /data
WORKDIR /data

RUN mkdir /etc/redis
ADD redis.conf /etc/redis/
COPY docker-entrypoint.sh /usr/local/bin/
ENTRYPOINT ["docker-entrypoint.sh"]

EXPOSE 6379
CMD ["redis-server"]

2、创建redis.conf

在redis源代码中提供的redis.conf的基础上进行修改

masterauth 当master服务设置了密码保护时，slave服务连接master的密码
requirepass 设置Redis连接密码

masterauth redis-secret-0123passw0rd
requirepass redis-secret-0123passw0rd

关闭高危命令

# Command renaming.
#
# It is possible to change the name of dangerous commands in a shared
# environment. For instance the CONFIG command may be renamed into something
# hard to guess so that it will still be available for internal-use tools
# but not available for general clients.
#
# Example:
#
# rename-command CONFIG b840fc02d524045429941cc15f59e41cb7be6c52
#
# It is also possible to completely kill a command by renaming it into
# an empty string:
#
# 删除所有数据库的所有key
rename-command FLUSHALL ""

# Lua解释器执行脚本
rename-command EVAL     ""

在dockerfile中增加如下命令：

RUN mkdir /etc/redis
ADD redis.conf /etc/redis/

替换redis源代码中的配置文件

3、创建docker-entrypoint.sh

#!/bin/sh
set -e

# first arg is `-f` or `--some-option`
# or first arg is `something.conf`
if [ "${1#-}" != "$1" ] || [ "${1%.conf}" != "$1" ]; then
	set -- redis-server "$@"
fi

# allow the container to be started with `--user`
if [ "$1" = 'redis-server' -a "$(id -u)" = '0' ]; then
	chown -R redis .
	exec gosu redis "$0" "$@"
fi

exec "$@"

4、创建redis镜像

按照自定义的tag创建redis镜像

docker build -t wxsc/redis:stable .

查看镜像信息

二、封装redis-sentinel容器

一主两从三哨兵，可以，如果两个哨兵投了不一样的，就再投一遍。

1、创建redis-sentinel Dockerfile

FROM wxsc/redis:stable

EXPOSE 26379
ADD sentinel.conf /etc/redis/sentinel.conf
RUN chown redis:redis /etc/redis/sentinel.conf
ENV SENTINEL_QUORUM 2
ENV SENTINEL_DOWN_AFTER 30000
ENV SENTINEL_FAILOVER 180000
COPY sentinel-entrypoint.sh /usr/local/bin/
RUN chmod +x /usr/local/bin/sentinel-entrypoint.sh
ENTRYPOINT ["sentinel-entrypoint.sh"]

2、创建 sentinel-entrypoint.sh

#!/bin/sh

sed -i "s/\$SENTINEL_QUORUM/$SENTINEL_QUORUM/g" /etc/redis/sentinel.conf
sed -i "s/\$SENTINEL_DOWN_AFTER/$SENTINEL_DOWN_AFTER/g" /etc/redis/sentinel.conf
sed -i "s/\$SENTINEL_FAILOVER/$SENTINEL_FAILOVER/g" /etc/redis/sentinel.conf

exec docker-entrypoint.sh redis-server /etc/redis/sentinel.conf --sentinel

3、创建 sentinel.conf

# Example sentinel.conf can be downloaded from http://download.redis.io/redis-stable/sentinel.conf

port 26379

dir /tmp

sentinel monitor mymaster redis-master 6379 $SENTINEL_QUORUM

sentinel auth-pass mymaster redis-secret-0123passw0rd

sentinel down-after-milliseconds mymaster $SENTINEL_DOWN_AFTER

sentinel parallel-syncs mymaster 1

sentinel failover-timeout mymaster $SENTINEL_FAILOVER

4、创建docker-compose.yml

master:
  image: wxsc/redis:stable
  container_name: redis-cluster_master
  command: redis-server /etc/redis/redis.conf
  ports:
    - "6379:6379"
slave_1:
  image: wxsc/redis:stable
  container_name: redis-cluster_slave_1
  command: redis-server /etc/redis/redis.conf --slaveof redis-master 6379
  links:
    - master:redis-master
  ports:
    - "6380:6379"
slave_2:
  image: wxsc/redis:stable
  container_name: redis-cluster_slave_2
  command: redis-server /etc/redis/redis.conf --slaveof redis-master 6379
  links:
    - master:redis-master
  ports:
    - "6381:6379"
sentinel_1:
  build: sentinel
  container_name: redis-cluster_sentinel_1
  command: redis-sentinel /etc/redis/sentinel.conf
  environment:
    - SENTINEL_DOWN_AFTER=5000
    - SENTINEL_FAILOVER=5000
  links:
    - master:redis-master
    - slave_1
    - slave_2
  ports:
    - "6390:6379"
    - "26390:26379"
sentinel_2:
  build: sentinel
  container_name: redis-cluster_sentinel_2
  environment:
    - SENTINEL_DOWN_AFTER=5000
    - SENTINEL_FAILOVER=5000
  links:
    - master:redis-master
    - slave_1
    - slave_2
  ports:
    - "6391:6379"
    - "26391:26379"
sentinel_3:
  build: sentinel
  container_name: redis-cluster_sentinel_3
  environment:
    - SENTINEL_DOWN_AFTER=5000
    - SENTINEL_FAILOVER=5000
  links:
    - master:redis-master
    - slave_1
    - slave_2
  ports:
    - "6392:6379"
    - "26392:26379"

5、安装Docker Compose

yum -y install epel-release
yum -y install python-pip
pip install -U docker-compose

错误： AttributeError: ‘module’ object has no attribute 'GSSException’

pip install paramiko==2.0.2

6、创建redis-sentinel镜像

docker-compose build

7、创建redis-sentinel集群

docker-compose up -d

查看镜像：

查看容器：

三、redis-sentinel集群外网访问

使用docker部署redis sentinel集群时，若要对宿主机外部提供服务，需要在配置各个容器时使用host网络模式；默认的bridge模式将导致sentinel容器返回master容器的内部IP，外部无法访问。

1、修改redis-sentinel Dockerfile

FROM wxsc/redis:stable

ADD sentinel.conf /etc/redis/sentinel.conf
RUN chown redis:redis /etc/redis/sentinel.conf
ENV SENTINEL_QUORUM 2
ENV SENTINEL_DOWN_AFTER 30000
ENV SENTINEL_FAILOVER 180000
COPY sentinel-entrypoint.sh /usr/local/bin/
RUN chmod +x /usr/local/bin/sentinel-entrypoint.sh
ENTRYPOINT ["sentinel-entrypoint.sh"]

2、创建redis-sentinel镜像

按照自定义的tag创建redis-sentinel镜像

docker build -t wxsc/redis-sentinel:stable .