docker-compose搭建consul集群环境

摘要
Consul 有以下几个关键特性:

①服务发现：支持服务发现。你可以通过 DNS 或 HTTP 的方式获取服务信息。
②健康检查：支持健康检查。可以提供与给定服务相关联的任何数量的健康检查（如 web 状态码或 cpu 使用率）。
③K/V 存储：键/值对存储。你可用通过 consul 存储如动态配置之类的相关信息。
④多数据中心：支持多数据中心，开箱即用。
⑤WEB-UI：支持 WEB-UI。点点点，你就能够了解你的服务现在的运行情况，一目了然，对开发运维是非常友好的。

一、案例概述
公司提出一个新需求，使用Docker将Consul、Consul Template、Registrator和Nginx组装成一个值得信任且可扩展的服务框架，可在这个框架中添加和移除服务，不需要重写任何配置，也不需要重启任何服务，一切都能正常运行

1.1、Docker Compose
1.1.1、Docker Compose的前身是Fig，它是一个定义及运行多个Docker容器的工具

1.1.2、使用Docker Compose不再需要使用Shell脚本来启动容器

1.1.3、Docker Compose非常适合组合多个容器进行开发的场景

1.2、Consul
1.2.1、Consul是HashiCorp公司推出的开源工具，用于实现分布式系统的服务发现与配置

1.2.2、Consul的特性

支持健康检查，允许存储键值对

基于Golong语言，可移植性强

支持ACL访问控制

1.2.3、与Docker等轻量级容器可无缝配合

二、使用docker-compose搭建consul集群环境
2.1、案例环境
2.2、案例需求
①实现单机网络下容器与容器之间互通
②使用Docker Compose创建容器
③搭建Consul服务实现自动发现和更新

2.3、Docker Compose容器编排
2.3.1、YAML是一种标记语言很直观的数据序列化格式
2.3.2、文件格式及编写注意事项
①不支持制表符tab键缩进，需要使用空格缩进
②通常开头缩进2个空格
③字符后缩进1个空格，如冒号、逗号、横杆
④用#号注释
⑤如果包含特殊字符用单引号引起来
⑥布尔值必须用引号括起来
2.4、Docker Compose配置常用字段
2.5、Docker Compose常用命令
2.6、Compose命令说明
2.6.1、基本的使用格式

 docker-compose [options] [COMMAND] [ARGS...]

2.6.2、docker-compose选项

–verbose 输出更多调试信息
–version 打印版本并退出
-f,–file FILE使用特定的compose模板文件，默认为docker-compose.yml
-p,–project-name NAME 指定项目名称，默认使用目录名称

三、案例部署
3.1、server1和server2安装docker
3.2、server1上安装Compose

上传docker-compose到/usr/local/bin
[root@server1 bin]# chmod +x /usr/local/bin/docker-compose 
[root@server1 bin]# mkdir compose
[root@server1 bin]# cd compose/
[root@server1 compose]# vim docker-compose.yml
version: '3'
services:
  nginx:
    hostname: nginx
    build:
      context: /usr/local/bin/compose/
      dockerfile: Dockerfile
    ports:
     - 1217:80
     - 1218:443
    networks:
     - xxy
    volumes:
     - ./wwwroot:/usr/local/nginx/html
networks:
  xxy:
[root@server1 compose]# vim Dockerfile
FROM centos:7
MAINTAINER nginx
RUN yum -y update
RUN yum -y install gcc gcc-c++ make pcre-devel zlib-devel
ADD nginx-1.12.2.tar.gz /opt
WORKDIR /opt/nginx-1.12.2
RUN ./configure \
--prefix=/usr/local/nginx \
--user=nginx \
--group=nginx \
--with-http_stub_status_module && make && make install
RUN useradd -M -s /sbin/nologin nginx
ENV PATN /usr/local/nginx/sbin:$PATH
EXPOSE 80
EXPOSE 443
RUN echo 'daemon off;' >> /usr/local/nginx/conf/nginx.conf
ADD run.sh /run.sh
RUN chmod 755 /run.sh
CMD ["/run.sh"]
[root@server1 compose]# vi run.sh
#!/bin/bash
/usr/local/nginx/sbin/nginx
[root@server1 compose]# ll
总用量 972
-rw-r--r--. 1 root root    251 11月 30 19:30 docker-compose.yml
-rw-r--r--. 1 root root    507 11月 30 19:38 Dockerfile
-rw-r--r--. 1 root root 981687 11月 27 11:38 nginx-1.12.2.tar.gz
-rw-r--r--. 1 root root     40 11月 30 19:39 run.sh
[root@server1 compose]# docker-compose -f docker-compose.yml up -d
...
Step 15/15 : CMD ["/run.sh"]
 ---> Running in 1baac9b54d8c
Removing intermediate container 1baac9b54d8c
 ---> 3694a389bb17
Successfully built 3694a389bb17
Successfully tagged compose_nginx:latest
WARNING: Image for service nginx was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
Creating compose_nginx_1 ... done

[root@server1 compose]# ll
总用量 972
-rw-r--r--. 1 root root    267 11月 30 19:46 docker-compose.yml
-rw-r--r--. 1 root root    507 11月 30 19:38 Dockerfile
-rw-r--r--. 1 root root 981687 11月 27 11:38 nginx-1.12.2.tar.gz
-rw-r--r--. 1 root root     40 11月 30 19:39 run.sh
drwxr-xr-x. 2 root root      6 11月 30 19:48 wwwroot
[root@server1 compose]# cd wwwroot/
[root@server1 wwwroot]# vi index.html
[root@server1 wwwroot]# cat index.html 
<h1>this is xxy!!!</h1>

测试
3.3、在server1上安装consul

[root@server1 ~]# mkdir consul
[root@server1 ~]# rz -E
rz waiting to receive.
[root@server1 ~]# cp consul_0.9.2_linux_amd64.zip /root/consul
[root@server1 ~]# cd consul/
[root@server1 consul]# unzip consul_0.9.2_linux_amd64.zip 
Archive:  consul_0.9.2_linux_amd64.zip
  inflating: consul                  
[root@server1 consul]# ll
总用量 51772
-rwxr-xr-x. 1 root root 42777946 8月  10 2017 consul
-rw-r--r--. 1 root root 10233177 11月 30 19:53 consul_0.9.2_linux_amd64.zip

[root@server1 consul]# consul agent \
> -server \
> -bootstrap \
> -ui \
> -data-dir=/var/lib/consul-data \
> -bind=20.0.0.10 \
> -client=0.0.0.0 \
> -node=consul-server01 &> /var/log/consul.log &
[1] 68658

root@server1 consul]# jobs 
[1]+  运行中               consul agent -server -bootstrap -ui -data-dir=/var/lib/consul-data -bind=20.0.0.10 -client=0.0.0.0 -node=consul-server01 &>/var/log/consul.log &
[root@server1 consul]# consul members
Node             Address         Status  Type    Build  Protocol  DC
consul-server01  20.0.0.10:8301  alive   server  0.9.2  2         dc1
[root@server1 consul]# consul info | grep leader
	leader = true
	leader_addr = 20.0.0.10:8300
[root@server1 consul]# docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS                                         NAMES
c6aa5de4da3a        compose_nginx       "/run.sh"           11 minutes ago      Up 11 minutes       0.0.0.0:1217->80/tcp, 0.0.0.0:1218->443/tcp   compose_nginx_1
[root@server1 consul]# docker exec -it c6aa5de4da3a /bin/bash
[root@nginx nginx-1.12.2]# yum -y install net-tools
[root@nginx nginx-1.12.2]# ifconfig 
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.18.0.2  netmask 255.255.0.0  broadcast 172.18.255.255
        ether 02:42:ac:12:00:02  txqueuelen 0  (Ethernet)
        RX packets 100  bytes 320242 (312.7 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 75  bytes 4497 (4.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1  (Local Loopback)
        RX packets 4  bytes 260 (260.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 4  bytes 260 (260.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

3.4、通过httpd api 获取集群信息

查看集群server成员
[root@server1 consul]# curl 127.0.0.1:8500/v1/status/peers
集群Raf leader
[root@server1 consul]# curl 127.0.0.1:8500/vi/status/leader
注册的所有服务
[root@server1 consul]# curl 127.0.0.1:8500/vi/catalog/services
查看nginx服务信息
[root@server1 consul]# curl 127.0.0.1:8500/vi/catalog/nginx
集群节点详细信息
[root@server1 consul]# curl 127.0.0.1:8500/vi/catalog/nodes

3.5、在server2上安装Gliderlabs/Registrator

安装registrator
[root@server2 ~]# docker run -d \
> --name=registrator \   #容器名称
> --net=host \          #指定网络
> -v /var/run/docker.sock:/tmp/docker.sock \   #挂载
> --restart=always \
> gliderlabs/registrator:latest \              #镜像名称
> -ip=20.0.0.20 \
> consul://20.0.0.10:8500

查看镜像
[root@server2 ~]# docker images
REPOSITORY               TAG                 IMAGE ID            CREATED             SIZE
gliderlabs/registrator   latest              3b59190c6c80        4 years ago         23.8MB

查看容器
[root@server2 ~]# docker ps -a
\CONTAINER ID        IMAGE                           COMMAND                  CREATED              STATUS              PORTS               NAMES
928b8c8580ad        gliderlabs/registrator:latest   "/bin/registrator -i…"   About a minute ago   Up About a minute                       registrator

网页登录http://20.0.0.10:8500,查看节点服务状态
3.6、在server2上安装两个httpd及nginx容器

[root@server2 ~]# docker run -dit -p 83:80 --name test1 -h test1 nginx
[root@server2 ~]# docker run -dit -p 84:80 --name test1 -h test2 nginx
[root@server2 ~]# docker run -dit -p 88:80 --name test3 -h test3 httpd
[root@server2 ~]# docker run -dit -p 89:80 --name test4 -h test4 httpd

查看容器状态
[root@server2 ~]# docker ps -a
CONTAINER ID        IMAGE                           COMMAND                  CREATED              STATUS              PORTS                NAMES
178332ffa010        httpd                           "httpd-foreground"       6 seconds ago        Up 4 seconds        0.0.0.0:89->80/tcp   test4
140f0c57c15a        httpd                           "httpd-foreground"       47 seconds ago       Up 46 seconds       0.0.0.0:88->80/tcp   test3
37b796ddca46        nginx                           "/docker-entrypoint.…"   About a minute ago   Up About a minute   0.0.0.0:84->80/tcp   test2
f47a5aaf1e4d        nginx                           "/docker-entrypoint.…"   2 minutes ago        Up 2 minutes        0.0.0.0:83->80/tcp   test1

网页登录http://20.0.0.10:8500,查看节点服务状态
将两个nginx服务移除查看，网页监控服务也被移除

[root@server2 ~]# docker stop 37b796ddca46
37b796ddca46
[root@server2 ~]# docker rm 37b796ddca46
37b796ddca46
[root@server2 ~]# docker stop f47a5aaf1e4d
f47a5aaf1e4d
[root@server2 ~]# docker rm f47a5aaf1e4d
f47a5aaf1e4d
[root@server2 ~]# docker ps -a
CONTAINER ID        IMAGE                           COMMAND                  CREATED             STATUS              PORTS                NAMES
178332ffa010        httpd                           "httpd-foreground"       5 minutes ago       Up 5 minutes        0.0.0.0:89->80/tcp   test4
140f0c57c15a        httpd                           "httpd-foreground"       6 minutes ago       Up 6 minutes        0.0.0.0:88->80/tcp   test3

3.7、在server1上安装consul-template
consul-template是一个守护进程，用于实时查询consul集群信息，并更新文件系统上任意数量的指定模板，生成配置文件，更新完成以后，可以选择运行shell命令执行更新操作，重新加载Nginx。consul-template可以查询consul中的服务目录、Key、Key-values等。这种强大的抽象功能和查询语言模板可以使consul-template特别适合动态的创建配置文件

准备template nginx 模板文件
[root@server1 consul]# vim nginx.ctmpl
upstream http_backend {
     
  {
     {
     range service "nginx"}}
   server {
     {
     .Address}}:{
     {
     .Port}};
    {
     {
     end}}
}

server {
     
  listen 83;
  server_name localhost 20.0.0.10;
  access_log /var/log/nginx/aa-access.log;
  index index.html index.php;
  location / {
     
    proxy_set_header HOST $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header Client-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_pass http://http_backend;
   }
}

编译安装nginx(将nginx上传到/root目录下)
[root@server1 consul]# yum -y install gcc gcc-c++ make pcre-devel zlib-devel
[root@server1 ~]# tar zxvf nginx-1.12.2.tar.gz -C /opt
[root@server1 ~]# cd /opt/nginx-1.12.2/
[root@server1 nginx-1.12.2]# ./configure --prefix=/usr/local/nginx
[root@server1 nginx-1.12.2]# make && make install

配置nginx
[root@server1 nginx-1.12.2]# vim /usr/local/nginx/conf/nginx.conf
http {
     
    include       mime.types;
    include       vhost/*.conf;        #添加虚拟主机目录
    default_type  application/octet-stream;

创建虚拟主机目录
[root@server1 nginx-1.12.2]# mkdir /usr/local/nginx/conf/vhost

创建日志文件目录
[root@server1 nginx-1.12.2]# mkdir /var/log/nginx

启动nginx并查看端口
[root@server1 nginx-1.12.2]# /usr/local/nginx/sbin/nginx
[root@server1 nginx-1.12.2]# netstat -anpt | grep nginx
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      69573/nginx: master

配置并启动template（上传template到/root目录下）
[root@server1 ~]# unzip consul-template_0.19.3_linux_amd64.zip      #解压缩
Archive:  consul-template_0.19.3_linux_amd64.zip
  inflating: consul-template
[root@server1 ~]# mv consul-template /usr/local/bin    #移动
[root@server1 ~]# consul-template -consul-addr 20.0.0.10:8500 -template "/root/consul/nginx.ctmpl:/usr/local/nginx/conf/vhost/xh.conf:/usr/local/nginx/sbin/nginx -s reload" --log-level=info

再开一个server1终端查看
[root@server1 ~]# cd /usr/local/nginx/conf/vhost/
[root@server1 vhost]# cat xh.conf
upstream http_backend {
     

    server 20.0.0.20:83;    #会自动识别并添加，等会将容器服务关闭后查看

 }

 server {
     
   listen 83;
   server_name localhost 20.0.0.10;
   access_log /var/log/nginx/aa-access.log;
   index index.html index.php;
    location / {
     
      proxy_set_header HOST $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header Client-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_pass http://http_backend;
   }
}

网页访问然后查看日志

server2上再启动一个nginx服务容器
[root@server2 ~]# docker run -dit -p 89:80 --name test6 -h test6 nginx
[root@server2 ~]# docker run -dit -p 88:80 --name test6 -h test6 nginx
在server1上查看
[root@server1 vhost]# cat xh.conf
upstream http_backend {
     

     server 20.0.0.20:83;

     server 20.0.0.20:88;

     server 20.0.0.20:89;       #节点自动添加

}

server {
     
       listen 83;
       server_name localhost 20.0.0.10;
       access_log /var/log/nginx/aa-access.log;
       index index.html index.php;
       location / {
     
          proxy_set_header HOST $host;
          proxy_set_header X-Real-IP $remote_addr;
          proxy_set_header Client-IP $remote_addr;
          proxy_set_header X-Fprwarded-For $proxy_add_x_forwarded_for;
          proxy_pass http://http_backend;
                 }
       }

在server2上查看日志（访问20.0.0.10可以访问后面的节点）
[root@server2 ~]# docker logs -f test5
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
20.0.0.10 - - [01/Dec/2020:05:45:42 +0000] "GET / HTTP/1.0" 200 612 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.67 Safari/537.36 Edg/87.0.664.47" "-"

[root@server2 ~]# docker logs -f test6
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
20.0.0.10 - - [01/Dec/2020:05:48:53 +0000] "GET / HTTP/1.0" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.67 Safari/537.36 Edg/87.0.664.47" "-"

删除一个容器查看能否自动移除

删除容器
[root@server2 ~]# docker stop 601bce2a82e6
601bce2a82e6
[root@server2 ~]# docker rm 601bce2a82e6
601bce2a82e6

查看会发现节点自动移除
[root@server1 vhost]# cat xh.conf
upstream http_backend {
     

     server 20.0.0.20:83;

     server 20.0.0.20:88;

}

server {
     
       listen 83;
       server_name localhost 20.0.0.10;
       access_log /var/log/nginx/aa-access.log;
       index index.html index.php;
       location / {
     
          proxy_set_header HOST $host;
          proxy_set_header X-Real-IP $remote_addr;
          proxy_set_header Client-IP $remote_addr;
          proxy_set_header X-Fprwarded-For $proxy_add_x_forwarded_for;
          proxy_pass http://http_backend;
                 }
       }

3.8、consul多节点

上传consul压缩包到server3上

解压缩
[root@server3 ~]# unzip consul_0.9.2_linux_amd64.zip
Archive:  consul_0.9.2_linux_amd64.zip
  inflating: consul
[root@server3 ~]# mv consul /usr/local/bin/

加入已有的群集中
[root@server3 ~]# consul agent \
> -bootstrap \
> -ui \
> -data-dir=/var/lib/consul-data \
> -server \
> -bind=20.0.0.30 \
> -client=0.0.0.0 \
> -node=consul-server3 \
> -enable-script-checks=true \
> -datacenter=dc1 \
> join 20.0.0.10 &> /var/log/consul.log &
[1] 17753

查看运行状态
[root@server3 ~]# jobs
[1]+  运行中               consul agent -bootstrap -ui -data-dir=/var/lib/consul-data -server -bind=20.0.0.30 -client=0.0.0.0 -node=consul-server3 -enable-script-checks=true -datacenter=dc1 join 20.0.0.10 &>/var/log/consul.log &