多机Nomad+Consul+consul-template+Nginx反向代理
目录
- 多机Nomad+Consul+consul-template+Nginx反向代理
-
- 1 Nomad与Consul连接
-
- 1.1 Nomad
- 1.2 Consul
- 1.3 Nomad Job(http-echo容器)
- 2 consul-template
- 3 Nginx反向代理
多机Nomad+Consul+consul-template+Nginx反向代理
虚拟机1：Ubuntu18.04，IP：192.168.10.11
虚拟机2：Ubuntu18.04，IP：192.168.10.22
测试过程：Nomad起2个job容器分别运行在2个虚拟机，consul通过check脚本检查2个容器，并将其IP和端口自动更新到consul-template中，虚拟机1输出一个文件。最后利用虚拟机1的Nginx反向代理进行域名访问2个容器。
对比单机上的测试，目的在于验证consul是否能发现非本机所运行的服务，并且通过template渲染IP端口到nginx配置文件。
此外，可以通过访问不同的域名，这些域名指向同一IP（虚拟机1），但反向代理到不同的IP和端口上。即只对外暴露虚拟机1的IP即可。
准备工作：修改docker用户权限，修改主机名以区分nomad结点
1 Nomad与Consul连接
1.1 Nomad
虚拟机1创建/home/.../nomad.d/nomad.hcl：
datacenter = "dc1"
data_dir = "/home/.../nomad/data"
server {
enabled = true
bootstrap_expect = 2
server_join {
retry_join = ["192.168.10.11:4648"，"192.168.10.22:4648"]
}
}
client {
enabled = true
servers = ["192.168.10.11:4647"]
}
虚拟机2创建/home/.../nomad.d/nomad.hcl：
datacenter = "dc1"
data_dir = "/home/.../nomad/data"
server {
enabled = true
bootstrap_expect = 2
server_join {
retry_join = ["192.168.10.11:4648"，"192.168.10.22:4648"]
}
}
client {
enabled = true
servers = ["192.168.10.22:4647"]
}
分别执行nomad agent -config /home/.../nomad.d启动nomad agent。
nomad server members和nomad node status可以查看状态。
1.2 Consul
虚拟机1创建/home/.../consul.d/consul.hcl：
{
"datacenter": "dc1",
"data_dir": "/home/.../consul/data",
"log_level": "INFO",
"node_name": "consul_server1",
"server": true,
"bind_addr": "192.168.10.11",
"client_addr": "0.0.0.0",
"bootstrap_expect":2,
"ui": true,
"enable_script_checks": true,
}
虚拟机2创建/home/.../consul.d/consul.hcl：
{
"datacenter": "dc1",
"data_dir": "/home/.../consul/data",
"log_level": "INFO",
"node_name": "consul_server2",
"server": true,
"bind_addr": "192.168.10.22",
"client_addr": "0.0.0.0",
"bootstrap_expect":2,
"ui": true,
"enable_script_checks": true,
}
虚拟机1创建/home/.../consul.d/http-echo.hcl：
service {
name = "http-echo"
port = 5678
check {
args = ["/home/.../consul/check.sh"]
interval = "3s"
}
}
虚拟机2创建/home/.../consul.d/http-echo2.hcl：
service {
name = "http-echo2"
port = 8765
check {
args = ["/home/.../consul/check2.sh"]
interval = "3s"
}
}
虚拟机1创建/home/.../consul/check.sh文件作为consul检测http-echo的脚本，注意chmod给权限：
#!/bin/bash
http=`docker ps | grep "http-echo" | grep "5678->5678"`
if [ -n "$http" ];#非空
then
exit 0
else
exit 2
fi
虚拟机2创建/home/.../consul/check2.sh文件作为consul检测http-echo2的脚本，注意chmod给权限：
#!/bin/bash
http=`docker ps | grep "http-echo" | grep "8765->8765"`
if [ -n "$http" ];#非空
then
exit 0
else
exit 2
fi
exit 0表示检测成功，1表示警告，其他表示失败。
分别执行consul agent -config-dir /home/.../consul.d启动consul agent。
consul members或浏览器访问192.168.10.11:8500查看状态。
1.3 Nomad Job(http-echo容器)
虚拟机1创建http-echo.nomad以备之后用：
job "http-echo" {
datacenters = ["dc1"]
constraint {
attribute = "${node.unique.name}"
value = "ubuntu1"
}
group "example" {
count = 1
network {
port "http" {
static = "5678"
}
}
task "server" {
driver = "docker"
config {
image = "hashicorp/http-echo"
ports = ["http"]
args = [
"-listen",
":5678",
"-text",
"hello world",
]
}
}
}
}
虚拟机2创建http-echo2.nomad以备之后用：
job "http-echo2" {
datacenters = ["dc1"]
constraint {
attribute = "${node.unique.name}"
value = "ubuntu2"
}
group "example2" {
count = 1
network {
port "http2" {
static = "8765"
}
}
task "server2" {
driver = "docker"
config {
image = "hashicorp/http-echo"
ports = ["http2"]
args = [
"-listen",
":8765",
"-text",
"hello world222",
]
}
}
}
}
constraint是限定参数，${node.unique.name}表示限定节点名唯一，即只能在特定结点运行job。目的是保证两个Job分别在两个虚拟机运行。
2 consul-template
源码：https://github.com/hashicorp/consul-template
二进制文件：https://releases.hashicorp.com/consul-template/
参考：configuration-file，Templating Language
虚拟机分别给权限：
sudo cp ./consul-template /usr/bin/consul-template
sudo chmod 777 /usr/bin/consul-template
虚拟机1创建两个template配置文件consul-template-config.hcl和consul-template-config2.hcl ：
consul {
address = "192.168.10.11:8500"
retry {
enabled = true
attempts = 12
backoff = "1s"
}
}
template {
source = "/home/.../consul/http-echo.ctmpl"
destination = "/home/.../consul/ip_port.txt"
perms = 0600
command = "/home/.../consul/template.sh"
}
consul {
address = "192.168.10.22:8500"
retry {
enabled = true
attempts = 12
backoff = "1s"
}
}
template {
source = "/home/.../consul/http-echo2.ctmpl"
destination = "/home/.../consul/ip_port2.txt"
perms = 0600
command = "/home/.../consul/template2.sh"
}
source表示模版文件，destination表示输出文件，perms表示文件访问权限，command表示每次变化执行的命令。
虚拟机1创建2个模版文件/home/.../consul/http-echo.ctmpl和/home/.../consul/http-echo2.ctmpl：
{{range service "http-echo"}}server {{.Address}}:{{.Port}};{{end}}
{{range service "http-echo2"}}server {{.Address}}:{{.Port}};{{end}}
创建空文件/home/.../consul/ip_port.txt、/home/.../consul/ip_port2.txt和/home/.../consul/ng.conf、/home/.../consul/ng2.conf，或者perms改成0644允许自动创建。
虚拟机1创建两个template脚本/home/.../consul/template.sh和/home/.../consul/template2.sh如下，注意chmod给权限：
#!/bin/bash
txt=`cat ./ip_port.txt`
if [[ $txt =~ "server" ]];
then
echo "upstream http_backend {" | cat > ./ng.conf
cat ./ip_port.txt >> ./ng.conf
echo "
}
server {
listen 80;
server_name test.com;
location / {
proxy_pass http://http_backend;
proxy_redirect off;
proxy_set_header Host \$host;
proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;
proxy_set_header X-Real-IP \$remote_addr;
}
}" | cat >> ./ng.conf
echo "a" | sudo -S service nginx reload
else
> ./ng.conf
fi
#!/bin/bash
txt=`cat ./ip_port2.txt`
if [[ $txt =~ "server" ]];
then
echo "upstream http_backend2 {" | cat > ./ng2.conf
cat ./ip_port2.txt >> ./ng2.conf
echo "
}
server {
listen 80;
server_name test2.com;
location / {
proxy_pass http://http_backend2;
proxy_redirect off;
proxy_set_header Host \$host;
proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;
proxy_set_header X-Real-IP \$remote_addr;
}
}" | cat >> ./ng2.conf
echo "a" | sudo -S service nginx reload
else
> ./ng2.conf
fi
因为如果upstream中没有server，service nginx reload会报错，所以需要判断一下输出文件是否有内容，没有的话就把nginx反向代理的配置文件设为空。
执行consul-template:
consul-template \
-config=consul-template-config.hcl \
-config=consul-template-config2.hcl
3 Nginx反向代理
安装Nginx：https://blog.csdn.net/weixin_43739110/article/details/121079232
在虚拟机1中/etc/nginx/nginx.conf加：
...
http{
...
include /home/.../consul/*.conf;
...
}
...
service nginx restart重启Nginx。
nomad job run http-echo.nomad和nomad job run http-echo2.nomad启动之前创建的两个Nomad job容器。
浏览器访问192.168.10.11:8500可以结果。
运行容器后，会看到ip_port.txt和ip_port2.txt中的内容变为：
server 192.168.10.11:5678;
server 192.168.10.22:8765;
/home/.../consul/ng.conf中的内容变为：
upstream http_backend {
server 192.168.10.11:5678;
}
server {
listen 80;
server_name test.com;
location / {
proxy_pass http://http_backend;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Real-IP $remote_addr;
}
}
/home/.../consul/ng2.conf中的内容变为：
upstream http_backend2 {
server 192.168.10.22:8765;
}
server {
listen 80;
server_name test2.com;
location / {
proxy_pass http://http_backend2;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Real-IP $remote_addr;
}
}
/etc/hosts文件中添加：
192.168.10.11 test.com
192.168.10.22 test2.com
浏览器登陆test.com和test2.com可以看到（或执行curl test.com、curl test2.com命令）：
nomad job stop http-echo关闭Nomad job，会看到/home/.../consul/ip_port.txt和/home/.../consul/ng.conf中的内容消失。
浏览器登陆test.com会502错误。