MQTT协议已经成为当前物联网领域的关键技术之一,当前市面上主流的实现MQTT协议的产品主要有 EMQX、Mosquito、NanoMQ等。本文以EMQX开源版为基础,构建 MQTT Broker 集群,并使用 HAProxy代理 MQTT Broker 集群,由 HAProxy 开启双向认证,并终结TLS,HAProxy 到 MQTT Broker 集群的流量采用非加密模式。
设备端携带设备证书向NLB所在的域名发起业务请求,NLB将流量转发到 HAProxy 负载均衡服务,HAProxy校验设备证书的有效性,当校验通过后终结TLS,将流量分发到某个MQTT Broker节点。HAProxy的负载策略可采用循环负载。
{
"signing": {
"default": {
"expiry": "262800h"
},
"profiles": {
"intermediate": {
"usages": ["cert sign", "crl sign"],
"expiry": "700800h",
"ca_constraint": {
"is_ca": true,
"max_path_len": 1
}
},
"device": {
"usages": [
"client auth",
"signing",
"digital signing",
"key encipherment",
"server auth"
],
"expiry": "262800h"
}
}
}
}
{
"CN": "emqx.com",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "WuHan",
"O": "emqx",
"OU": "EMQX Root CA",
"ST": "China"
}
]
}
cfssl gencert -initca root.json | cfssljson -bare CaRoot
{
"CN": "intermediate.emqx.com",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "WuHan",
"L": "XS",
"O": "EMQX intermediate CA",
"OU": "EMQX intermediate CA"
}
]
}
cfssl gencert -ca=./CaRoot.pem -ca-key=./CaRoot-key.pem -config=./config.json -profile=intermediate ./intermediate.json | cfssljson -bare CaIntermediate
{
"CN": "DEVICEAXXXXXXXX",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "WuHan",
"L": "XS",
"O": "EMQX DEVICE",
"OU": "EMQX DEVICE"
}
]
}
cfssl gencert -ca=./CaRoot.pem -ca-key=./CaRoot-key.pem -config=./config.json -profile=device ./device-a-csr.json | cfssljson -bare device-a
{
"CN": "emqx.xxx.com",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "Hubei",
"L": "Wuhan",
"O": "emqx",
"OU": "EMQX Server"
}
]
}
cfssl gencert -ca=./CaRoot.pem -ca-key=./CaRoot-key.pem -config=./config.json -profile=device ./emqx-server-csr.json | cfssljson -bare emqx-server
emqx-server-key.pem
、证书申请文件 emqx-server.csr
、服务端证书emqx-server.pem
。cat emqx-server-key.pem emqx-server.pem |tee server-allinone.pem
修改/etc/haproxy/haproxy.cfg文件,开启MQTT Broker访问端口,并开启双向认证。
frontend emqx-tls
bind *:8091 ssl crt /etc/emqx/certs/server-allinone.pem ca-file /etc/haproxy/certs/CaIntermediate.pem ca-verify-file /etc/haproxy/certs/CaRoot.pem verify required
# 一层证书信任模型,设备证书直接由根证书签发时,ca-file 配置根证书
# bind *:8091 ssl crt /etc/emqx/certs/server-allinone.pem ca-file /etc/haproxy/certs/CaRoot.pem verify required
mode tcp
log global
option tcplog
default_backend emqx-backend
backend emqx-backend
mode tcp
option tcplog
balance roundrobin
default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 500000 maxqueue 10000 weight 100
server emqx-165 10.0.1.165:1883 check-send-proxy send-proxy-v2 check inter 10s fall 2 rise 5
server emqx-171 10.0.1.171:1883 check-send-proxy send-proxy-v2 check inter 10s fall 2 rise 5
当 HAProxy 设置了 check-send-proxy 与 send-proxy-v2 参数后,需要 EMQX 开启 proxy-protocol 协议。开启方式时修改 /etc/emqx/emqx.conf 配置文件,在 1883 监听端口配置 proxy-protocol = true
listeners.tcp.default {
bind = "0.0.0.0:1883"
max_connections = 500000
proxy_protocol = true
}
使用 MQTT客户端工具向NLB所在的IP地址发起连接请求,请求参数携带根CA证书,设备证书,设备私钥,详细信息如下图所示:
点击CONNECT按钮发起连接请求。
如上图所示,客户端成功连接到MQTT Broker。
HAProxy 作为一款非常优秀的四层均衡程序,在大量的生产项目中得到广泛的应用,相较于NGINX在七层负载场景中广泛应用,四层负载均衡服务在TCP服务负载业务场景上有着更好的性能优势。云商的NLB服务作为高可靠的负载均衡服务,为何不直接将NLB的流量负载分发到MQTT Broker节点,还额外的安装HAProxy负载均衡程序,主要原因是云商NLB的功能通常比较基础,无法满足企业额外的定制化需求,所以,大量的业务场景中使用云商的NLB作为第一层负载入口,定制化的业务处理交给后边的HAProxy负载均衡服务实现。