OpenResty动态更新后端地址

OpenResty

需求：

OpenResty运行在Docker中，后端代理的server存在IP不固定的情况，需要OpenResty动态更新后端地址

实现思路：

1. 在OpenResty中定义共享Dict，将后端IP保存在其中
2. 设置/add和/delete的location，通过Web调用增删后端IP
3. 设置proxy_pass代理，代理的地址从共享Dict中获取实现请求

过程验证(这里所有服务在一个Centos 7 Docker中启动)：

1. 启动两个后端服务

• 启动三个终端，在第一个终端中启动web服务test1

[root@f2d9f997edac /]# mkdir -p test1 && cd test1 && echo test1 > test
[root@f2d9f997edac test1]# python -m SimpleHTTPServer 8081
Serving HTTP on 0.0.0.0 port 8081 ...

• 在第二个终端中启动web服务test2

[root@f2d9f997edac /]# mkdir -p test2 && cd test2 && echo test2 > test
[root@f2d9f997edac test2]# python -m SimpleHTTPServer 8082
Serving HTTP on 0.0.0.0 port 8082 ...

• 在第三个终端中验证test1和test2

[root@f2d9f997edac /]# curl 127.0.0.1:8081/test
test1
[root@f2d9f997edac /]# curl 127.0.0.1:8082/test
test2

2. 安装OpenResty并配置

• 在第三个终端中，执行下面命令安装Openresty（此处为CentOS系统，其余系统的安装命令请自行搜索）

[root@f2d9f997edac /]# yum install -y openresty

• 编辑配置文件，验证配置文件并启动Openrestry

[root@f2d9f997edac /]# cat /usr/local/openresty/nginx/conf/nginx.conf
worker_processes  4;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    #access_log  logs/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    lua_shared_dict ups 10m;

    server {
        listen 80;
        location /add {
            content_by_lua_block {
            local ups = ngx.shared.ups
                local ip = ngx.req.get_uri_args()["ip"]
                local port = ngx.req.get_uri_args()["port"]
                local no = ngx.req.get_uri_args()["no"]
                if ip == nil or port == nil or no == nil then
                    ngx.say("usage: /add?ip=x.x.x.x&port=xx&no=xx")
                    return
                end
                ups:set("host-"..no, ip)
                ups:set("port-"..no, port)
                -- ups["host-"..no] = ip
                -- ups["port-"..no] = port
                local count = 0
                for k in pairs(ups:get_keys(0)) do count = count + 1  end
                ups:set("servers", count)
                ngx.say("OK")
            }
        }
        location /get {
            content_by_lua_block {
                local ups = ngx.shared.ups
                local no = ngx.req.get_uri_args()["no"]
                if no == nil then
                    ngx.say("usage: /get?no=xx")
                    return
                end
                if ups:get("host-" .. no) == nil then
                    ngx.say("the no server is vaild: " .. no)
                    return
                end
                local count = 0
                for k in pairs(ups:get_keys(0)) do count = count + 1  end
                ups:set("servers", count)
                ngx.say("ip: " .. ups:get("host-" .. no))
                ngx.say("port: " .. ups:get("port-" .. no))
                ngx.say("servers: " .. math.floor(ups:get("servers")/2))
            }
        }
        location /delete {
            content_by_lua_block {
                local ups = ngx.shared.ups
                local no = ngx.req.get_uri_args()["no"]
                if no == nil then
                    ngx.say("usage: /get?no=xx")
                    return
                end
                if ups:get("host-" .. no) == nil then
                    ngx.say("the no server is vaild: " .. no)
                    return
                end
                ups:delete("host-" .. no)
                ups:delete("port-" .. no)
                ngx.say("OK")
            }
        }

        location / {
            set_by_lua_block $cur_ups {
                local ups = ngx.shared.ups
                local items = ups:get("servers")
                local n = math.random(math.floor(items/2))
                local ss = string.gsub(ups:get_keys(0)[n], "%a+-", "")
                s = ups:get("host-"..ss) .. ":" .. ups:get("port-"..ss)
                return s
            }
            proxy_next_upstream off;
            proxy_set_header    Host $host;
            proxy_http_version  1.1;
            proxy_set_header    Connection  "";
            proxy_pass $scheme://$cur_ups;
        }
    }
}
[root@f2d9f997edac /]# openresty -t
nginx: the configuration file /usr/local/openresty/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/openresty/nginx/conf/nginx.conf test is successful
[root@f2d9f997edac /]# openresty

3. 增加后端服务并验证（/get通过no查询后端服务的IP和端口）

• 在add的时候指定后端服务的no(序列号)、IP和端口信息
• no用来区分不同后端服务，执行相同no的提交会覆盖原来的no信息
• proxy_pass拿到的后端服务随机产生(2中的math.random(math.floor(items/2))处)

[root@f2d9f997edac /]# curl '127.0.0.1:80/add?no=1&ip=127.0.0.1&port=8081'
OK
[root@f2d9f997edac /]# curl '127.0.0.1:80/get?no=1'
ip: 127.0.0.1
port: 8081
servers: 1
[root@f2d9f997edac /]# curl '127.0.0.1:80/add?no=2&ip=127.0.0.1&port=8082'
OK
[root@f2d9f997edac /]# curl '127.0.0.1:80/get?no=2'
ip: 127.0.0.1
port: 8082
servers: 2
[root@f2d9f997edac /]# curl '127.0.0.1:80/test'
test1
[root@f2d9f997edac /]# curl '127.0.0.1:80/test'
test2
[root@f2d9f997edac /]#

4. 删除后端服务并验证
   /delete通过no删除后端服务的IP和端口信息

[root@f2d9f997edac /]# curl '127.0.0.1:80/delete?no=1'
OK
[root@f2d9f997edac /]# curl '127.0.0.1:80/get?no=1'
the no server is vaild: 1
[root@f2d9f997edac /]# curl '127.0.0.1:80/get?no=2'
ip: 127.0.0.1
port: 8082
servers: 1
[root@f2d9f997edac /]# curl '127.0.0.1:80/test'
test2
[root@f2d9f997edac /]# curl '127.0.0.1:80/test'
test2
[root@f2d9f997edac /]#

存在不足：

1. 如果由程序控制OpenResty的/add和/delete来维护后端服务信息，需要保证与序号no的对应
2. proxy_pass中的后端服务地址和端口对通过随机拿到，存在负载不均衡的情况，可以根据自己的实际需求修改，比如修改为ip_hash，但需要考虑保存在共享内存Dict中后端服务的数量