一、背景
Sonatype当前不支持Nexus负载平衡，除非将Nexus实例置于启用智能代理或通过更新功能的两个实例之前。
关于nexus作为docker镜像仓库的高可用设置，网上nexus+keepalived+rsync的方式挺扯的，官方开源版本默认又不支持高可用，那么通过nginx代理不失为一个办法，只是需要将镜像推两遍，每个仓库推一遍。
二、实验环境
nexus01 192.168.1.105
nexus02 192.168.1.106
nginx_vip 192.168.1.108（nexus.example.com）
装有docker的客户端 192.168.1.107
三、nexus服务端和客户端安装配置
在nexus服务器上
在nexus01服务器上安装nexus，生成keystore.jks
# keytool \
-genkeypair \
-keystore keystore.jks \
-alias Test_nexus \
-keypass Nexus@123 \
-storepass Nexus@123 \
-keyalg RSA \
-keysize 2048 \
-validity 5000 \
-dname "CN=*.example.com,OU=Test,O=Test,L=ShenZhen,ST=GuangDong,C=CN" \
-ext "SAN=DNS:nexus.example.com,IP:192.168.1.105" \
-ext "BC=ca:true"
在nexus02服务器上安装nexus，生成keystore.jks
# keytool \
-genkeypair \
-keystore keystore.jks \
-alias Test_nexus \
-keypass Nexus@123 \
-storepass Nexus@123 \
-keyalg RSA \
-keysize 2048 \
-validity 5000 \
-dname "CN=*.example.com,OU=Test,O=Test,L=ShenZhen,ST=GuangDong,C=CN" \
-ext "SAN=DNS:nexus.example.com,IP:192.168.1.106" \
-ext "BC=ca:true"
在装有docker的nexus客户端服务器上
# vim /etc/hosts
#################################
192.168.1.108 nexus.example.com
#################################
四、nginx配置
注意：需要使用 nginx+keepalived的形式实现高可用，此处只写了一个nginx的配置
关闭selinux
# sed -i 's/^SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
# setenforce 0
nignx监听vip设置
# echo "1" >/proc/sys/net/ipv4/ip_nonlocal_bind
# echo 'net.ipv4.ip_nonlocal_bind = 1' >>/etc/sysctl.conf
# sysctl -p
生成nginx自签名证书
# openssl req -x509 -nodes \
-days 3650 \
-newkey rsa:2048 \
-subj "/C=CN/ST=Guangdong Province/L=Shenzhen/O=Test/OU=Test/CN=nexus.example.com" \
-keyout /etc/nginx/ssl/nginx.key \
-out /etc/nginx/ssl/nginx.crt
# yum -y install nginx
# vim /etc/nginx/nginx.conf
#############################################
# For more information on configuration, see:
# * Official English Documentation: http://nginx.org/en/docs/
# * Official Russian Documentation: http://nginx.org/ru/docs/
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
# Load dynamic modules. See /usr/share/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;
events {
worker_connections 1024;
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
client_max_body_size 500M;
include /etc/nginx/mime.types;
default_type application/octet-stream;
# Load modular configuration files from the /etc/nginx/conf.d directory.
# See http://nginx.org/en/docs/ngx_core_module.html#include
# for more information.
include /etc/nginx/conf.d/*.conf;
upstream nexus {
server 192.168.1.105:10086 max_fails=1 fail_timeout=180s;
server 192.168.1.106:10086 max_fails=1 fail_timeout=180s;
}
server {
listen 192.168.1.108:10086 ssl;
ssl_certificate /etc/nginx/ssl/nginx.crt;
ssl_certificate_key /etc/nginx/ssl/nginx.key;
location / {
proxy_pass https://nexus;
#Proxy Settings
proxy_connect_timeout 3;
proxy_send_timeout 90;
proxy_read_timeout 90;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_redirect off;
}
}
}
#############################################
根据 include /etc/nginx/conf.d/*.conf 我们也可以将nexus的代理配置写在自配置文件中：
# vim /etc/nginx/conf.d/nexus.conf
###################################################
upstream nexus {
server 192.168.1.105:10086 max_fails=1 fail_timeout=180s;
server 192.168.1.106:10086 max_fails=1 fail_timeout=180s;
}
server{
listen192.168.1.108:10086 ssl;
ssl_certificate /etc/nginx/ssl/nginx.crt;
ssl_certificate_key /etc/nginx/ssl/nginx.key;
location / {
proxy_pass https://nexus;
#Proxy Settings
proxy_connect_timeout 3;
proxy_send_timeout 90;
proxy_read_timeout 90;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_redirect off;
}
}
###########################################################
五、、在docke服务器上获取nginx的公钥
# keytool -printcert -sslserver nexus.example.com:2020 -rfc > /etc/pki/ca-trust/source/anchors/nexus.crt
注：此处2020为创建的nexus的一个仓库端口
# cat /etc/pki/ca-trust/source/anchors/nexus.crt
# update-ca-trust
# systemctl restart docker
# docker login nexus.example.com:2020 -u test -p "Test@123"
# cat /root/.docker/config.json
从输出可知，docker节点获取的就是nginx的公钥。
六、参考
负载均衡Nexus工件存储库
https://qastack.cn/devops/366/load-balancing-nexus-artifact-repository
High Availability Clustering (Legacy)
https://help.sonatype.com/repomanager3/resiliency-and-high-availability/high-availability-clustering-%28legacy%29
http://tengine.taobao.org/document_cn/http_upstream_consistent_hash_cn.html
https://www.digitalocean.com/community/tutorials/how-to-configure-nginx-with-ssl-as-a-reverse-proxy-for-jenkins
https://www.digitalocean.com/community/tutorials/how-to-add-and-delete-users-on-an-ubuntu-14-04-vps
https://www.digitalocean.com/community/tutorials/openssl-essentials-working-with-ssl-certificates-private-keys-and-csrs
https://www.digitalocean.com/community/tutorials/how-to-configure-nginx-with-ssl-as-a-reverse-proxy-for-jenkins