【K8S 六】Harbor镜像仓库高可用方案（更新：2022-06-21）

目录

制定方案 

安装Harbor

解决harbor使用DNS同步问题

安装Keepalive

配置镜像同步策略

主节点配置一次性拉取复制策略（非“复合方案”，请略过）

“主、备节点”设置“Push-based”复制模式和“事件驱动”触发模式推送镜像到

应用Harbor

---

制定方案 

 主备方案：

复合方案：

角色	hostname	IP地址
主节点	host-master	192.168.10.190
备节点	host-slave	192.168.10.191
浮动IP	reg.harbor.4a	192.168.10.200

主备方案:

由于VIP的浮动，主备节点其实是互为主备；在部署harbor时，需要注意主备节点上的harbor.yml中hostname不要配置为浮动IP（reg.harbor.4a或192.168.10.200），应配置为各自IP或者hostname；

早先，将VIP的域名reg.harbor.4a配置到190和191上的harbor.yml中（hostname: reg.harbor.4a）导致一个问题：只有主节点可做为target被添加，用作镜像同步（也就是无法在主节点的仓库管理中创建备节点的target，即便添加了也无法连通）。

复合方案：

如果已经有一个Harbor仓库，而为了提高可用性，又部署了“主备集群”，为了方便使用旧库的镜像，在“主节点”设置“Pull-based”复制模式和“手动”触发模式一次性拉取镜像，同时“主备节点”均设置“Push-based”复制模式和“事件驱动”触发模式推送镜像。

因为在部署“主备节点”时，将hostname配置成了各自的IP或hostname，在使用时，我们却需要vip或其域名，故此需要在/etc/docker/daemon.json中配置"insecure-registries"：

 "insecure-registries": ["reg.harbor.4a"], 

或者在“主备节点”增加/etc/docker/certs.d/reg.harbor.4a：

cp -a /etc/docker/certs.d/{host-master,reg.harbor.4a}

提示：增加证书的方式，随着主节点切换，需要切换操作台，如果要避免此种场景，则“主备节点”使用同一套证书。

安装Harbor

HARBOR_ADDR_DNS="reg.harbor.4a"
HARBOR_ADDR_VIP="192.168.10.200"
HARBOR_HOST_NAME="master-node"
LOCAL_IPADDR="192.168.10.190"
PASSWD_4_HARBOR="your_passwd"
HARBOR_DATADIR="/data"

function install_harbor {
  echo "... begin  config harbor..."
  /usr/bin/cp -f  $script_dir/registry-offline/docker-compose-Linux-x86_64.1.25.4 /usr/local/bin/docker-compose
  if [ $? -eq 0 ]; then
    chmod 755 /usr/local/bin/docker-compose
    docker-compose --version 
  fi
  tar xvf $script_dir/registry-offline/harbor-offline-installer-v2.5.0.tgz -C /opt/ >/dev/null 2>&1
  if [ $? -eq 0 ]; then
    /usr/bin/cp -f $script_dir/k8s-offline/configfiles/harbor.yml /opt/harbor/
    sed -i "s/HARBOR_HOST_NAME/$HARABOR_HOST_NAME/g" /opt/harbor/harbor.yml
    sed -i "s/PASSWD_4_HARBOR/$PASSWD_4_HARBOR/g" /opt/harbor/harbor.yml
    sed -i "s/HARBOR_DATADIR/\\$HARBOR_DATADIR/g" /opt/harbor/harbor.yml
  fi
  echo "... end config harbor..."
  echo "... begin generate certification..."
  _gen_harbor_certs
  echo "... end generate certification..."
  echo "... begin  install harbor..."
  cd /opt/harbor/ && ./install.sh >/dev/null 2>&1
  echo "... end install harbor..."
  echo "... wait harbor starting..."
  while [ -n "$(docker-compose ps |grep starting)" ]; do
      sleep 5s
  done
  docker-compose ps
  echo "... harbir installed successfully..."

}

function _gen_harbor_certs {
  if [ ! -d /etc/ssl/certs/harbor-certs/ ]; then
    mkdir -p /etc/ssl/certs/harbor-certs/
    cd /etc/ssl/certs/harbor-certs/
    # 生成ca证书私钥ca.key
    openssl genrsa -out ca.key 4096
    # 生成ca证书ca.crt
    openssl req -x509 -new -nodes -sha512 -days 36500  -subj "/CN=$HARBOR_HOST_NAME"  -key ca.key  -out ca.crt

    # 生成服务器证书
    ## 生成服务器证书私钥
    openssl genrsa -out server.key 4096
    ## 生成服务器证书
    openssl req  -new -sha512  -subj "/CN=$HARBOR_HOST_NAME"  -key server.key  -out server.csr
    ## 生成 x509 v3 扩展名文件
    _create_v3
    ## 使用v3.ext生成服务器证书
    openssl x509 -req -sha512 -days 36500 -extfile v3.ext -CA ca.crt -CAkey ca.key -CAcreateserial -in server.csr -out server.crt
  fi
  if [ ! -d /etc/docker/certs.d/$HARBOR_HOST_NAME ]; then
    mkdir -p /etc/docker/certs.d/$HARBOR_HOST_NAME
    mkdir -p /etc/docker/certs.d/$HARBOR_ADDR_DNS
    /usr/bin/cp -f /etc/ssl/certs/harbor-certs/server.crt /etc/docker/certs.d/$HARBOR_HOST_NAME
    /usr/bin/cp -f /etc/ssl/certs/harbor-certs/server.crt /etc/docker/certs.d/$HARBOR_ADDR_DNS
  fi
}

function _create_v3 {
cat > /etc/ssl/certs/harbor-certs/v3.ext <<-EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth 
subjectAltName = @alt_names
[alt_names]
DNS.1=$HARBOR_HOST_NAME
DNS.1=$HARBOR_ADDR_DNS
IP.1=$HARBOR_ADDR_VIP
IP.2=$LOCAL_IPADDR
EOF
}

解决harbor使用DNS同步问题

修改/opt/harbor/docker-compose.yml ，在core和jobservice两个服务中，添加/etc/hosts卷映射，使用host的/etc/hosts，以便可以解析到reg.harbor.4a。如果不配置则会报错：

Jun  8 11:59:20 172-103-1-1-dynamic.midco.net core[10724]: 2022-06-08T03:59:20Z 
[ERROR] [/pkg/reg/adapter/native/adapter.go:126]: 
failed to ping registry http://192.168.10.190: Get "https://reg.harbor.4a/service/token?service=harbor-registry": 
dial tcp: lookup reg.harbor.4a on 127.0.0.11:53: read udp 127.0.0.1:45456->127.0.0.11:53: i/o timeout

所以如果Harbor配置的hostname是IP地址则跳过本步骤。

grep -B13 hosts /opt/harbor/docker-compose.yml  
    shm_size: '1gb'
  core:
    image: goharbor/harbor-core:v2.5.0
    container_name: harbor-core
    env_file:
      - ./common/config/core/env
    restart: always
    cap_drop:
      - ALL
    cap_add:
      - SETGID
      - SETUID
    volumes:
      - /etc/hosts:/etc/hosts
--
  jobservice:
    image: goharbor/harbor-jobservice:v2.5.0
    container_name: harbor-jobservice
    env_file:
      - ./common/config/jobservice/env
    restart: always
    cap_drop:
      - ALL
    cap_add:
      - CHOWN
      - SETGID
      - SETUID
    volumes:
      - /etc/hosts:/etc/hosts

安装Keepalive

HOST_NAME="host-master"
IFACE_NAME=bond0
HARBOR_ADDR_VIP="192.168.10.200"

function keepalived_install {
  yum -y install $script_dir/keepalived/*
  if [ $? -eq 0 ]; then
    mv /etc/keepalived/keepalived.conf{,.bak}
    /usr/bin/cp -f $script_dir/keepalived.conf /etc/keepalived/
    /usr/bin/cp -f $script_dir/check_harbor.sh /etc/keepalived/check_harbor.sh
    chmod a+x /etc/keepalived/check_harbor.sh
    sed -i "s/HOST_NAME/$HOST_NAME/g" /etc/keepalived/keepalived.conf
    sed -i "s/IFACE_NAME/$IFACE_NAME/g" /etc/keepalived/keepalived.conf
    sed -i "s/HARBOR_ADDR_VIP/$HARBOR_ADDR_VIP/g" /etc/keepalived/keepalived.conf
    read -p "install master node,[yes/no]:" confim
    if [ "$confim" = "yes" ];then
      sed -i "s/ROLE_NAME/MASTER/g" /etc/keepalived/keepalived.conf
      sed -i "s/PRIO_NUM/100/g" /etc/keepalived/keepalived.conf
    else
      sed -i "s/ROLE_NAME/BACKUP/g" /etc/keepalived/keepalived.conf
      sed -i "s/PRIO_NUM/50/g" /etc/keepalived/keepalived.conf
    fi
  fi

  systemctl restart keepalived
  systemctl status keepalived -l
  sleep 5s
  ip add show $IFACE_NAME
}

 keepalived.conf文件：

vrrp_script check_harbor {
        script "/etc/keepalived/check_harbor.sh"
        interval 10   # 间隔时间，单位为秒，默认1秒
        fall 2        # 脚本几次失败转换为失败
        rise 2        # 脚本连续监测成功后，把服务器从失败标记为成功的次数
        timeout 5
        init_fail
} 
global_defs {
        router_id HOST_NAME
        enable_script_security
        lvs_sync_daemon IFACE_NAME VI_1
}
vrrp_instance VI_1 {
        state  ROLE_NAME
        interface IFACE_NAME
        virtual_router_id 31    # 如果同一个局域网中有多套keepalive，那么要保证该id唯一   
        priority PRIO_NUM             
        advert_int 1            
        authentication {
                auth_type PASS        
                auth_pass k8s-testing
        }
        virtual_ipaddress {
                HARBOR_ADDR_VIP
        }
        track_script {
                check_harbor
        }
}

check_harbor.sh探测脚本：

#!/bin/bash
#count=$(docker-compose -f /opt/harbor/docker-compose.yml ps -a|grep healthy|wc -l)
# 不能频繁调用docker-compose 否则会有非常多的临时目录被创建：/tmp/_MEI*
count=$(docker ps |grep goharbor|grep healthy|wc -l)
status=$(ss -tlnp|grep -w 443|wc -l)
if [ $count -ne 9 -a  ];then
   exit 8
elif [ $status -lt 2 ];then
   exit 9
else
   exit 0
fi

配置镜像同步策略

主节点配置一次性拉取复制策略（非“复合方案”，请略过）

在“主节点”设置“Pull-based”复制模式和“手动”触发模式一次性拉取镜像。首先，admin登录在“主节点”Harbor，首先添加旧仓库：系统管理-仓库管理-新建目标，如下图所示：

然后，创建复制策略：系统管理-复制管理-创建策略，如图：

“主、备节点”设置“Push-based”复制模式和“事件驱动”触发模式推送镜像到

在主备节点，各自使用admin登录Harbor，系统管理-仓库管理-新建目标，将对方的hostname或IP添加目标，如下图所示：

注：提供者千万不要选择Harbor，否则push会失败，报如下错误：

 [ERROR] [/replication/operation/controller.go:108]: the execution 1 failed: failed to do the prepare work for pushing/uploading resources: http error: code 404, message ……

注：如果远程注册表使用自签名或不受信任的证书，必须取消选中该复选框。否则报错：
Jun 13 17:20:30 172-103-2-1-dynamic.midco.net core[3289]: 2022-06-13T09:20:30Z [ERROR] [/lib/http/error.go:54]: 
{"errors":[{"code":"UNKNOWN","message":"unknown: Get \"https://192.168.10.200/api/version\": x509: certificate signed by unknown authority"}]} 

接下来在“主备节点”各自创建复制策略： 

应用Harbor

我们使用http方式访问harbor时，需要在docker的/etc/docker/daemon.json中配置"insecure-registries": ["reg.harbor.4a"]；当我们使用https时，也要有相同的配置，否则需要将harbor的证书放到/etc/docker/certs.d/reg.harbor.4a下，维护起来较麻烦。