内网环境基于 k8s 的大型网站电商解决方案(一）

一、环境说明

1、所有系统为rockylinux8.6最小化安装，所有服务器均为内网，只有manager为双网卡（可访问互联网），提供内网yum源、DNS解析、时间同步等
2、 k8s搭建高可用集群版本为1.24.6(基于containerd部署) 3台控制节点，2台工作节点
3、搭建rancher平台管理k8s集群(注：rancher为centos7.9最小化安装）
4、 mysql版本为8.0.31 搭建MGR
5、 ceph版本为quincy版，通过cephadm搭建
6、镜像存放在 harbor 仓库，版本为2.6.0
8、电商项目使用 LNMP 架构
9、PHP 和 Nginx 共享同一个 pvc：基于 cephfs 划分 pv
10、使用 Prometheus 监控电商平台，在 Grafana 可视化展示监控数据
11、搭建 efk+logstash+kafka 日志收集平台
12、K8S升级，将k8s升级至1.25.2，备份etcd

规划如下：

序号	系统名	IP地址	配置	作用	备注
1	master1	192.168.8.81	8G /sda 60G	K8S控制节点
2	master2	192.168.8.82	8G /sda 60G	K8S控制节点
3	master3	192.168.8.83	8G /sda 60G	K8S控制节点
4	node1	192.168.8.84	8G /sda 60G	K8S工作节点
5	node2	192.168.8.85	8G /sda 60G	K8S工作节点
6	master	192.168.8.88	8G /sda 60G	K8S VIP
7	harbor1	192.168.8.91	8G /sda 60G	harbor 私有仓库
8	harbor2	192.168.8.92	8G /sda 60G	harbor 私有仓库
9	rancher	192.168.8.96	8G /sda 60G	rancher管理平台	centos7.9
10	mysqla	192.168.8.51	8G /sda 60G	mysql数据库
11	mysqlb	192.168.8.52	8G /sda 60G	mysql数据库
12	mysqlc	192.168.8.53	8G /sda 60G	mysql数据库
13	mysql	192.168.8.55	8G /sda 60G	mysql数据库 vip
14	cepha	192.168.8.61	8G /sda 60G，/sdb 20G，/sdc 20G	ceph集群
15	cephb	192.168.8.62	8G /sda 60G，/sdb 20G，/sdc 20G	ceph集群
16	cephc	192.168.8.63	8G /sda 60G，/sdb 20G，/sdc 20G	ceph集群
17	nfs	192.168.8.100	8G /sda 60G	nfs共享
18	manager	192.168.8.80	8G /sda 60G	yum源、dns、ntp等

二、基础环境搭建

1、安装系统rockylinux8.6最小化

网卡模式为仅主机

手动分区

所有服务器配置DNS为192.168.8.80,gateway 192.168.8.1

关闭selinux

sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config

2、安装基础软件包及chrony（manager)

设置防火墙

firewall-cmd --add-service=http --add-service=ntp --add-service=dns --permanent
firewall-cmd --reload

安装基础软件包

yum install vim net-tools bash-completion wget -y

安装chronyc

yum install chrony -y
sed -i 's/2.pool.ntp.org/ntp.aliyun.com/g' /etc/chrony.conf
echo 'allow 192.168.8.0/24' >> /etc/chrony.conf
systemctl enable --now chronyd
systemctl status chronyd
chronyc sources

3、配置manager服务器yum源

yum install httpd -y
systemctl enable  --now  httpd
mkdir /var/www/html/k8s
mkdir /var/www/html/ceph
mkdir /var/www/html/epel 
mkdir /var/www/html/docker
dnf install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
sed -i 's+download.docker.com+mirrors.aliyun.com/docker-ce+' /etc/yum.repos.d/docker-ce.repo

cat << EOF  > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg

EOF

cat  >  /etc/yum.repos.d/ceph.repo  << EOF
[ceph-norch]
name=ceph-norch
baseurl=https://mirrors.aliyun.com/ceph/rpm-quincy/el8/noarch/
enable=1
gpgcheck=0

[ceph-x86_64]
name=ceph-x86_64
baseurl=https://mirrors.aliyun.com/ceph/rpm-quincy/el8/x86_64/
enable=1
gpgcheck=0

[ceph-source]
name=ceph-source
baseurl=https://mirrors.aliyun.com/ceph/rpm-quincy/el8/SRPMS/
enable=1
gpgcheck=0

EOF

4、搭建yum源服务器（manager）

mount /dev/sr0 /mnt/
cp -r  /mnt/*  /var/www/html/
mv AppStream appstream
mv BaseOS  baseos

dnf install  -y kubelet-1.24.6 kubeadm-1.24.6 kubectl-1.24.6 --downloadonly --destdir /var/www/html/k8s/
dnf install  -y kubelet kubeadm kubectl --downloadonly --destdir /var/www/html/k8s/
dnf install  -y docker-ce --downloadonly --destdir /var/www/html/docker/
dnf install  -y cephadm   --downloadonly --destdir /var/www/html/ceph/
dnf install  -y ceph-common --downloadonly --destdir /var/www/html/ceph/
dnf install  -y perl  --downloadonly --destdir /var/www/html/epel/

yum install createrepo
createrepo /var/www/html/k8s
createrepo /var/www/html/docker
createrepo /var/www/html/ceph
createrepo /var/www/html/epel
systemctl restart httpd

http://192.168.8.80/appstream/

5、安装dnsmasq(manager)

yum install dnsmasq -y
echo 'listen-address=192.168.8.80' >> /etc/dnsmasq.conf
cat >> /etc/hosts << EOF
192.168.8.80 manager
192.168.8.81 master1
192.168.8.82 master2
192.168.8.83 master3
192.168.8.84 node1
192.168.8.85 node2
192.168.8.88 master
192.168.8.91 harbor1
192.168.8.92 harbor2
192.168.8.96 rancher
192.168.8.51 mysqla
192.168.8.52 mysqlb
192.168.8.53 mysqlc
192.168.8.55 mysql
192.168.8.61 cepha
192.168.8.62 cephb
192.168.8.63 cephc
192.168.8.100 nfs

EOF

systemctl enable --now dnsmasq

6、安装docker

7、配置内网服务器yum源及NTP配置

除80外所有服务器上执行

rm -rf /etc/yum.repos.d/*
cat > /etc/yum.repos.d/base.repo << EOF
[appstream]
name=appstream
baseurl=http://manager/appstream
enable=1
gpgcheck=0

[baseos]
name=baseos
baseurl=http://manager/baseos
enable=1
gpgcheck=0

[k8s]
name=k8s
baseurl=http://manager/k8s
enable=1
gpgcheck=0

[docker]
name=k8s
baseurl=http://manager/docker
enable=1
gpgcheck=0

[ceph]
name=ceph
baseurl=http://manager/ceph
enable=1
gpgcheck=0

[epel]
name=epel
baseurl=http://manager/epel
enable=1
gpgcheck=0

EOF

yum install -y wget bash-completion vim net-tools  chrony
sed -i 's/2.pool.ntp.org/manager/g' /etc/chrony.conf
systemctl enable --now chronyd
chronyc sources

所有服务器关机，打快照

三、搭建harbor私有仓库(harbor1、harbor2）

1、安装docker

yum install -y  docker-ce
systemctl start docker && systemctl enable docker

2、修改内核参数

modprobe br_netfilter
echo "modprobe br_netfilter" >> /etc/profile
cat > /etc/sysctl.d/docker.conf <


3、安装docker-compose
  上传docker-compose-linux-x86_64至/root
wget https://github.com/goharbor/harbor/releases/download/v2.6.0/harbor-offline-installer-v2.6.0.tgz
wget https://github.com/docker/compose/releases/download/v2.11.0/docker-compose-linux-x86_64
mv docker-compose-linux-x86_64 /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
4、生成ca证书
mkdir /data/ssl -p
cd /data/ssl/
openssl genrsa -out ca.key 2048
openssl req -new -x509 -days 365 -key ca.key -out ca.pem

Country Name (2 letter code) [XX]: CN
 State or Province Name (full name) []:xinjiang
 Locality Name (eg, city) [Default City]:urumqi      
 Organization Name (eg, company) [Default Company Ltd]:myhub
 Organizational Unit Name (eg, section) []:CA
 Common Name (eg, your name or your server's hostname) []:harbor1
 Email Address []:[email protected]
5、生成域名证书
openssl genrsa -out myhub.key  2048
openssl req -new -key myhub.key -out myhub.csr

Country Name (2 letter code) [XX]:CN
 State or Province Name (full name) []:xinjiang
 Locality Name (eg, city) [Default City]:urumqi
 Organization Name (eg, company) [Default Company Ltd]:myhub
 Organizational Unit Name (eg, section) []:CA
 Common Name (eg, your name or your server's hostname) []:myhub
 Email Address []:[email protected]
Please enter the following 'extra' attributes
 to be sent with your certificate request
 A challenge password []:
 An optional company name []:
openssl x509 -req -in myhub.csr -CA ca.pem -CAkey ca.key -CAcreateserial -out myhub.pem -days 365
openssl x509 -noout -text -in myhub.pem

6、安装harbor
mkdir /data/install -p
ll /data/ssl
 
cd 
mv harbor-offline-installer-v2.6.0.tgz /data/install/
cd /data/install/
tar -xvf harbor-offline-installer-v2.6.0.tgz
cd harbor
cp harbor.yml.tmpl harbor.yml
vim harbor.yml
hostname: harbor1  ( harbor2)
certificate: /data/ssl/myhub.pem
private_key: /data/ssl/myhub.key
harbor_admin_password: password
docker load -i harbor.v2.6.0.tar.gz
./install.sh

7、 停止harbor
cd /data/install/harbor
docker-compose stop
8、启动harbor
cd /data/install/harbor
docker-compose start
firewall-cmd --add-service=http --add-service=https --permanent;firewall-cmd --reload
http://192.168.8.91     http://192.168.8.92

9、配置镜像自动同步
新建项目myhub    http://192.168.8.91

 配置仓库


 新建复制规则


10、在192.168.8.80上测试
cat > /etc/docker/daemon.json  << EOF
{
"registry-mirrors": ["http://hub-mirror.c.163.com","https://0x3urqgf.mirror.aliyuncs.com"],
"insecure-registries": [ "192.168.8.91","harbor1" ]
}

EOF

systemctl daemon-reload
systemctl restart docker

docker login 192.168.8.91

 上传镜像至仓库
docker pull nginx
docker pull busybox
docker tag busybox:latest 192.168.8.91/myhub/busybox:latest
docker tag nginx:latest 192.168.8.91/myhub/nginx:latest
docker push 192.168.8.91/myhub/busybox:latest
docker push 192.168.8.91/myhub/nginx:latest

http://192.168.8.91     http://192.168.8.92登录验证

 
镜像已自动同步
四、安装k8s高可用集群（master1-3，node1-2)
1、修改内核参数（五台设备上执行）
modprobe br_netfilter
lsmod | grep br_netfilter

cat > /etc/sysctl.d/k8s.conf <

2、开启Ipvs 五台设备
lsmod|grep ip_vs
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh

lsmod|grep ip_vs

modprobe br_netfilter
echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
echo 1 > /proc/sys/net/ipv4/ip_forward

3、安装containerd（五台设备执行）
dnf install -y device-mapper-persistent-data lvm2 ipvsadm iproute-tc
dnf install -y containerd

containerd config default > /etc/containerd/config.toml
更改配置文件
sed -i 's#registry.k8s.io#192.168.8.91/myhub#g' /etc/containerd/config.toml

vim /etc/containerd/config.toml
增加如下内容
[plugins."io.containerd.grpc.v1.cri".registry]
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
    [plugins."io.containerd.grpc.v1.cri".registry.mirrors."192.168.8.91"]
      endpoint = ["https://192.168.8.91"]
  [plugins."io.containerd.grpc.v1.cri".registry.configs]
    [plugins."io.containerd.grpc.v1.cri".registry.configs."192.168.8.91".tls]
      insecure_skip_verify = true
    [plugins."io.containerd.grpc.v1.cri".registry.configs."192.168.8.91".auth]
      username = "admin"
      password = "password"
启动containerd
systemctl daemon-reload
systemctl enable --now containerd.service
systemctl status containerd.service
crictl config runtime-endpoint unix:///run/containerd/containerd.sock
crictl config image-endpoint unix:///run/containerd/containerd.sock
4、安装kubelet kubeadm kubectl
    在三台master设备上执行
firewall-cmd --permanent --add-port=6443/tcp
firewall-cmd --permanent --add-port=2379-2380/tcp
firewall-cmd --permanent --add-port=10250-10252/tcp
firewall-cmd --permanent --add-port=16443/tcp
firewall-cmd --permanent --add-port=80/tcp
firewall-cmd --reload
dnf install  -y kubelet-1.24.6 kubeadm-1.24.6 kubectl-1.24.6
systemctl enable kubelet
    在node1及node2上执行
firewall-cmd --permanent --add-port=6443/tcp
firewall-cmd --permanent --add-port=2379-2380/tcp
firewall-cmd --permanent --add-port=10250-10252/tcp
firewall-cmd --permanent --add-port=16443/tcp
firewall-cmd --permanent --add-port=80/tcp
firewall-cmd --reload
dnf install  -y kubelet-1.24.6 kubeadm-1.24.6
systemctl enable kubelet
 5、下载镜像并上传（manager)
kubeadm config images list
docker pull registry.aliyuncs.com/google_containers/kube-apiserver:v1.24.6
docker pull registry.aliyuncs.com/google_containers/kube-proxy:v1.24.6
docker pull registry.aliyuncs.com/google_containers/kube-scheduler:v1.24.6
docker pull registry.aliyuncs.com/google_containers/kube-controller-manager:v1.24.6
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.7
docker pull registry.aliyuncs.com/google_containers/etcd:3.5.3-0
docker pull registry.aliyuncs.com/google_containers/coredns:v1.8.6
docker tag registry.aliyuncs.com/google_containers/kube-apiserver:v1.24.6  \
192.168.8.91/myhub/kube-apiserver:v1.24.6
docker tag registry.aliyuncs.com/google_containers/kube-proxy:v1.24.6  \
192.168.8.91/myhub/kube-proxy:v1.24.6
docker tag registry.aliyuncs.com/google_containers/kube-scheduler:v1.24.6  \
192.168.8.91/myhub/kube-scheduler:v1.24.6
docker tag registry.aliyuncs.com/google_containers/kube-controller-manager:v1.24.6  \
192.168.8.91/myhub/kube-controller-manager:v1.24.6
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.7  \
192.168.8.91/myhub/pause:3.7
docker tag registry.aliyuncs.com/google_containers/etcd:3.5.3-0  \
192.168.8.91/myhub/etcd:3.5.3-0
docker tag registry.aliyuncs.com/google_containers/coredns:v1.8.6  \
192.168.8.91/myhub/coredns:v1.8.6

docker pull docker.io/calico/cni:v3.24.4
docker pull docker.io/calico/node:v3.24.4
docker pull docker.io/calico/kube-controllers:v3.24.4
docker tag docker.io/calico/cni:v3.24.4 192.168.8.91/myhub/cni:v3.24.4
docker tag docker.io/calico/node:v3.24.4 192.168.8.91/myhub/node:v3.24.4
docker tag docker.io/calico/kube-controllers:v3.24.4 192.168.8.91/myhub/kube-controllers:v3.24.4
docker push 192.168.8.91/myhub/cni:v3.24.4
docker push  192.168.8.91/myhub/node:v3.24.4 
docker push  192.168.8.91/myhub/kube-controllers:v3.24.4
wget -O /var/www/html/packages/calico.yaml https://docs.projectcalico.org/manifests/calico.yaml


6、安装keepalive+nginx
1)（三台master上执行）
yum install nginx keepalived nginx-mod-stream -y
cat << EOF > /etc/nginx/nginx.conf
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

# Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}


stream {

    log_format  main  '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';

    access_log  /var/log/nginx/k8s-access.log  main;

    upstream k8s-apiserver {
       server 192.168.8.81:6443 weight=5 max_fails=3 fail_timeout=30s;
       server 192.168.8.82:6443 weight=5 max_fails=3 fail_timeout=30s;
       server 192.168.8.83:6443 weight=5 max_fails=3 fail_timeout=30s;
    }

    server {
       listen 16443;
       proxy_pass k8s-apiserver;
    }
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    server {
        listen       80 default_server;
        server_name  _;

        location / {
        }
    }
}
EOF
2)keeplive配置（master1)
cat << EOF > /etc/keepalived/keepalived.conf
global_defs { 
   notification_email { 
     [email protected] 
     [email protected] 
     [email protected] 
   } 
   notification_email_from [email protected]  
   smtp_server 127.0.0.1 
   smtp_connect_timeout 30 
   router_id NGINX_MASTER
} 

vrrp_script check_nginx {
    script "/etc/keepalived/check_nginx.sh"
}

vrrp_instance VI_1 { 
    state MASTER 
    interface ens160  # 修改为实际网卡名
    virtual_router_id 80
    priority 100    
    advert_int 1
    authentication { 
        auth_type PASS      
        auth_pass password 
    }  
    virtual_ipaddress { 
        192.168.8.88/24
    } 
    track_script {
        check_nginx
    } 
}
EOF

cat << EOF > /etc/keepalived/check_nginx.sh
#!/bin/bash
counter=`ps -C nginx --no-header | wc -l`
if [ $counter -eq 0 ]; then
    service nginx start
    sleep 2
    counter=`ps -C nginx --no-header | wc -l`
    if [ $counter -eq 0 ]; then
        service  keepalived stop
    fi
fi
EOF

chmod +x /etc/keepalived/check_nginx.sh
firewall-cmd --add-rich-rule='rule protocol value="vrrp" accept' --permanent
firewall-cmd --reload
systemctl enable keepalived.service
systemctl enable nginx.service
systemctl start nginx.service
systemctl start keepalived.service
 2)keeplive配置（master2)
cat << EOF > /etc/keepalived/keepalived.conf
global_defs { 
   notification_email { 
     [email protected] 
     [email protected] 
     [email protected] 
   } 
   notification_email_from [email protected]  
   smtp_server 127.0.0.1 
   smtp_connect_timeout 30 
   router_id NGINX_MASTER
} 

vrrp_script check_nginx {
    script "/etc/keepalived/check_nginx.sh"
}

vrrp_instance VI_1 { 
    state MASTER 
    interface ens160  # 修改为实际网卡名
    virtual_router_id 80
    priority 80    
    advert_int 1
    authentication { 
        auth_type PASS      
        auth_pass password 
    }  
    virtual_ipaddress { 
        192.168.8.88/24
    } 
    track_script {
        check_nginx
    } 
}
EOF

cat << EOF > /etc/keepalived/check_nginx.sh
#!/bin/bash
counter=`ps -C nginx --no-header | wc -l`
if [ $counter -eq 0 ]; then
    service nginx start
    sleep 2
    counter=`ps -C nginx --no-header | wc -l`
    if [ $counter -eq 0 ]; then
        service  keepalived stop
    fi
fi
EOF

chmod +x /etc/keepalived/check_nginx.sh
firewall-cmd --add-rich-rule='rule protocol value="vrrp" accept' --permanent
firewall-cmd --reload
systemctl enable keepalived.service
systemctl enable nginx.service
systemctl start nginx.service
systemctl start keepalived.service
 3)keeplive配置（master3)
cat << EOF > /etc/keepalived/keepalived.conf
global_defs { 
   notification_email { 
     [email protected] 
     [email protected] 
     [email protected] 
   } 
   notification_email_from [email protected]  
   smtp_server 127.0.0.1 
   smtp_connect_timeout 30 
   router_id NGINX_MASTER
} 

vrrp_script check_nginx {
    script "/etc/keepalived/check_nginx.sh"
}

vrrp_instance VI_1 { 
    state MASTER 
    interface ens160  # 修改为实际网卡名
    virtual_router_id 80
    priority 50    
    advert_int 1
    authentication { 
        auth_type PASS      
        auth_pass password 
    }  
    virtual_ipaddress { 
        192.168.8.88/24
    } 
    track_script {
        check_nginx
    } 
}
EOF

cat << EOF > /etc/keepalived/check_nginx.sh
#!/bin/bash
counter=`ps -C nginx --no-header | wc -l`
if [ $counter -eq 0 ]; then
    service nginx start
    sleep 2
    counter=`ps -C nginx --no-header | wc -l`
    if [ $counter -eq 0 ]; then
        service  keepalived stop
    fi
fi
EOF

chmod +x /etc/keepalived/check_nginx.sh
firewall-cmd --add-rich-rule='rule protocol value="vrrp" accept' --permanent
firewall-cmd --reload
systemctl enable keepalived.service
systemctl enable nginx.service
systemctl start nginx.service
systemctl start keepalived.service
4）测试
master1
ip addr

systemctl stop keepalived
master2

 master1
systemctl start keepalived
ip addr
7、初始化k8s集群（master节点上执行）
配置文件
cat > kubeadm-config.yaml <

初始化集群
kubeadm init --config kubeadm-config.yaml --ignore-preflight-errors=SystemVerification | tee $HOME/k8s.txt
 执行命令
  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config
k8s命令自动补全
source /usr/share/bash-completion/bash_completion
source <(kubectl completion bash)		
echo "source <(kubectl completion bash)" >> ~/.bashrc
8、安装网络组件
wget http://192.168.8.80/packages/calico.yaml
sed -i 's#docker.io/calico#192.168.8.91/myhub#g' calico.yaml
kubectl apply -f calico.yaml


监控pod状态，待所有pod状态为running
kubectl get pods -n kube-system -w
kubectl get nodes

 9、将master2、master3加入集群
master2  master3
mkdir -p /etc/kubernetes/pki/etcd 
mkdir -p ~/.kube/
master1
scp /etc/kubernetes/pki/ca.crt  master2:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/ca.key  master2:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/sa.key  master2:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/sa.pub  master2:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/front-proxy-ca.crt  master2:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/front-proxy-ca.key  master2:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/etcd/ca.crt  master2:/etc/kubernetes/pki/etcd/
scp /etc/kubernetes/pki/etcd/ca.key  master2:/etc/kubernetes/pki/etcd/
scp /etc/kubernetes/pki/ca.crt  master3:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/ca.key  master3:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/sa.key  master3:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/sa.pub  master3:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/front-proxy-ca.crt  master3:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/front-proxy-ca.key  master3:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/etcd/ca.crt  master3:/etc/kubernetes/pki/etcd/
scp /etc/kubernetes/pki/etcd/ca.key  master3:/etc/kubernetes/pki/etcd/
kubeadm token create --print-join-command

 master2  master3
kubeadm join 192.168.8.88:16443 --token 8682o7.gl9oij7g1hbmymmc --discovery-token-ca-cert-hash \
sha256:eb8beb55fd9a315e1d2cb675a36d762641e2f2caabf12a4cb1b40d32a40d2311 \
--control-plane --ignore-preflight-errors=SystemVerification

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

 10、工作节点加入集群
两台node节点上执行
kubeadm join 192.168.8.88:16443 --token 8682o7.gl9oij7g1hbmymmc --discovery-token-ca-cert-hash \
sha256:eb8beb55fd9a315e1d2cb675a36d762641e2f2caabf12a4cb1b40d32a40d2311

firewall-cmd --permanent --zone=trusted --change-interface=tunl0
firewall-cmd --permanent --add-port=30000-32767/tcp
firewall-cmd --reload
 11、验证
在master1上查看
监控pod状态，待所有pod状态为running
kubectl get nodes
kubectl get componentstatuses
kubectl cluster-info
kubectl -n kube-system get pod

 12、测试
1)测试网络
kubectl run busybox --image 192.168.8.91/myhub/busybox --restart=Never --rm -it busybox -- sh

2) 测试pod
cat > mypod.yaml << EOF
apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - name: nginx
    image: 192.168.8.91/myhub/nginx
    ports:
    - containerPort: 80
EOF

kubectl apply -f mypod.yaml
kubectl get pods -o wide

curl 10.10.104.1

13、部署ingress
manager
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v1.1.1
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v1.1.1 \
192.168.8.91/myhub/kube-webhook-certgen:v1.1.1
docker push 192.168.8.91/myhub/kube-webhook-certgen:v1.1.1
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:v1.1.1
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:v1.1.1 \
192.168.8.91/myhub/nginx-ingress-controller:v1.1.1
docker push 192.168.8.91/myhub/nginx-ingress-controller:v1.1.1
master1:
cat > deploy.yaml << EOF
apiVersion: v1
kind: Namespace
metadata:
  name: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx

---
# Source: ingress-nginx/templates/controller-serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.0.10
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.1.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx
  namespace: ingress-nginx
automountServiceAccountToken: true
---
# Source: ingress-nginx/templates/controller-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.0.10
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.1.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller
  namespace: ingress-nginx
data:
  allow-snippet-annotations: 'true'
---
# Source: ingress-nginx/templates/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.0.10
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.1.0
    app.kubernetes.io/managed-by: Helm
  name: ingress-nginx
rules:
  - apiGroups:
      - ''
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
      - namespaces
    verbs:
      - list
      - watch
  - apiGroups:
      - ''
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ''
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - networking.k8s.io
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ''
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - networking.k8s.io
    resources:
      - ingresses/status
    verbs:
      - update
  - apiGroups:
      - networking.k8s.io
    resources:
      - ingressclasses
    verbs:
      - get
      - list
      - watch
---
# Source: ingress-nginx/templates/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.0.10
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.1.0
    app.kubernetes.io/managed-by: Helm
  name: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ingress-nginx
subjects:
  - kind: ServiceAccount
    name: ingress-nginx
    namespace: ingress-nginx
---
# Source: ingress-nginx/templates/controller-role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.0.10
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.1.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx
  namespace: ingress-nginx
rules:
  - apiGroups:
      - ''
    resources:
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ''
    resources:
      - configmaps
      - pods
      - secrets
      - endpoints
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ''
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - networking.k8s.io
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - networking.k8s.io
    resources:
      - ingresses/status
    verbs:
      - update
  - apiGroups:
      - networking.k8s.io
    resources:
      - ingressclasses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ''
    resources:
      - configmaps
    resourceNames:
      - ingress-controller-leader
    verbs:
      - get
      - update
  - apiGroups:
      - ''
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ''
    resources:
      - events
    verbs:
      - create
      - patch
---
# Source: ingress-nginx/templates/controller-rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.0.10
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.1.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx
  namespace: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: ingress-nginx
subjects:
  - kind: ServiceAccount
    name: ingress-nginx
    namespace: ingress-nginx
---
# Source: ingress-nginx/templates/controller-service-webhook.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.0.10
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.1.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller-admission
  namespace: ingress-nginx
spec:
  type: ClusterIP
  ports:
    - name: https-webhook
      port: 443
      targetPort: webhook
      appProtocol: https
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/component: controller
---
# Source: ingress-nginx/templates/controller-service.yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
  labels:
    helm.sh/chart: ingress-nginx-4.0.10
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.1.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  type: NodePort
  ipFamilyPolicy: SingleStack
  ipFamilies:
    - IPv4
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: http
      appProtocol: http
    - name: https
      port: 443
      protocol: TCP
      targetPort: https
      appProtocol: https
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/component: controller
---
# Source: ingress-nginx/templates/controller-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.0.10
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.1.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/instance: ingress-nginx
      app.kubernetes.io/component: controller
  revisionHistoryLimit: 10
  minReadySeconds: 0
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/component: controller
    spec:
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
        - name: controller
          image: 192.168.8.91/myhub/nginx-ingress-controller:v1.1.1
          imagePullPolicy: IfNotPresent
          lifecycle:
            preStop:
              exec:
                command:
                  - /wait-shutdown
          args:
            - /nginx-ingress-controller
            - --election-id=ingress-controller-leader
            - --controller-class=k8s.io/ingress-nginx
            - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
            - --validating-webhook=:8443
            - --validating-webhook-certificate=/usr/local/certificates/cert
            - --validating-webhook-key=/usr/local/certificates/key
          securityContext:
            capabilities:
              drop:
                - ALL
              add:
                - NET_BIND_SERVICE
            runAsUser: 101
            allowPrivilegeEscalation: true
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: LD_PRELOAD
              value: /usr/local/lib/libmimalloc.so
          livenessProbe:
            failureThreshold: 5
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          ports:
            - name: http
              containerPort: 80
              protocol: TCP
            - name: https
              containerPort: 443
              protocol: TCP
            - name: webhook
              containerPort: 8443
              protocol: TCP
          volumeMounts:
            - name: webhook-cert
              mountPath: /usr/local/certificates/
              readOnly: true
          resources:
            requests:
              cpu: 100m
              memory: 90Mi
      nodeSelector:
        kubernetes.io/os: linux
      serviceAccountName: ingress-nginx
      terminationGracePeriodSeconds: 300
      volumes:
        - name: webhook-cert
          secret:
            secretName: ingress-nginx-admission
---
# Source: ingress-nginx/templates/controller-ingressclass.yaml
# We don't support namespaced ingressClass yet
# So a ClusterRole and a ClusterRoleBinding is required
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.0.10
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.1.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: nginx
  namespace: ingress-nginx
spec:
  controller: k8s.io/ingress-nginx
---
# Source: ingress-nginx/templates/admission-webhooks/validating-webhook.yaml
# before changing this value, check the required kubernetes version
# https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#prerequisites
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.0.10
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.1.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
  name: ingress-nginx-admission
webhooks:
  - name: validate.nginx.ingress.kubernetes.io
    matchPolicy: Equivalent
    rules:
      - apiGroups:
          - networking.k8s.io
        apiVersions:
          - v1
        operations:
          - CREATE
          - UPDATE
        resources:
          - ingresses
    failurePolicy: Fail
    sideEffects: None
    admissionReviewVersions:
      - v1
    clientConfig:
      service:
        namespace: ingress-nginx
        name: ingress-nginx-controller-admission
        path: /networking/v1/ingresses
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ingress-nginx-admission
  namespace: ingress-nginx
  annotations:
    helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-4.0.10
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.1.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ingress-nginx-admission
  annotations:
    helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-4.0.10
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.1.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
rules:
  - apiGroups:
      - admissionregistration.k8s.io
    resources:
      - validatingwebhookconfigurations
    verbs:
      - get
      - update
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: ingress-nginx-admission
  annotations:
    helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-4.0.10
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.1.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ingress-nginx-admission
subjects:
  - kind: ServiceAccount
    name: ingress-nginx-admission
    namespace: ingress-nginx
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: ingress-nginx-admission
  namespace: ingress-nginx
  annotations:
    helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-4.0.10
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.1.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
rules:
  - apiGroups:
      - ''
    resources:
      - secrets
    verbs:
      - get
      - create
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ingress-nginx-admission
  namespace: ingress-nginx
  annotations:
    helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-4.0.10
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.1.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: ingress-nginx-admission
subjects:
  - kind: ServiceAccount
    name: ingress-nginx-admission
    namespace: ingress-nginx
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/job-createSecret.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: ingress-nginx-admission-create
  namespace: ingress-nginx
  annotations:
    helm.sh/hook: pre-install,pre-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-4.0.10
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.1.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
spec:
  template:
    metadata:
      name: ingress-nginx-admission-create
      labels:
        helm.sh/chart: ingress-nginx-4.0.10
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/version: 1.1.0
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/component: admission-webhook
    spec:
      containers:
        - name: create
          image: 192.168.8.91/myhub/kube-webhook-certgen:v1.1.1
          imagePullPolicy: IfNotPresent
          args:
            - create
            - --host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc
            - --namespace=$(POD_NAMESPACE)
            - --secret-name=ingress-nginx-admission
          env:
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          securityContext:
            allowPrivilegeEscalation: false
      restartPolicy: OnFailure
      serviceAccountName: ingress-nginx-admission
      nodeSelector:
        kubernetes.io/os: linux
      securityContext:
        runAsNonRoot: true
        runAsUser: 2000
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/job-patchWebhook.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: ingress-nginx-admission-patch
  namespace: ingress-nginx
  annotations:
    helm.sh/hook: post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-4.0.10
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.1.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
spec:
  template:
    metadata:
      name: ingress-nginx-admission-patch
      labels:
        helm.sh/chart: ingress-nginx-4.0.10
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/version: 1.1.0
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/component: admission-webhook
    spec:
      containers:
        - name: patch
          image: 192.168.8.91/myhub/kube-webhook-certgen:v1.1.1
          imagePullPolicy: IfNotPresent
          args:
            - patch
            - --webhook-name=ingress-nginx-admission
            - --namespace=$(POD_NAMESPACE)
            - --patch-mutating=false
            - --secret-name=ingress-nginx-admission
            - --patch-failure-policy=Fail
          env:
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          securityContext:
            allowPrivilegeEscalation: false
      restartPolicy: OnFailure
      serviceAccountName: ingress-nginx-admission
      nodeSelector:
        kubernetes.io/os: linux
      securityContext:
        runAsNonRoot: true
        runAsUser: 2000

EOF

kubectl apply -f deploy.yaml
kubectl get pods -n ingress-nginx

2)根据域名的访问
创建deployment
kubectl create deployment nginx1 --image=192.168.8.91/myhub/nginx
kubectl create deployment nginx2 --image=192.168.8.91/myhub/nginx

创建service
kubectl expose deployment nginx1 --port=8080 --target-port=80
kubectl expose deployment nginx2 --port=8081 --target-port=80

更改nginx文件内容
kubectl exec -it nginx1-56797bcd6c-wqz6l -- /bin/bash
echo 'mynginx1' > /usr/share/nginx/html/index.html
exit

kubectl exec -it nginx2-85f7fcc878-kj7m9 -- /bin/bash
echo 'mynginx2' > /usr/share/nginx/html/index.html
exit

cat > myingress.yaml << EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx
  rules:
  - host: nginx1.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx1
            port:
              number: 8080
  - host: nginx2.example.com
    http:
      paths:      
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx2
            port:
              number: 8081
EOF

kubectl apply -f myingress.yaml
kubectl get ingress

manager
cat >> /etc/hosts  << EOF
192.168.8.85  nginx1.example.com  nginx2.example.com
EOF

curl nginx1.example.com
curl nginx2.example.com

 五、安装ceph
1、安装cephadm（cepha)
systemctl disable firewalld.service
systemctl stop firewalld
dnf -y install docker-ce
systemctl daemon-reload
systemctl start docker && systemctl enable docker.service

cat > /etc/docker/daemon.json  << EOF
{
"insecure-registries": [ "192.168.8.91","harbor1" ]
}

EOF

systemctl daemon-reload
systemctl restart docker

docker login 192.168.8.91

dnf -y install cephadm
dnf -y install ceph-common

2、下载镜像（manager)
docker pull quay.io/ceph/ceph:v17
docker pull quay.io/ceph/haproxy:2.3
docker pull quay.io/prometheus/prometheus:v2.33.4
docker pull quay.io/prometheus/node-exporter:v1.3.1
docker pull quay.io/prometheus/alertmanager:v0.23.0
docker pull quay.io/ceph/ceph-grafana:8.3.5
docker pull quay.io/ceph/keepalived:2.1.5
docker pull docker.io/maxwo/snmp-notifier:v1.2.1
docker pull docker.io/grafana/promtail:2.4.0
docker pull docker.io/grafana/loki:2.4.0
docker tag quay.io/ceph/ceph:v17 192.168.8.91/myhub/ceph/ceph:v17
docker push 192.168.8.91/myhub/ceph/ceph:v17

docker save -o /var/www/html/packages/ceph.tar quay.io/ceph/ceph:v17  \
quay.io/ceph/haproxy:2.3 quay.io/ceph/keepalived:2.1.5 \
quay.io/prometheus/node-exporter:v1.3.1  quay.io/prometheus/prometheus:v2.33.4 \
quay.io/ceph/ceph-grafana:8.3.5 docker.io/maxwo/snmp-notifier:v1.2.1 \
docker.io/grafana/promtail:2.4.0  docker.io/grafana/loki:2.4.0  \
quay.io/prometheus/alertmanager:v0.23.0

chmod 644 /var/www/html/packages/ceph.tar
3、初始化集群
sed -i 's#quay.io/ceph/ceph:v17#192.168.8.91/myhub/ceph/ceph:v17#g' /usr/sbin/cephadm
wget http://manager/packages/ceph.tar
docker load -i ceph.tar
cephadm bootstrap --mon-ip 192.168.8.61 --cluster-network 192.168.8.0/24

 集群初如化成功
sudo /usr/sbin/cephadm shell --fsid 5307d468-5cf9-11ed-9f4f-000c298a5a85 \
-c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring

exit
ceph -s
ceph version


安装 ceph-common(cephb、cephc)
systemctl disable firewalld.service
systemctl stop firewalld
dnf -y install docker-ce
systemctl daemon-reload
systemctl start docker && systemctl enable docker.service

cat > /etc/docker/daemon.json  << EOF
{
"insecure-registries": [ "192.168.8.91","harbor1" ]
}

EOF

systemctl daemon-reload
systemctl restart docker

docker login 192.168.8.91

wget http://manager/packages/ceph.tar
docker load -i ceph.tar

dnf -y install ceph-common
cepha
ceph -s
docker ps
key 文件：
 cat /etc/ceph/ceph.client.admin.keyring 
 [client.admin]
     key = AQDH2RljgCNXBRAAKaOpbXMB6hD9G3uVC9KRYQ==
     caps mds = "allow *"
     caps mgr = "allow *"
     caps mon = "allow *"
     caps osd = "allow *"

  加入cephb cephc
ssh-copy-id -f -i /etc/ceph/ceph.pub cephb
ssh-copy-id -f -i /etc/ceph/ceph.pub cephc

ceph orch host add cephb 192.168.8.62
ceph orch host add cephc 192.168.8.63

ceph orch host ls
 
填加标签 
ceph orch host label add cephb _admin
ceph orch host label add cephc _admin
ceph -s
mon  mgr填加节点后会自动部署

 填加OSD
ceph orch daemon add osd cepha:/dev/sdb,/dev/sdc
ceph orch daemon add osd cephb:/dev/sdb,/dev/sdc
ceph orch daemon add osd cephc:/dev/sdb,/dev/sdc
ceph -s

web查看https://192.168.8.61:8443

4、ceph对接k8s
 创建cephfs
ceph fs volume create k8sfs --placement=3

ceph fs ls
ceph -s
ceph fs status
ceph fs volume ls
 k8s工作节点安装ceph-common(node1  node2)
yum install  ceph-common -y
scp 192.168.8.61:/etc/ceph/* /etc/ceph/
cat /etc/ceph/ceph.client.admin.keyring |grep key|awk -F" " '{print $3}' > /etc/ceph/admin.secret
mkdir /data
mount -t ceph 192.168.8.61:6789:/ /data -o name=admin,secretfile=/etc/ceph/admin.secret
df -h | grep /data

cat >> /etc/fstab  << EOF
192.168.8.61:6789:/ /data  ceph  name=admin,secretfile=/etc/ceph/admin.secret,_netdev   0 0
EOF

mount -a

node1:
cd /data/
mkdir k8sdata
chmod 0777 k8sdata/

cat /etc/ceph/admin.secret

master1:
echo "AQAcOWdjqeg6GBAABDGsSg8xRLw1qyiVR7fU0A==" | base64

cat > cephfs-secret.yaml << EOF
apiVersion: v1
kind: Secret
metadata:
  name: cephfs-secret
data:
  key: QVFBY09XZGpxZWc2R0JBQUJER3NTZzh4Ukx3MXF5aVZSN2ZVMEE9PQo=

EOF

kubectl apply -f cephfs-secret.yaml

cat > cephfs-pv.yaml << EOF
apiVersion: v1
kind: PersistentVolume
metadata:
  name: cephfs-pv
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteMany
  cephfs:
    monitors:
      - 192.168.8.61:6789
    path: /k8sdata
    user: admin
    readOnly: false
    secretRef:
      name: cephfs-secret
  persistentVolumeReclaimPolicy: Recycle

EOF


cat > cephfs-pvc.yaml << EOF
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: cephfs-pvc
spec:
  accessModes:
    - ReadWriteMany
  volumeName: cephfs-pv
  resources:
    requests:
      storage: 1Gi

EOF

kubectl apply -f cephfs-pv.yaml
kubectl get pv
kubectl apply -f cephfs-pvc.yaml
kubectl get pvc

创建测试pod1
cat  > cephfs-pod-1.yaml << EOF
apiVersion: v1
kind: Pod
metadata:
  name: cephfs-pod-1
spec:
  containers:
  - image: 192.168.8.91/myhub/nginx
    name: nginx
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - name: test-v1
      mountPath: /usr/share/nginx/html
  volumes:
  - name: test-v1
    persistentVolumeClaim:
      claimName: cephfs-pvc

EOF

kubectl apply -f cephfs-pod-1.yaml
kubectl get pods -o wide

kubectl exec -it cephfs-pod-1 -- /bin/bash
echo 'mycephfs-nginx' > /usr/share/nginx/html/index.html
exit
curl 10.10.104.4


创建测试pod2
cat  > cephfs-pod-2.yaml << EOF
apiVersion: v1
kind: Pod
metadata:
  name: cephfs-pod-2
spec:
  containers:
  - image: 192.168.8.91/myhub/nginx
    name: nginx
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - name: test-v2
      mountPath: /usr/share/nginx/html
  volumes:
  - name: test-v2
    persistentVolumeClaim:
      claimName: cephfs-pvc

EOF

kubectl apply -f cephfs-pod-2.yaml
kubectl get pods -o wide



curl 10.10.166.130

 可查看两个pod已经共享k8sdata目录
kubectl delete pods cephfs-pod-1
kubectl delete pods cephfs-pod-2
六、安装mysql(MGR）
1、三台设置防火墙（mysqla、mysqlb、mysqlc）
firewall-cmd --permanent --add-port=3306/tcp
firewall-cmd --permanent --add-port=33060/tcp 
firewall-cmd --permanent --add-port=33061/tcp 
firewall-cmd --permanent --add-port=13306/tcp 
firewall-cmd --reload
2、安装mysql
下载软件
mysql-commercial-common-8.0.31-1.1.el8.x86_64.rpm
mysql-commercial-icu-data-files-8.0.31-1.1.el8.x86_64.rpm
mysql-commercial-libs-8.0.31-1.1.el8.x86_64.rpm
mysql-commercial-server-8.0.31-1.1.el8.x86_64.rpm
mysql-commercial-client-plugins-8.0.31-1.1.el8.x86_64.rpm
mysql-commercial-client-8.0.31-1.1.el8.x86_64.rpm
mysql-commercial-backup-8.0.31-1.1.el8.x86_64.rpm
安装需求包及软件包
yum install -y  perl

rpm -ivh mysql-commercial-common-8.0.31-1.1.el8.x86_64.rpm
rpm -ivh mysql-commercial-icu-data-files-8.0.31-1.1.el8.x86_64.rpm
rpm -ivh mysql-commercial-client-plugins-8.0.31-1.1.el8.x86_64.rpm
rpm -ivh mysql-commercial-libs-8.0.31-1.1.el8.x86_64.rpm
rpm -ivh mysql-commercial-client-8.0.31-1.1.el8.x86_64.rpm
rpm -ivh mysql-commercial-server-8.0.31-1.1.el8.x86_64.rpm
rpm -ivh mysql-commercial-backup-8.0.31-1.1.el8.x86_64.rpm
3、启动mysqld
systemctl start mysqld
systemctl enable mysqld
grep password /var/log/mysqld.log

mysql -uroot -p'V&qksQCh>7!B'
alter user root@localhost identified by 'Admin@123';
exit
 
4、安装插件及配置用户
mysqla   mysqlb   mysqlc

mysql -uroot -pAdmin@123
install plugin group_replication soname 'group_replication.so';
show plugins;
创建用户  mysqla   mysqlb   mysqlc
set sql_log_bin=0;
create user repl@'192.168.8.%' identified by 'Admin@123';
grant replication slave on *.* to repl@'192.168.8.%';
set sql_log_bin=1;
exit
5、配置mgr
mysqla:
cat >>  /etc/my.cnf << EOF
server_id=51
gtid_mode=on
binlog_checksum=NONE
enforce-gtid-consistency=on
disabled_storage_engines='MyISAM,BLACKHOLE,FEDERATED,ARCHIVE,MEMORY'
log_bin=binlog
log_slave_updates=ON
binlog_format=ROW
master_info_repository=TABLE
relay_log_info_repository=TABLE 
transaction_write_set_extraction=XXHASH64
loose_group_replication_recovery_use_ssl=on
group_replication_group_name='5de7369a-031f-4006-adee-ef023fc3b591'
group_replication_start_on_boot=off
group_replication_local_address='192.168.8.51:33061'
group_replication_group_seeds='192.168.8.51:33061,192.168.8.52:33061,192.168.8.53:33061'
group_replication_bootstrap_group=off

EOF

systemctl restart mysqld
group_name可由uuidgen生成
  
mysql -uroot -pAdmin@123
change master to
master_user='repl',
master_password='Admin@123'
for channel 'group_replication_recovery';

set global group_replication_bootstrap_group=on;
start group_replication;
set global group_replication_bootstrap_group=off;

select * from performance_schema.replication_group_members;
exit

增加mysqlb

cat >> /etc/my.cnf << EOF
server_id=52
gtid_mode=on
binlog_checksum=NONE
enforce-gtid-consistency=on
disabled_storage_engines='MyISAM,BLACKHOLE,FEDERATED,ARCHIVE,MEMORY'
log_bin=binlog
log_slave_updates=ON
binlog_format=ROW
master_info_repository=TABLE
relay_log_info_repository=TABLE 
transaction_write_set_extraction=XXHASH64
loose_group_replication_recovery_use_ssl=on
loose_group_replication_group_name='5de7369a-031f-4006-adee-ef023fc3b591'
loose_group_replication_start_on_boot=off
loose_group_replication_local_address='192.168.8.52:33061'
loose_group_replication_group_seeds='192.168.8.51:33061,192.168.8.52:33061,192.168.8.53:33061'
loose_group_replication_bootstrap_group=off

EOF

systemctl restart mysqld

mysql -uroot -pAdmin@123
change master to
master_user='repl',
master_password='Admin@123'
for channel 'group_replication_recovery';

start group_replication;
exit
 增加mysqlc
cat >> /etc/my.cnf << EOF
server_id=53
gtid_mode=on
binlog_checksum=NONE
enforce-gtid-consistency=on
disabled_storage_engines='MyISAM,BLACKHOLE,FEDERATED,ARCHIVE,MEMORY'
log_bin=binlog
log_slave_updates=ON
binlog_format=ROW
master_info_repository=TABLE
relay_log_info_repository=TABLE 
transaction_write_set_extraction=XXHASH64
loose_group_replication_recovery_use_ssl=on
loose_group_replication_group_name='5de7369a-031f-4006-adee-ef023fc3b591'
loose_group_replication_start_on_boot=off
loose_group_replication_local_address='192.168.8.53:33061'
loose_group_replication_group_seeds='192.168.8.51:33061,192.168.8.52:33061,192.168.8.53:33061'
loose_group_replication_bootstrap_group=off

EOF

systemctl restart mysqld

mysql -uroot -pAdmin@123
change master to
master_user='repl',
master_password='Admin@123'
for channel 'group_replication_recovery';

start group_replication;
exit
6、修改配置文件
mysqla  mysqlb   mysqlc:
改  group_replication_start_on_boot=off  为    group_replication_start_on_boot=on
去掉loose_ 
sed -i 's#group_replication_start_on_boot=off#group_replication_start_on_boot=on#g' \
/etc/my.cnf
sed -i 's#loose_group#group#g' /etc/my.cnf
systemctl restart mysqld
7、验证
mysqla
mysql -uroot -pAdmin@123
select * from performance_schema.replication_group_members;
show variables like 'group_replication%';
show variables like '%read_only%';


8、改为多主模式
mysqla:(主节点）
mysql -uroot -pAdmin@123
stop group_replication;

set global group_replication_single_primary_mode=off; 
set global group_replication_enforce_update_everywhere_checks=on;
set global group_replication_bootstrap_group=on;
start group_replication;
set global group_replication_bootstrap_group=off;
msyqlb  mysqlc(从节点）:
mysql -uroot -pAdmin@123
stop group_replication;

set global group_replication_single_primary_mode=off; 
set global group_replication_enforce_update_everywhere_checks=on;
start group_replication;
验证
select * from performance_schema.replication_group_members;
show variables like '%read_only%';
create database testdb;
create user myuser@'192.168.8.%' identified by 'Admin@123';
grant all  on testdb.* to myuser@'192.168.8.%';
use testdb
DROP TABLE IF EXISTS `tb1`;
 
CREATE TABLE `tb1` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `name` varchar(20) DEFAULT NULL,
  `dep_id` int(11) DEFAULT NULL,
  `age` int(11) DEFAULT NULL,
  `salary` decimal(10,2) DEFAULT NULL,
  `cus_id` int(11) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=109 DEFAULT CHARSET=utf8;
 
INSERT INTO `tb1` VALUES ('1', '鲁班', '1', '10', '1000.00', '1');
INSERT INTO `tb1` VALUES ('2', '后裔', '1', '20', '2000.00', '1');

select * from testdb.tb1;


9、安装 keepalived+nginx 实现 MGR高可用
 1)（三台mysql上执行）
yum install nginx keepalived nginx-mod-stream -y
cat << EOF > /etc/nginx/nginx.conf
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

# Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}


stream {

    log_format  main  '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';

    access_log  /var/log/nginx/k8s-access.log  main;

    upstream mysql-server {
       server 192.168.8.51:3306 weight=5 max_fails=3 fail_timeout=30s;
       server 192.168.8.52:3306 weight=5 max_fails=3 fail_timeout=30s;
       server 192.168.8.53:3306 weight=5 max_fails=3 fail_timeout=30s;
    }

    server {
       listen 13306;
       proxy_pass mysql-server;
    }
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    server {
        listen       80 default_server;
        server_name  _;

        location / {
        }
    }
}
EOF
2)keeplive配置（mysqla)
cat << EOF > /etc/keepalived/keepalived.conf
global_defs { 
   notification_email { 
     [email protected] 
     [email protected] 
     [email protected] 
   } 
   notification_email_from [email protected]  
   smtp_server 127.0.0.1 
   smtp_connect_timeout 30 
   router_id NGINX_MASTER
} 

vrrp_script check_nginx {
    script "/etc/keepalived/check_nginx.sh"
}

vrrp_instance VI_1 { 
    state MASTER 
    interface ens160  # 修改为实际网卡名
    virtual_router_id 80
    priority 100    
    advert_int 1
    authentication { 
        auth_type PASS      
        auth_pass password 
    }  
    virtual_ipaddress { 
        192.168.8.55/24
    } 
    track_script {
        check_nginx
    } 
}
EOF

cat << EOF > /etc/keepalived/check_nginx.sh
#!/bin/bash
counter=`ps -C nginx --no-header | wc -l`
if [ $counter -eq 0 ]; then
    service nginx start
    sleep 2
    counter=`ps -C nginx --no-header | wc -l`
    if [ $counter -eq 0 ]; then
        service  keepalived stop
    fi
fi
EOF

chmod +x /etc/keepalived/check_nginx.sh
firewall-cmd --add-rich-rule='rule protocol value="vrrp" accept' --permanent
firewall-cmd --reload
systemctl enable keepalived.service
systemctl enable nginx.service
systemctl start nginx.service
systemctl start keepalived.service
 3)keeplive配置（mysqlb)
cat << EOF > /etc/keepalived/keepalived.conf
global_defs { 
   notification_email { 
     [email protected] 
     [email protected] 
     [email protected] 
   } 
   notification_email_from [email protected]  
   smtp_server 127.0.0.1 
   smtp_connect_timeout 30 
   router_id NGINX_MASTER
} 

vrrp_script check_nginx {
    script "/etc/keepalived/check_nginx.sh"
}

vrrp_instance VI_1 { 
    state MASTER 
    interface ens160  # 修改为实际网卡名
    virtual_router_id 80
    priority 80    
    advert_int 1
    authentication { 
        auth_type PASS      
        auth_pass password 
    }  
    virtual_ipaddress { 
        192.168.8.55/24
    } 
    track_script {
        check_nginx
    } 
}
EOF

cat << EOF > /etc/keepalived/check_nginx.sh
#!/bin/bash
counter=`ps -C nginx --no-header | wc -l`
if [ $counter -eq 0 ]; then
    service nginx start
    sleep 2
    counter=`ps -C nginx --no-header | wc -l`
    if [ $counter -eq 0 ]; then
        service  keepalived stop
    fi
fi
EOF

chmod +x /etc/keepalived/check_nginx.sh
firewall-cmd --add-rich-rule='rule protocol value="vrrp" accept' --permanent
firewall-cmd --reload
systemctl enable keepalived.service
systemctl enable nginx.service
systemctl start nginx.service
systemctl start keepalived.service
4)keeplive配置（mysqlc)
cat << EOF > /etc/keepalived/keepalived.conf
global_defs { 
   notification_email { 
     [email protected] 
     [email protected] 
     [email protected] 
   } 
   notification_email_from [email protected]  
   smtp_server 127.0.0.1 
   smtp_connect_timeout 30 
   router_id NGINX_MASTER
} 

vrrp_script check_nginx {
    script "/etc/keepalived/check_nginx.sh"
}

vrrp_instance VI_1 { 
    state MASTER 
    interface ens160  # 修改为实际网卡名
    virtual_router_id 80
    priority 50    
    advert_int 1
    authentication { 
        auth_type PASS      
        auth_pass password 
    }  
    virtual_ipaddress { 
        192.168.8.55/24
    } 
    track_script {
        check_nginx
    } 
}
EOF

cat << EOF > /etc/keepalived/check_nginx.sh
#!/bin/bash
counter=`ps -C nginx --no-header | wc -l`
if [ $counter -eq 0 ]; then
    service nginx start
    sleep 2
    counter=`ps -C nginx --no-header | wc -l`
    if [ $counter -eq 0 ]; then
        service  keepalived stop
    fi
fi
EOF

chmod +x /etc/keepalived/check_nginx.sh
firewall-cmd --add-rich-rule='rule protocol value="vrrp" accept' --permanent
firewall-cmd --reload
systemctl enable keepalived.service
systemctl enable nginx.service
systemctl start nginx.service
systemctl start keepalived.service
10、测试mysql读写（manager)
yum install mysql -y
mysql -umyuser -pAdmin@123 -h192.168.8.55 -P13306
select * from testdb.tb1;

腾讯云技术深度探索：构建高效云原生微服务架构我的运维人生云原生架构腾讯云运维开发技术共享
腾讯云技术深度探索：构建高效云原生微服务架构在当今快速发展的技术环境中，云原生技术已成为企业数字化转型的关键驱动力。腾讯云作为行业领先的云服务提供商，不断推出创新的产品和技术，助力企业构建高效、可扩展的云原生微服务架构。本文将深入探讨腾讯云在微服务领域的最新进展，并通过一个实际案例展示如何在腾讯云平台上构建云原生应用。腾讯云微服务架构概览腾讯云微服务架构基于云原生理念，旨在帮助企业快速实现应用的容
Day17笔记-高阶函数 ~在杰难逃~ Python 笔记 python 开发语言 pycharm 数据分析
高阶函数【重点掌握】函数的本质：函数是一个变量，函数名是一个变量名，一个函数可以作为另一个函数的参数或返回值使用如果A函数作为B函数的参数，B函数调用完成之后，会得到一个结果，则B函数被称为高阶函数常用的高阶函数：map(),reduce(),filter(),sorted()1.map()map(func,iterable)，返回值是一个iterator【容器，迭代器】func:函数iterab
docker igotyback eureka 云原生
Docker容器的文件系统是隔离的，但是可以通过挂载卷（Volumes）或绑定挂载（BindMounts）将宿主机的文件系统目录映射到容器内部。要查看Docker容器的映射路径，可以使用以下方法：查看容器配置：使用dockerinspect命令可以查看容器的详细配置信息，包括挂载的卷。例如：bashdockerinspect在输出的JSON格式中，查找"Mounts"部分，这里会列出所有的挂载信息
ArrayList 源码解析程序猿进阶 Java基础 ArrayList List java 面试性能优化架构设计 idea
ArrayList是Java集合框架中的一个动态数组实现，提供了可变大小的数组功能。它继承自AbstractList并实现了List接口，是顺序容器，即元素存放的数据与放进去的顺序相同，允许放入null元素，底层通过数组实现。除该类未实现同步外，其余跟Vector大致相同。每个ArrayList都有一个容量capacity，表示底层数组的实际大小，容器内存储元素的个数不能多于当前容量。当向容器中添
react-intl——react国际化使用方案苹果酱0567 面试题汇总与解析 java 开发语言中间件 spring boot 后端
国际化介绍i18n：internationalization国家化简称，首字母+首尾字母间隔的字母个数+尾字母，类似的还有k8s(Kubernetes)React-intl是React中最受欢迎的库。使用步骤安装#usenpmnpminstallreact-intl-D#useyarn项目入口文件配置//index.tsximportReactfrom"react";importReactDOMf
《 C++ 修炼全景指南：四》揭秘 C++ List 容器背后的实现原理，带你构建自己的双向链表 Lenyiin 技术指南 C++修炼全景指南 c++list 链表 stl
本篇博客，我们将详细讲解如何从头实现一个功能齐全且强大的C++List容器，并深入到各个细节。这篇博客将包括每一步的代码实现、解释以及扩展功能的探讨，目标是让初学者也能轻松理解。一、简介1.1、背景介绍在C++中，std::list是一个基于双向链表的容器，允许高效的插入和删除操作，适用于频繁插入和删除操作的场景。与动态数组不同，list允许常数时间内的插入和删除操作，支持双向遍历。这篇文章将详细
docker from指令的含义_多个FROM-含义 weixin_39722188 docker from指令的含义
小编典典什么是基本图片？一组文件，加上EXPOSE端口ENTRYPOINT和CMD。您可以添加文件并基于该基础图像构建新图像，Dockerfile并以FROM指令开头：后面提到的图像FROM是新图像的“基础图像”。这是否意味着如果我neo4j/neo4j在FROM指令中声明，则在运行映像时，neo数据库将自动运行并且可在端口7474的容器中使用？仅当您不覆盖CMD和时ENTRYPOINT。但是图像
Dockerfile FROM 两个 redDelta
Docker相关视频讲解：什么是容器Docker介绍实现"DockerfileFROM两个"的步骤步骤表格步骤操作1创建一个Dockerfile文件2写入FROM指令3构建第一个镜像4创建第二个Dockerfile文件5写入FROM指令6构建第二个镜像7合并两个镜像操作步骤说明步骤1：创建一个Dockerfile文件使用任意文本编辑器创建一个名为Dockerfile的文件。登录后复制#Docker
Dockerfile命令详解之 FROM 清风怎不知意容器化 java 前端 javascript
许多同学不知道Dockerfile应该如何写，不清楚Dockerfile中的指令分别有什么意义，能达到什么样的目的，接下来我将在容器化专栏中详细的为大家解释每一个指令的含义以及用法。专栏订阅传送门https://blog.csdn.net/qq_38220908/category_11989778.html指令不区分大小写。但是，按照惯例，它们应该是大写的，以便更容易地将它们与参数区分开来。(引用
leetcode 11. 盛最多水的容器 Source_Chang
leetcode核心思想：双指针，数字小的那个指针移动classSolution{public:intmaxArea(vector&height){intleft=0;intright=height.size()-1;intmaxArea=0;while(left
Kubernetes数据持久化看清所苡看轻 kubernetes(k8s)emptyDir HostPath pv pvc kubernetes
在k8s中，Volume（数据卷）存在明确的生命周期（与包含该数据卷的容器组（pod）相同）。因此Volume的生命周期比同一容器组（pod）中任意容器的生命周期要更长，不管容器重启了多少次，数据都被保留下来。当然，如果pod不存在了，数据卷自然退出了。此时，根据pod所使用的数据卷类型不同，数据可能随着数据卷的退出而删除，也可能被真正持久化，并在下次容器组重启时仍然可以使用。从根本上来说，一个数
leetcode刷题day13|二叉树Part01（递归遍历、迭代遍历、统一迭代、层序遍历）小冉在学习 leetcode 算法职场和发展
递归遍历思路：使用递归的方式比较简单。1、递归函数的传参：因为最后输出一个数组，所以需要传入根节点和一个容器，本来想写数组，但发现长度不能确定，所以选择list。2、终止条件：当访问的节点为空时，return3、递归函数的逻辑：先访问一个节点，递归访问其他节点144.二叉树的前序遍历代码如下：classSolution{publicListpreorderTraversal(TreeNoderoo
Kubernetes部署MySQL数据持久化沫殇-MS Kubernetes MySQL数据库 kubernetes mysql 容器
一、安装配置NFS服务端1、安装nfs-kernel-server：sudoapt-yinstallnfs-kernel-server2、服务端创建共享目录#列出所有可用块设备的信息lsblk#格式化磁盘sudomkfs-text4/dev/sdb#创建一个目录：sudomkdir-p/data/nfs/mysql#更改目录权限：sudochown-Rnobody:nogroup/data/nfs
Kubernetes的3种数据持久化方式 Seal^_^ 【云原生】容器化与编排技术持续集成 #Kubernetes kubernetes 容器云原生 EmptyDir 面试 HostPath
Kubernetes的3种数据持久化方式1.EmptyDir2.HostPath3.PersistentVolume(PV)TheBegin点点关注，收藏不迷路Kubernetes提供了几种数据持久化方式，以满足不同场景的需求：1.EmptyDir用途：临时数据存储，Pod内容器间共享。特点：生命周期与Pod相同，Pod删除时数据也删除。2.HostPath用途：访问宿主机特定文件或目录。特点：增
【Kubernetes】常见面试题汇总（十一） summer.335 Kubernetes kubernetes 容器云原生
目录33.简述Kubernetes外部如何访问集群内的服务？34.简述Kubernetesingress？35.简述Kubernetes镜像的下载策略？33.简述Kubernetes外部如何访问集群内的服务？（1）对于Kubernetes，集群外的客户端默认情况，无法通过Pod的IP地址或者Service的虚拟IP地址：虚拟端口号进行访问。（2）通常可以通过以下方式进行访问Kubernetes集群
k8s中Service暴露的种类以及用法听说唐僧不吃肉 K8S kubernetes 容器云原生
一、说明在Kubernetes中，有几种不同的方式可以将服务（Service）暴露给外部流量。这些方式通过定义服务的spec.type字段来确定。二、详解1.ClusterIP定义：默认类型，服务只能在集群内部访问。作用：通过集群内部IP地址暴露服务。示例：spec:type:ClusterIPports:-port:80targetPo
Kubernetes 自定义控制器开发 IT回忆录 Kubenetes kubernetes
目录前言一、CRD二、创建数据库表（Mysql）二、控制器开发1.使用kubernetes的examplecontroller模板2.在controller.go中新增数据表监听方法3.修改tools工具生成资源对象结构体定义这里记录开发k8s控制器的一般方式，controller开发主要使用k8s提供的client-go库进行。前言Controller监听集群内部资源对象的变化，编辑资源对象(增
用kubedam搭建的k8s证书过期处理方法我滴鬼鬼呀wks k8s 1024程序员节
kubeadm部署的k8s证书过期1、查看证书过期时间kubeadmalphacertscheck-expiration若证书已经过期无法试用kubectl命令建议修改服务器时间到未过期的时间段2、配置kube-controller-manager.yaml文件cat/etc/kubernetes/manifests/kube-controller-manager.yamlapiVersion:v
k8s证书过期问题处理 olina_qin kubernetes 容器云原生
k8s证书过期问题处理opensslx509-in/etc/kubernetes/pki/apiserver.crt-noout-dateskubeadmcertsrenewallsystemctlrestartkubeleopensslx509-in/etc/kubernetes/pki/apiserver.crt-noout-text|grep"NotAfter"cp/etc/kubernet
Kubernetes Ingress 控制器（Nginx）安装与使用教程农优影
KubernetesIngress控制器（Nginx）安装与使用教程kubernetes-ingressNGINXandNGINXPlusIngressControllersforKubernetes项目地址:https://gitcode.com/gh_mirrors/ku/kubernetes-ingress1.项目目录结构及介绍在nginxinc/kubernetes-ingress仓库中，
【K8s】专题十一：Kubernetes 集群证书过期处理方法行者Sun1989 Kubernetes kubernetes 云原生容器
本文内容均来自个人笔记并重新梳理，如有错误欢迎指正！如果对您有帮助，烦请点赞、关注、转发、订阅专栏！专栏订阅入口Linux专栏|Docker专栏|Kubernetes专栏往期精彩文章【Docker】（全网首发）KylinV10下MySQL容器内存占用异常的解决方法【Docker】（全网首发）KylinV10下MySQL容器内存占用异常的解决方法（续）【Docker】MySQL源码构建Docker镜
Java面试笔记记录6 今天背八股了吗 java 面试笔记
1.Spring是什么？特性？有哪些模块？Spring是一个轻量级、非入侵式的控制反转Ioc和面向切面AOP的框架。特性：1.Ioc和DISpring的核心就是一个大的工厂容器，可以维护所有对象的创建和依赖关系，Spring工厂用于生成Bean，并且管理Bean的生命周期，实现高内聚低耦合的设计理念。2.AOP编程Spring提供面向切面编程，可以方便实现对程序进行权限拦截、运行监控等切面功能。3
Docker学习十一：Kubernetes概述爱打羽球的程序猿 Docker学习系列 docker kubernetes 学习
一、Kubernetes简介2006年，Google提出了云计算的概念，当时的云计算领域还是以虚拟机为代表的云平台。2013年，Docker横空出世，Docker提出了镜像、仓库等核心概念，规范了服务的交付标准，使得复杂服务的落地变得更加简单，之后Docker又定义了OCI标准，Docker在容器领域称为事实的标准。但是，Docker诞生只是帮助定义了开发和交付标准，如果想要在生产环境中大批量的使
Cloud Native Weekly | 华为云抢先发布Redis5.0，红帽宣布收购混合云提供商 weixin_34302561 数据库 devops 大数据
1——华为云抢先发布Redis5.02——DigitalOceanK8s服务正式上线3——红帽宣布收购混合云提供商NooBaa4——微软发布多项AzureKubernetes服务更新1华为云抢先发布Redis5.012月17日，华为云在DCS2.0的基础上，快人一步，抢先推出了新的Redis5.0产品，这是一个崭新的突破。目前国内在缓存领域的发展普遍停留在Redis4.0阶段，华为云率先发布了Re
Halo 开发者指南——容器私有化部署 SHENHUANJIE Docker Halo 华为云 SWR Registry
华为云SWR私有化部署镜像构建dockerbuild-thalo-dev/halo:2.20.0.上传镜像镜像标签sudodockertag{镜像名称}:{版本名称}swr.cn-south-1.myhuaweicloud.com/{组织名称}/{镜像名称}:{版本名称}sudodockertaghalo-dev/halo:2.20.0swr.cn-south-1.myhuaweicloud.co
SpringBoot整合ES搜索引擎实现网站热搜词及热度计算码踏云端 springboot Elasticsearch spring boot elasticsearch 后端热搜词热度计算 java
博主简介：历代文学网（PC端可以访问：https://literature.sinhy.com/#/literature?__c=1000，移动端可微信小程序搜索“历代文学”）总架构师，15年工作经验，精通Java编程，高并发设计，Springboot和微服务，熟悉Linux，ESXI虚拟化以及云原生Docker和K8s，热衷于探索科技的边界，并将理论知识转化为实际应用。保持对新技术的好奇心，乐于
小白 | 华为云docker设置镜像加速器伏一工具安装华为云 docker 容器
一、操作场景通过dockerpull命令下载镜像中心的公有镜像时，往往会因为网络原因而需要很长时间，甚至可能因超时而下载失败。为此，容器镜像服务提供了镜像下载加速功能，帮助您获得更快的下载体验。二、约束与限制构建镜像的客户端所安装的容器引擎（Docker）版本必须为1.11.2及以上。“华北-乌兰察布一”、“亚太-雅加达”、“拉美-墨西哥城一”、“拉美-墨西哥城二”和“拉美-圣保罗一”区域不支持该
docker改容器IP的两种方法 redmond88 linux docker tcp/ip 容器
最简单实用的方法：docker默认的内网网段为172.17.0.0/16，如果公司内网网段也是172.17.x.x的话，就会发生路由冲突。解决办法改路由比较办法，可以一开始就将docker配置的bip改成169.254.0.1/24，可以避免冲突。在daemon配置文件里加个"bip":“169.254.0.1/24”，重启docker就可以了1234[root@st-dev6~]#vim/etc
java获取applicationcontext,SpringBoot获取ApplicationContext的3种方式花儿街参考
ApplicationContext是什么？简单来说就是Spring中的容器，可以用来获取容器中的各种bean组件，注册监听事件，加载资源文件等功能。ApplicationContext获取的几种方式1直接使用Autowired注入@ComponentpublicclassBook1{@AutowiredprivateApplicationContextapplicationContext;pub
（k8s）Kubernetes 从0到1容器编排之旅道不贱卖，法不轻传 kubernets kubernetes 容器云原生
一、引言在当今数字化的浪潮中，Kubernetes如同一艘强大的航船，引领着容器化应用的部署与管理。它以其卓越的灵活性、可扩展性和可靠性，成为众多企业和开发者的首选。然而，要真正发挥Kubernetes的强大威力，仅仅掌握基本操作是远远不够的。本文将带你深入探索Kubernetes使用过程中的奇技妙法，为你开启一段优雅的容器编排之旅。二、高级资源管理之精妙艺术1.资源配额与限制：雕琢资源之美•Ku
[黑洞与暗粒子]没有光的世界 comsci
无论是相对论还是其它现代物理学,都显然有个缺陷,那就是必须有光才能够计算但是,我相信,在我们的世界和宇宙平面中,肯定存在没有光的世界.... 那么,在没有光的世界,光子和其它粒子的规律无法被应用和考察,那么以光速为核心的 &nbs
jQuery Lazy Load 图片延迟加载 aijuans jquery
基于 jQuery 的图片延迟加载插件，在用户滚动页面到图片之后才进行加载。对于有较多的图片的网页，使用图片延迟加载，能有效的提高页面加载速度。版本： jQuery v1.4.4+ jQuery Lazy Load v1.7.2 注意事项：需要真正实现图片延迟加载，必须将真实图片地址写在 data-original 属性中。若 src
使用Jodd的优点 Kai_Ge jodd
1. 简化和统一 controller ，抛弃 extends SimpleFormController ，统一使用 implements Controller 的方式。 2. 简化 JSP 页面的 bind, 不需要一个字段一个字段的绑定。 3. 对 bean 没有任何要求，可以使用任意的 bean 做为 formBean。使用方法简介
jpa Query转hibernate Query 120153216 Hibernate
public List<Map> getMapList(String hql, Map map) { org.hibernate.Query jpaQuery = entityManager.createQuery(hql); if (null != map) { for (String parameter : map.keySet()) { jp
Django_Python3添加MySQL/MariaDB支持 2002wmj mariaDB
现状首先，[email protected] 中默认的引擎为 django.db.backends.mysql 。但是在Python3中如果这样写的话，会发现 django.db.backends.mysql 依赖 MySQLdb[5] ，而 MySQLdb 又不兼容 Python3 于是要找一种新的方式来继续使用MySQL。 MySQL官方的方案首先据MySQL文档[3]说，自从MySQL
在SQLSERVER中查找消耗IO最多的SQL 357029540 SQL Server
返回做IO数目最多的50条语句以及它们的执行计划。 select top 50 (total_logical_reads/execution_count) as avg_logical_reads, (total_logical_writes/execution_count) as avg_logical_writes, (tot
spring UnChecked 异常官方定义！ 7454103 spring
如果你接触过spring的事物管理！那么你必须明白 spring的非捕获异常！即 unchecked 异常！因为 spring 默认这类异常事物自动回滚！！ public static boolean isCheckedException(Throwable ex) { return !(ex instanceof RuntimeExcep
mongoDB 入门指南、示例 adminjun java mongodb 操作
一、准备工作 1、下载mongoDB 下载地址：http://www.mongodb.org/downloads 选择合适你的版本相关文档：http://www.mongodb.org/display/DOCS/Tutorial 2、安装mongoDB A、不解压模式：将下载下来的mongoDB-xxx.zip打开，找到bin目录，运行mongod.exe就可以启动服务，默
CUDA 5 Release Candidate Now Available aijuans CUDA
The CUDA 5 Release Candidate is now available at http://developer.nvidia.com/<wbr></wbr>cuda/cuda-pre-production. Now applicable to a broader set of algorithms, CUDA 5 has advanced fe
Essential Studio for WinRT网格控件测评 Axiba JavaScript html5
Essential Studio for WinRT界面控件包含了商业平板应用程序开发中所需的所有控件，如市场上运行速度最快的grid 和chart、地图、RDL报表查看器、丰富的文本查看器及图表等等。同时，该控件还包含了一组独特的库，用于从WinRT应用程序中生成Excel、Word以及PDF格式的文件。此文将对其另外一个强大的控件——网格控件进行专门的测评详述。网格控件功能 1、
java 获取windows系统安装的证书或证书链 bewithme windows
有时需要获取windows系统安装的证书或证书链，比如说你要通过证书来创建java的密钥库。有关证书链的解释可以查看此处。 public static void main(String[] args) { SunMSCAPI providerMSCAPI = new SunMSCAPI(); S
NoSQL数据库之Redis数据库管理(set类型和zset类型) bijian1013 redis 数据库 NoSQL
4.sets类型 Set是集合，它是string类型的无序集合。set是通过hash table实现的，添加、删除和查找的复杂度都是O(1)。对集合我们可以取并集、交集、差集。通过这些操作我们可以实现sns中的好友推荐和blog的tag功能。 sadd：向名称为key的set中添加元
异常捕获何时用Exception，何时用Throwable bingyingao
用Exception的情况 try { //可能发生空指针、数组溢出等异常 } catch (Exception e) {
【Kafka四】Kakfa伪分布式安装 bit1129 kafka
在http://bit1129.iteye.com/blog/2174791一文中，实现了单Kafka服务器的安装，在Kafka中，每个Kafka服务器称为一个broker。本文简单介绍下，在单机环境下Kafka的伪分布式安装和测试验证 1. 安装步骤 Kafka伪分布式安装的思路跟Zookeeper的伪分布式安装思路完全一样，不过比Zookeeper稍微简单些(不
Project Euler bookjovi haskell
Project Euler是个数学问题求解网站，网站设计的很有意思，有很多problem，在未提交正确答案前不能查看problem的overview，也不能查看关于problem的discussion thread，只能看到现在problem已经被多少人解决了，人数越多往往代表问题越容易。看看problem 1吧： Add all the natural num
Java-Collections Framework学习与总结-ArrayDeque BrokenDreams Collections
表、栈和队列是三种基本的数据结构，前面总结的ArrayList和LinkedList可以作为任意一种数据结构来使用，当然由于实现方式的不同，操作的效率也会不同。这篇要看一下java.util.ArrayDeque。从命名上看
读《研磨设计模式》-代码笔记-装饰模式-Decorator bylijinnan java 设计模式
声明：本文只为方便我个人查阅和理解，详细的分析以及源代码请移步原作者的博客http://chjavach.iteye.com/ import java.io.BufferedOutputStream; import java.io.DataOutputStream; import java.io.FileOutputStream; import java.io.Fi
Maven学习(一) chenyu19891124 Maven私服
学习一门技术和工具总得花费一段时间，5月底6月初自己学习了一些工具，maven+Hudson+nexus的搭建，对于maven以前只是听说，顺便再自己的电脑上搭建了一个maven环境，但是完全不了解maven这一强大的构建工具，还有ant也是一个构建工具，但ant就没有maven那么的简单方便，其实简单点说maven是一个运用命令行就能完成构建，测试，打包，发布一系列功
[原创]JWFD工作流引擎设计----节点匹配搜索算法(用于初步解决条件异步汇聚问题) 补充 comsci 算法工作 PHP 搜索引擎嵌入式
本文主要介绍在JWFD工作流引擎设计中遇到的一个实际问题的解决方案，请参考我的博文"带条件选择的并行汇聚路由问题"中图例A2描述的情况(http://comsci.iteye.com/blog/339756),我现在把我对图例A2的一个解决方案公布出来，请大家多指点节点匹配搜索算法(用于解决标准对称流程图条件汇聚点运行控制参数的算法) 需要解决的问题：已知分支
Linux中用shell获取昨天、明天或多天前的日期 daizj linux shell 上几年昨天获取上几个月
在Linux中可以通过date命令获取昨天、明天、上个月、下个月、上一年和下一年 # 获取昨天 date -d 'yesterday' # 或 date -d 'last day' # 获取明天 date -d 'tomorrow' # 或 date -d 'next day' # 获取上个月 date -d 'last month' #
我所理解的云计算 dongwei_6688 云计算
在刚开始接触到一个概念时，人们往往都会去探寻这个概念的含义，以达到对其有一个感性的认知，在Wikipedia上关于“云计算”是这么定义的，它说： Cloud computing is a phrase used to describe a variety of computing co
YII CMenu配置 dcj3sjt126com yii
Adding id and class names to CMenu We use the id and htmlOptions to accomplish this. Watch. //in your view $this->widget('zii.widgets.CMenu', array( 'id'=>'myMenu', 'items'=>$this-&g
设计模式之静态代理与动态代理 come_for_dream 设计模式
静态代理与动态代理代理模式是java开发中用到的相对比较多的设计模式，其中的思想就是主业务和相关业务分离。所谓的代理设计就是指由一个代理主题来操作真实主题，真实主题执行具体的业务操作，而代理主题负责其他相关业务的处理。比如我们在进行删除操作的时候需要检验一下用户是否登陆，我们可以删除看成主业务，而把检验用户是否登陆看成其相关业务
【转】理解Javascript 系列 gcc2ge JavaScript
理解Javascript_13_执行模型详解摘要: 在《理解Javascript_12_执行模型浅析》一文中,我们初步的了解了执行上下文与作用域的概念，那么这一篇将深入分析执行上下文的构建过程，了解执行上下文、函数对象、作用域三者之间的关系。函数执行环境简单的代码:当调用say方法时，第一步是创建其执行环境，在创建执行环境的过程中，会按照定义的先后顺序完成一系列操作:1.首先会创建一个
Subsets II hcx2013 set
Given a collection of integers that might contain duplicates, nums, return all possible subsets. Note: Elements in a subset must be in non-descending order. The solution set must not conta
Spring4.1新特性——Spring缓存框架增强 jinnianshilongnian spring4
目录 Spring4.1新特性——综述 Spring4.1新特性——Spring核心部分及其他 Spring4.1新特性——Spring缓存框架增强 Spring4.1新特性——异步调用和事件机制的异常处理 Spring4.1新特性——数据库集成测试脚本初始化 Spring4.1新特性——Spring MVC增强 Spring4.1新特性——页面自动化测试框架Spring MVC T
shell嵌套expect执行命令 liyonghui160com
一直都想把expect的操作写到bash脚本里,这样就不用我再写两个脚本来执行了,搞了一下午终于有点小成就,给大家看看吧. 系统:centos 5.x 1.先安装expect yum -y install expect 2.脚本内容: cat auto_svn.sh #!/bin/bash
Linux实用命令整理 pda158 linux
0. 基本命令　　linux 基本命令整理　　1. 压缩解压　　tar -zcvf a.tar.gz a #把a压缩成a.tar.gz 　　tar -zxvf a.tar.gz #把a.tar.gz解压成a 　　2. vim小结　　2.1 vim替换　　:m,ns/word_1/word_2/gc
独立开发人员通向成功的29个小贴士 shoothao 独立开发
概述：本文收集了关于独立开发人员通向成功需要注意的一些东西,对于具体的每个贴士的注解有兴趣的朋友可以查看下面标注的原文地址。明白你从事独立开发的原因和目的。保持坚持制定计划的好习惯。万事开头难，第一份订单是关键。培养多元化业务技能。提供卓越的服务和品质。谨小慎微。营销是必备技能。学会组织，有条理的工作才是最有效率的。 “独立
JAVA中堆栈和内存分配原理 uule java
1、栈、堆 1.寄存器：最快的存储区, 由编译器根据需求进行分配,我们在程序中无法控制.2. 栈：存放基本类型的变量数据和对象的引用，但对象本身不存放在栈中，而是存放在堆（new 出来的对象）或者常量池中（字符串常量对象存放在常量池中。）3. 堆：存放所有new出来的对象。4. 静态域：存放静态成员（static定义的）5. 常量池：存放字符串常量和基本类型常量（public static f

内网环境基于 k8s 的大型网站电商解决方案(一）

一、环境说明

二、基础环境搭建

1、安装系统rockylinux8.6最小化

2、安装基础软件包及chrony（manager)

3、配置manager服务器yum源

4、搭建yum源服务器（manager）

5、安装dnsmasq(manager)

6、安装docker

7、配置内网服务器yum源及NTP配置

三、搭建harbor私有仓库(harbor1、harbor2）

1、安装docker

2、修改内核参数

3、安装docker-compose

4、生成ca证书

5、生成域名证书

6、安装harbor

7、 停止harbor

8、启动harbor

9、配置镜像自动同步

10、在192.168.8.80上测试

四、安装k8s高可用集群（master1-3，node1-2)

1、修改内核参数（五台设备上执行）

2、开启Ipvs 五台设备

3、安装containerd（五台设备执行）

4、安装kubelet kubeadm kubectl

5、下载镜像并上传（manager)

6、安装keepalive+nginx

1)（三台master上执行）

2)keeplive配置（master1)

2)keeplive配置（master2)

3)keeplive配置（master3)

master2

7、初始化k8s集群（master节点上执行）

8、安装网络组件

9、将master2、master3加入集群

10、工作节点加入集群

11、验证

12、测试

13、部署ingress

五、安装ceph

1、安装cephadm（cepha)

2、下载镜像（manager)

3、初始化集群

4、ceph对接k8s

六、安装mysql(MGR）

1、三台设置防火墙（mysqla、mysqlb、mysqlc）

2、安装mysql

3、启动mysqld

4、安装插件及配置用户

5、配置mgr

6、修改配置文件

8、改为多主模式

9、安装 keepalived+nginx 实现 MGR高可用

1)（三台mysql上执行）

2)keeplive配置（mysqla)

3)keeplive配置（mysqlb)

4)keeplive配置（mysqlc)

10、测试mysql读写（manager)

你可能感兴趣的:(kubernetes,容器,云原生)

7、停止harbor