centos7安装k3s和rancher

一. 安装k3s

官方文档 https://docs.k3s.io/zh/

参考 https://www.pudn.com/news/62dfbeec55398e076be025c8.html

参考 https://blog.csdn.net/RancherLabs/article/details/128093716

1.1 配置yum阿里源

yum -y install wget
mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.bak
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
yum clean all
yum makecache

1.2 关闭防火墙

# 查看防火墙状态
firewall-cmd --state
# 临时停止防火墙
systemctl stop firewalld.service
# 禁止防火墙开机启动
systemctl disable firewalld.service

1.3 修改hostname

hostnamectl set-hostname master

1.4 安装containerd

# 安装依赖
yum install -y yum-utils device-mapper-persistent-data lvm2

# 配置yum源
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

# 查看containerd版本，发现最新版是 1.6.14-3.1.el7
yum list containerd.io --showduplicates | sort -r

# 安装containerd
yum install containerd.io -y

# 生成配置文件
containerd config default > /etc/containerd/config.toml

# 替换 containerd 默认的 sand_box 镜像，编辑 /etc/containerd/config.toml
sed -i 's#sandbox_image = "registry.k8s.io/pause:3.6"#sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.6"#' /etc/containerd/config.toml

# 启动服务
systemctl restart containerd
systemctl enable containerd

安装 CRI 客户端 nerdctl
选择版本 https://github.com/containerd/nerdctl/releases

# 删除前面的 v
export version=1.1.0
wget https://github.com/containerd/nerdctl/releases/download/v${version}/nerdctl-full-${version}-linux-amd64.tar.gz
tar Cxzvvf /usr/local nerdctl-full-${version}-linux-amd64.tar.gz

如果腾讯云服务器下载github资源连接不了，可以参考 https://www.itbulu.com/git-github-outtime.html

验证是否可用

nerdctl pull nginx:alpine
nerdctl images
nerdctl rmi nginx:alpine

1.5 containerd安装mysql8

存储k3s数据，不使用etcd

参考 https://blog.csdn.net/qq_38983728/article/details/128485470

新增数据库k3s

1.6 安装k3s

查看版本 https://github.com/k3s-io/k3s/releases

安装版本 v1.24.9+k3s1

因为helm安装rancher最新版2.7.0 k3s 版本必须小于v1.25.0

export INSTALL_K3S_VERSION=v1.24.9+k3s1

curl –sfL https://rancher-mirror.oss-cn-beijing.aliyuncs.com/k3s/k3s-install.sh | \
     INSTALL_K3S_MIRROR=cn sh -s - \
     --system-default-registry "registry.cn-hangzhou.aliyuncs.com" \
     --disable traefik \
     --datastore-endpoint='mysql://root:wwwroot@tcp(10.0.89.127:3306)/k3s'

说明：

• --disable traefik：不安装traefik，后面自己安装ingress-nginx
• --datastore-endpoint='mysql:：不使用etcd，使用mysql存储

安装完成

[root@master ~]# kubectl get nodes
NAME     STATUS   ROLES                  AGE   VERSION
master   Ready    control-plane,master   39s   v1.24.9+k3s1
[root@master ~]# kubectl get pods -n kube-system
NAME                                      READY   STATUS    RESTARTS   AGE
local-path-provisioner-79bf9db7dd-b4hzt   1/1     Running   0          91s
coredns-59cd496867-26sb4                  1/1     Running   0          91s
metrics-server-9dc9c745f-z82n2            1/1     Running   0          91s

1.7 local-path 默认存储

1.7.1 修改存储路径

local-path StorageClass 的部署文件位置：/var/lib/rancher/k3s/server/manifests/local-storage.yaml

默认存储路径

/var/lib/rancher/k3s/storage

替换 /var/lib/rancher/k3s/storage 为 /k3s/data

mkdir -p /k3s/data
sed -i 's#/var/lib/rancher/k3s/storage#/k3s/data#' /var/lib/rancher/k3s/server/manifests/local-storage.yaml

1.7.2 修改回收策略

默认回收策略reclaimPolicy是Delete，卸载之后会自动删除pv，导致持久化存储被删除，
修改为Retain（删除PV后后端存储上的数据仍然存在，如需彻底删除则需要手动删除后端存储volume）

替换 Delete 为 Retain

sed -i 's#reclaimPolicy: Delete#reclaimPolicy: Retain#' /var/lib/rancher/k3s/server/manifests/local-storage.yaml

例如：helm卸载应用后

[root@master ~]# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM                               STORAGECLASS   REASON   AGE
pvc-cd885a56-ebe9-4b40-a705-82a53524e1be   8Gi        RWO            Retain           Released   default/nexus-sonatype-nexus-data   local-path              42m

此时pv状态是Released
如果重新部署想要继续使用该pv

kubectl edit pv pvc-cd885a56-ebe9-4b40-a705-82a53524e1be

删除claimRef

保存，退出

此时pv状态是Available，接下来helm部署会自动使用该pv的存储

[root@master ~]# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
pvc-cd885a56-ebe9-4b40-a705-82a53524e1be   8Gi        RWO            Retain           Available           local-path              46m

1.7.3 执行

先删除在部署，否则报错Forbidden: updates to reclaimPolicy are forbidden.

kubectl delete -f /var/lib/rancher/k3s/server/manifests/local-storage.yaml
kubectl apply -f /var/lib/rancher/k3s/server/manifests/local-storage.yaml

1.8 卸载k3s

# master
/usr/local/bin/k3s-uninstall.sh
# node
/usr/local/bin/k3s-agent-uninstall.sh

二. 安装rancher

2.1 安装helm

2.1.1 下载

下载地址 https://github.com/helm/helm/releases

wget https://get.helm.sh/helm-v3.10.3-linux-amd64.tar.gz

服务器下载速度太慢，可以科学上网下载好了上传到服务器，安装lrzsz https://blog.csdn.net/qq_22356995/article/details/104071562

解压

tar -xvf helm-v3.10.3-linux-amd64.tar.gz

2.1.2 安装

将helm移到/usr/local/bin目录

mv linux-amd64/helm /usr/local/bin

查看版本

[root@master i]# helm version
version.BuildInfo{Version:"v3.10.3", GitCommit:"835b7334cfe2e5e27870ab3ed4135f136eecc704", GitTreeState:"clean", GoVersion:"go1.18.9"}

2.1.3 添加几个repo

helm repo add stable https://charts.helm.sh/stable
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update

2.1.4 报错

[root@master ~]# helm list -n kube-system
Error: Kubernetes cluster unreachable: Get "http://localhost:8080/version": dial tcp [::1]:8080: connect: connection refused

原因：helm 默认使用的是 kubectl 默认位置配置文件，对于 k3s 的情况，我们只需要明确为设置 KUBECONFIG 的值后即可正常使用

echo "export KUBECONFIG=/etc/rancher/k3s/k3s.yaml" >> /etc/profile && source /etc/profile

查看

[root@master ~]# helm list -n kube-system
NAME	NAMESPACE	REVISION	UPDATED	STATUS	CHART	APP VERSION

2.2 helm安装ingress-nginx

参考 https://blog.csdn.net/qq_38983728/article/details/123399245

2.3 添加rancher repo

helm repo add rancher-stable http://rancher-mirror.oss-cn-beijing.aliyuncs.com/server-charts/stable

[root@master ~]# helm search repo rancher-stable/rancher
NAME                  	CHART VERSION	APP VERSION	DESCRIPTION                                       
rancher-stable/rancher	2.7.0        	v2.7.0     	Install Rancher Server to manage Kubernetes clu...

2.4 helm安装rancher(自己证书，没有使用cert-manager)

本地虚拟机，自定义域名

添加命名空间

kubectl create ns cattle-system

2.4.1 生成证书

准备证书官方文档

mkdir -p /root/i/certs && cd /root/i/certs

vim create_self-signed-cert.sh

复制

#!/bin/bash -e

help ()
{
    echo  ' ================================================================ '
    echo  ' --ssl-domain: 生成ssl证书需要的主域名，如不指定则默认为www.rancher.local，如果是ip访问服务，则可忽略；'
    echo  ' --ssl-trusted-ip: 一般ssl证书只信任域名的访问请求，有时候需要使用ip去访问Server，那么需要给ssl证书添加扩展IP，多个IP用逗号隔开；'
    echo  ' --ssl-trusted-domain: 如果想多个域名访问，则添加扩展域名（SSL_TRUSTED_DOMAIN）,多个扩展域名用逗号隔开；'
    echo  ' --ssl-size: ssl加密位数，默认2048；'
    echo  ' --ssl-cn: 国家代码(2个字母的代号),默认CN;'
    echo  ' 使用示例:'
    echo  ' ./create_self-signed-cert.sh --ssl-domain=www.test.com --ssl-trusted-domain=www.test2.com \ '
    echo  ' --ssl-trusted-ip=1.1.1.1,2.2.2.2,3.3.3.3 --ssl-size=2048 --ssl-date=3650'
    echo  ' ================================================================'
}

case "$1" in
    -h|--help) help; exit;;
esac

if [[ $1 == '' ]];then
    help;
    exit;
fi

CMDOPTS="$*"
for OPTS in $CMDOPTS;
do
    key=$(echo ${OPTS} | awk -F"=" '{print $1}' )
    value=$(echo ${OPTS} | awk -F"=" '{print $2}' )
    case "$key" in
        --ssl-domain) SSL_DOMAIN=$value ;;
        --ssl-trusted-ip) SSL_TRUSTED_IP=$value ;;
        --ssl-trusted-domain) SSL_TRUSTED_DOMAIN=$value ;;
        --ssl-size) SSL_SIZE=$value ;;
        --ssl-date) SSL_DATE=$value ;;
        --ca-date) CA_DATE=$value ;;
        --ssl-cn) CN=$value ;;
    esac
done

# CA相关配置
CA_DATE=${CA_DATE:-3650}
CA_KEY=${CA_KEY:-cakey.pem}
CA_CERT=${CA_CERT:-cacerts.pem}
CA_DOMAIN=cattle-ca

# ssl相关配置
SSL_CONFIG=${SSL_CONFIG:-$PWD/openssl.cnf}
SSL_DOMAIN=${SSL_DOMAIN:-'www.rancher.local'}
SSL_DATE=${SSL_DATE:-3650}
SSL_SIZE=${SSL_SIZE:-2048}

## 国家代码(2个字母的代号),默认CN;
CN=${CN:-CN}

SSL_KEY=$SSL_DOMAIN.key
SSL_CSR=$SSL_DOMAIN.csr
SSL_CERT=$SSL_DOMAIN.crt

echo -e "\033[32m ---------------------------- \033[0m"
echo -e "\033[32m       | 生成 SSL Cert |       \033[0m"
echo -e "\033[32m ---------------------------- \033[0m"

if [[ -e ./${CA_KEY} ]]; then
    echo -e "\033[32m ====> 1. 发现已存在CA私钥，备份"${CA_KEY}"为"${CA_KEY}"-bak，然后重新创建 \033[0m"
    mv ${CA_KEY} "${CA_KEY}"-bak
    openssl genrsa -out ${CA_KEY} ${SSL_SIZE}
else
    echo -e "\033[32m ====> 1. 生成新的CA私钥 ${CA_KEY} \033[0m"
    openssl genrsa -out ${CA_KEY} ${SSL_SIZE}
fi

if [[ -e ./${CA_CERT} ]]; then
    echo -e "\033[32m ====> 2. 发现已存在CA证书，先备份"${CA_CERT}"为"${CA_CERT}"-bak，然后重新创建 \033[0m"
    mv ${CA_CERT} "${CA_CERT}"-bak
    openssl req -x509 -sha256 -new -nodes -key ${CA_KEY} -days ${CA_DATE} -out ${CA_CERT} -subj "/C=${CN}/CN=${CA_DOMAIN}"
else
    echo -e "\033[32m ====> 2. 生成新的CA证书 ${CA_CERT} \033[0m"
    openssl req -x509 -sha256 -new -nodes -key ${CA_KEY} -days ${CA_DATE} -out ${CA_CERT} -subj "/C=${CN}/CN=${CA_DOMAIN}"
fi

echo -e "\033[32m ====> 3. 生成Openssl配置文件 ${SSL_CONFIG} \033[0m"
cat > ${SSL_CONFIG} <> ${SSL_CONFIG} <> ${SSL_CONFIG}
    done

    if [[ -n ${SSL_TRUSTED_IP} ]]; then
        ip=(${SSL_TRUSTED_IP})
        for i in "${!ip[@]}"; do
          echo IP.$((i+1)) = ${ip[$i]} >> ${SSL_CONFIG}
        done
    fi
fi

echo -e "\033[32m ====> 4. 生成服务SSL KEY ${SSL_KEY} \033[0m"
openssl genrsa -out ${SSL_KEY} ${SSL_SIZE}

echo -e "\033[32m ====> 5. 生成服务SSL CSR ${SSL_CSR} \033[0m"
openssl req -sha256 -new -key ${SSL_KEY} -out ${SSL_CSR} -subj "/C=${CN}/CN=${SSL_DOMAIN}" -config ${SSL_CONFIG}

echo -e "\033[32m ====> 6. 生成服务SSL CERT ${SSL_CERT} \033[0m"
openssl x509 -sha256 -req -in ${SSL_CSR} -CA ${CA_CERT} \
    -CAkey ${CA_KEY} -CAcreateserial -out ${SSL_CERT} \
    -days ${SSL_DATE} -extensions v3_req \
    -extfile ${SSL_CONFIG}

echo -e "\033[32m ====> 7. 证书制作完成 \033[0m"
echo
echo -e "\033[32m ====> 8. 以YAML格式输出结果 \033[0m"
echo "----------------------------------------------------------"
echo "ca_key: |"
cat $CA_KEY | sed 's/^/  /'
echo
echo "ca_cert: |"
cat $CA_CERT | sed 's/^/  /'
echo
echo "ssl_key: |"
cat $SSL_KEY | sed 's/^/  /'
echo
echo "ssl_csr: |"
cat $SSL_CSR | sed 's/^/  /'
echo
echo "ssl_cert: |"
cat $SSL_CERT | sed 's/^/  /'
echo

echo -e "\033[32m ====> 9. 附加CA证书到Cert文件 \033[0m"
cat ${CA_CERT} >> ${SSL_CERT}
echo "ssl_cert: |"
cat $SSL_CERT | sed 's/^/  /'
echo

echo -e "\033[32m ====> 10. 重命名服务证书 \033[0m"
echo "cp ${SSL_DOMAIN}.key tls.key"
cp ${SSL_DOMAIN}.key tls.key
echo "cp ${SSL_DOMAIN}.crt tls.crt"
cp ${SSL_DOMAIN}.crt tls.crt

chmod +x create_self-signed-cert.sh

添加域名和IP

sh ./create_self-signed-cert.sh --ssl-domain=rancher.wanfei.com --ssl-trusted-ip=10.0.89.127 --ssl-size=2048 --ssl-date=3650

2.4.2 上传证书到k3s集群

把生成的pem证书上传到集群中去，ingress要用

export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
kubectl --kubeconfig=$kubeconfig create namespace cattle-system
kubectl --kubeconfig=$kubeconfig \
    -n cattle-system create \
    secret tls tls-rancher-ingress \
    --cert=./tls.crt \
    --key=./tls.key

rancher要用

kubectl -n cattle-system create secret generic tls-ca --from-file=cacerts.pem=./cacerts.pem

查看

[root@master certs]# kubectl get secret -n cattle-system
NAME                  TYPE                DATA   AGE
tls-rancher-ingress   kubernetes.io/tls   2      5m7s
tls-ca                Opaque              1      2m10s

2.4.3 安装

Chart 安装选项官方文档

使用您已有的证书

[root@master certs]# pwd
/root/i/certs
[root@master certs]# openssl x509 -noout -subject -in cacerts.pem
subject= /C=CN/CN=cattle-ca

helm安装

helm upgrade -i rancher rancher-stable/rancher --version 2.7.0 \
 --namespace cattle-system \
 --set hostname=rancher.wanfei.com \
 --set ingress.tls.source=secret \
 --set ingress.extraAnnotations.'kubernetes\.io/ingress\.class'=nginx \
 --set rancherImage=registry.cn-hangzhou.aliyuncs.com/rancher/rancher \
 --set replicas=1 \
 --set privateCA=true \
 --set bootstrapPassword="admin123456"

注意：rancher版本2.7.0时，k3s版本不能超过1.25.0，否则报错
Error: INSTALLATION FAILED: chart requires kubeVersion: < 1.25.0-0 which is incompatible with Kubernetes v1.25.4+k3s1

2.4.4 访问Rancher

添加hosts

10.0.89.127 rancher.wanfei.com

查看

[root@master ~]# kubectl get ingress -A
NAMESPACE       NAME      CLASS    HOSTS                ADDRESS       PORTS     AGE
cattle-system   rancher      rancher.wanfei.com   10.0.89.127   80, 443   21m

验证

[root@master ~]# kubectl -n cattle-system rollout status deploy/rancher
deployment "rancher" successfully rolled out
[root@master ~]# kubectl -n cattle-system get deploy rancher
NAME      READY   UP-TO-DATE   AVAILABLE   AGE
rancher   1/1     1            1           11m

访问 https://rancher.wanfei.com

输入上面设置的密码

2.5 helm安装rancher(cert-manger自动Https)

部署在云服务器，有域名

2.5.1 helm安装cert-manager

参考 https://blog.csdn.net/qq_38983728/article/details/103462543

2.5.2 my-values.yaml

mkdir -p /root/i/helm/rancher && cd /root/i/helm/rancher

cat < /root/i/helm/rancher/my-values.yaml
hostname: rancher.wanfei.wang
ingress:
  extraAnnotations:
    kubernetes.io/ingress.class: "nginx"    
    kubernetes.io/tls-acme: "true"  
  tls:
    source: letsEncrypt
    secretName: tls-rancher-ingress
letsEncrypt:
  email: [email protected]
  environment: production
  ingress:
    class: "nginx"

replicas: 1
bootstrapPassword: "admin123456"
EOF

2.5.3 安装

helm upgrade -i rancher rancher-stable/rancher --version 2.7.0 \
 --namespace cattle-system --create-namespace \
 -f my-values.yaml

访问 https://rancher.wanfei.wang