在阿里云托管kubernetes上利用 cert-manager 自动签发 TLS 证书[无坑版]

前言

排错的过程是痛苦的也是有趣的。

运维乃至IT，排错能力是拉开人与人之间的重要差距。

本篇会记录我的排错之旅。

由来

现如今我司所有业务都运行在阿里云托管kubernetes环境上，因为前端需要对外访问，所以需要对外域名，考虑申请https证书过于麻烦，所以希望借助免费的工具自动生成tls证书。

借鉴于网上或者阿里云的相关文档是存在大坑的，我认为有必要写一篇无坑版的利用cert-manager自动签发TLS证书。

思路

cert-manager是Kubernetes上一个管理SSL证书的插件，配合nginx-ingress可以对网站配置https访问，在加上letsencrypt提供免费的SSL证书，所有就产生了cert-manager+nginx-ingress+letsencrypt的免费套餐。详情请到GitHub查看：cert-manager

本文将介绍：基于阿里云托管kubernetes+cert-manager的 单域名，通配符域名证书申请。

部署

注：网上大多都使用helm 部署的，而helm部署确实非常简单，我认为最好最好不要使用他人的helm清单，不然出问题，就不晓得是怎么一个部署逻辑，还需要去分析资源清单。

前提

kubectl create namespace cert-manager  #创建 cert-manager 命名空间
kubectl label namespace cert-manager certmanager.k8s.io/disable-validation=true #标记 cert-manager 命名空间以禁用资源验证

配置CRDs

wget https://github.com/jetstack/cert-manager/releases/download/v0.15.1/cert-manager-legacy.crds.yaml
kubectl apply --validate=false -f cert-manager-legacy.crds.yaml

配置cert-manager

# wget https://github.com/jetstack/cert-manager/releases/download/v0.15.1/cert-manager.yaml 
# kubectl apply --validate=false -f cert-manager.yaml

这里会发现pod一直处于ContainerCreating 容器创建中
# kubectl get pods -n cert-manager 
NAME                                           READY   STATUS              RESTARTS   AGE
cert-manager-75856bb467-fr5zz                  0/1     ContainerCreating   0          16m
cert-manager-cainjector-597f5b4768-jqsvp       0/1     ContainerCreating   0          16m
cert-manager-webhook-5c9f7b5f75-gnphd          0/1     ContainerCreating   0          16m

#查看pod 详情，会发现是因为拉取镜像的问题，我的解决方案
#在香港机上拉取镜像打标签，推送到镜像仓库然后修改cert-manager.yaml的镜像地址
#查看原资源清单镜像
# cat cert-manager.yaml |grep image
          image: "quay.io/jetstack/cert-manager-cainjector:v0.15.1"
          image: "quay.io/jetstack/cert-manager-controller:v0.15.1"
          image: "quay.io/jetstack/cert-manager-webhook:v0.15.1"      

#香港机拉取--> 打标签 --> 推送
#   docker pull  quay.io/jetstack/cert-manager-cainjector:v0.15.1
# docker tag quay.io/jetstack/cert-manager-cainjector:v0.15.1  registry.cn-shenzhen.aliyuncs.com/realibox_test/cert-manager:cert-manager-cainjector-v0.15.1
# docker push registry.cn-shenzhen.aliyuncs.com/realibox_test/cert-manager:cert-manager-cainjector-v0.15.1
# docker pull quay.io/jetstack/cert-manager-controller:v0.15.1
# docker tag quay.io/jetstack/cert-manager-controller:v0.15.1 registry.cn-shenzhen.aliyuncs.com/realibox_test/cert-manager:cert-manager-controller-v0.15.1
# docker push registry.cn-shenzhen.aliyuncs.com/realibox_test/cert-manager:cert-manager-controller-v0.15.1
# docker pull quay.io/jetstack/cert-manager-webhook:v0.15.1
# docker tag quay.io/jetstack/cert-manager-webhook:v0.15.1 registry.cn-shenzhen.aliyuncs.com/realibox_test/cert-manager:cert-manager-webhook-v0.15.1
# docker push registry.cn-shenzhen.aliyuncs.com/realibox_test/cert-manager:cert-manager-webhook-v0.15.1
#现资源清单镜像 【临时镜像仓库地址为公开】
# cat cert-manager.yaml |grep image
          #image: "quay.io/jetstack/cert-manager-cainjector:v0.15.1"
          image: "registry.cn-shenzhen.aliyuncs.com/realibox_test/cert-manager:cert-manager-cainjector-v0.15.1"
          #image: "quay.io/jetstack/cert-manager-controller:v0.15.1"
          image: "registry.cn-shenzhen.aliyuncs.com/realibox_test/cert-manager:cert-manager-controller-v0.15.1"
          #image: "quay.io/jetstack/cert-manager-webhook:v0.15.1"
          image: "registry.cn-shenzhen.aliyuncs.com/realibox_test/cert-manager:cert-manager-webhook-v0.15.1"
# kubectl apply --validate=false -f cert-manager.yaml
# kubectl get pods -n cert-manager 
NAME                                           READY   STATUS    RESTARTS   AGE
cert-manager-75856bb467-fr5zz                  1/1     Running   0          1h
cert-manager-cainjector-597f5b4768-jqsvp       1/1     Running   0          1h
cert-manager-webhook-5c9f7b5f75-gnphd          1/1     Running   0          1h

验证cert-manager

# cat test-cert-manager.yaml 
##########################################################################
#Author:                     zisefeizhu
#QQ:                         2********0
#Date:                       2020-08-11
#FileName:                   test-cert-manager.yaml
#URL:                        https://www.cnblogs.com/zisefeizhu/
#Description:                The test script
#Copyright (C):              2020 All rights reserved
###########################################################################
apiVersion: v1
kind: Namespace
metadata:
  name: cert-manager-test
---
apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
  name: test-selfsigned
  namespace: cert-manager-test
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1alpha2 
kind: Certificate
metadata:
  name: selfsigned-cert
  namespace: cert-manager-test
spec:
  commonName: example.com
  secretName: selfsigned-cert-tls
  issuerRef:
    name: test-selfsigned

# kubectl apply -f test-cert-manager.yaml
  Normal  GeneratedKey  20s   cert-manager  Generated a new private key
  Normal  Requested     20s   cert-manager  Created new CertificateRequest resource "selfsigned-cert-2334779822"
  Normal  Issued        20s   cert-manager  Certificate issued successfully
  
# kubectl delete -f test-cert-manager.yaml  

通过dns配置域名证书

这里样式的是阿里云DNS操作的流程，如果需要其他平台的方法，可以自行开发，或者找已开源webhook，这是官方的例子：https://github.com/jetstack/cert-manager-webhook-example

这里用的是这个包：https://github.com/pragkent/alidns-webhook

配置alidns的webhook

# wget https://raw.githubusercontent.com/pragkent/alidns-webhook/master/deploy/bundle.yaml
# kubectl apply -f bundle.yaml
# kubectl get pods -n cert-manager  #查看webhook
NAME                                           READY   STATUS    RESTARTS   AGE
cert-manager-webhook-alidns-6b87bc8597-tc9pk   1/1     Running   2          1h

配置Issuer

cert-manager 提供了Issuer 和 ClusterIssuer 两种类型的签发机构，Issuer 只能用来签发自己所在命名空间下的证书，ClusterIssuer可以签发任意命名空间下的证书。

通过阿里云RAM创建一个账号，并授权AliyunDNSFullAccess，管理云解析（DNS）的权限，将账号的AK记下来，并通过下面的命令创建secret,这个secret用于webhook在DNS认证的时候，会向DNS解析里面写入一条txt类型的记录，认证完成后删除。

创建 alidns AccessKey Id 和 Secret

# kubectl -n cert-manager create secret generic alidns-access-key-id --from-literal=accessKeyId='xxxxxxx'
# kubectl -n cert-manager create secret generic alidns-access-key-secret --from-literal=accessKeySecret='xxxxxxx'

我这里用 ClusterIssuer 为例，创建 letsencrypt-prod.yaml 文件

# cat letsencrypt-prod.yaml 
##########################################################################
#Author:                     zisefeizhu
#QQ:                         2********0
#Date:                       2020-08-10
#FileName:                   letsencrypt-prod.yaml
#URL:                        https://www.cnblogs.com/zisefeizhu/
#Description:                The test script
#Copyright (C):              2020 All rights reserved
###########################################################################
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  labels:
    name: letsencrypt-prod
  name: letsencrypt-prod # 自定义的签发机构名称，后面会引用
spec:
  acme:
    email: [email protected] # 你的邮箱，证书快过期的时候会邮件提醒，不过我们可以设置自动续期
    solvers:
    - http01:
        ingress:
          class: nginx
    privateKeySecretRef:
      name: letsencrypt-prod # 指示此签发机构的私钥将要存储到哪个 Secret 对象中
    server: https://acme-v02.api.letsencrypt.org/directory # acme 协议的服务端，我们用Let's Encrypt

让我们看看acme协议的服务端信息

{
  "Dmrr3rQDHDQ": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",
  "keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
    "website": "https://letsencrypt.org"
  },
  "newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct",
  "newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce",
  "newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order",
  "revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert"
}

应用 yaml

# kubectl apply -f letsencrypt-prod.yaml

查看状态

# kubectl get  ClusterIssuer
NAME               READY   AGE
letsencrypt-prod   True    1h

至此，在kubernetes上利用 cert-manager 自动签发 TLS 证书 理论上部署完毕，下面进行验证！

验证

在这里我将提供两种验证方法：1. 手动签发证书 2. 自动签发证书

注意：这里存在一个大坑，请留意！

手动签发证书

# cat test-manual-cert.yaml
##########################################################################
#Author:                     zisefeizhu
#QQ:                         2********0
#Date:                       2020-08-11
#FileName:                   test-manual-cert.yaml
#URL:                        https://www.cnblogs.com/zisefeizhu/
#Description:                The test script
#Copyright (C):              2020 All rights reserved
###########################################################################
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: test-monkeyrun-net-cert
spec:
  secretName: tls-test-monkeyrun-net # 证书保存的 secret 名
  duration: 2160h # 90d
  renewBefore: 360h # 15d
  organization:
  - jetstack
  isCA: false
  keySize: 2048
  keyAlgorithm: rsa
  keyEncoding: pkcs1
  dnsNames:
  - test01.advance.realibox.com
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
    group: cert-manager.io

# kubectl apply -f test-manual-cert.yaml
certificate.cert-manager.io/test-monkeyrun-net-cert created

大坑

预警：坑要来了！！！

# kubectl get certificate   #检查是否生成证书文件
NAME                      READY   SECRET                   AGE
test-monkeyrun-net-cert   False   tls-test-monkeyrun-net   27s
# # kubectl get certificate
NAME                      READY   SECRET                   AGE
test-monkeyrun-net-cert   False   tls-test-monkeyrun-net   27s
# kubectl describe certificate test-monkeyrun-net-cert  #查看详情
Status:
  Conditions:
    Last Transition Time:  2020-08-11T02:53:06Z
    Message:               Waiting for CertificateRequest "test-monkeyrun-net-cert-1270901994" to complete
    Reason:                InProgress
    Status:                False
    Type:                  Ready
Events:
  Type    Reason     Age   From          Message
  ----    ------     ----  ----          -------
  Normal  Requested  40s   cert-manager  Created new CertificateRequest resource "test-monkeyrun-net-cert-1270901994"

这里证书生成是失败了的，原因：Waiting for CertificateRequest "test-monkeyrun-net-cert-1270901994" to complete

一直在请求，也就是说请求不到。这个问题从上周五就开始困扰着我。这也是个大坑，可看到的文档基本没有讲到

解决

查看有关证书生成的组件落到的节点：

# kubectl get pods -n cert-manager -o wide
NAME                                           READY   STATUS    RESTARTS   AGE   IP              NODE                       NOMINATED NODE   READINESS GATES
cert-manager-75856bb467-fr5zz                  1/1     Running   0          18h   192.168.0.98    cn-shenzhen.172.16.0.123              
cert-manager-cainjector-597f5b4768-jqsvp       1/1     Running   0          18h   192.168.0.101   cn-shenzhen.172.16.0.123              
cert-manager-webhook-5c9f7b5f75-gnphd          1/1     Running   0          18h   192.168.0.99    cn-shenzhen.172.16.0.123              
cert-manager-webhook-alidns-6b87bc8597-tc9pk   1/1     Running   2          17h   192.168.0.13    cn-shenzhen.172.16.0.122              

发现它们落在不同的节点上，灵光一闪，想起来了一件事：https://www.cnblogs.com/zisefeizhu/p/13262239.html 或许这个问题和自处一样呢？证书颁发者的pod与负载均衡器缠绕在不同的节点上，因此它无法通过入口与其自身进行通信。有可能哦

登陆阿里云看此kubernetes集群的外部流量引入策略

还真是的呢？根据之前对externaltrafficpolicy 的原理性了解，我有90%的把握是此处的问题，改为cluster

注：为什么我要在这里 我要登陆aliyun 点击更改而不是用命令 导出资源清单更改呢？这是因为阿里云的托管k8s有坑，这里如果用命令来改会导致nginx-ingress的lb的IP 也就是对外的公网IP发生变化，这样你的域名就全失效了因为IP变了.... 这个需要固定IP

再次测试

# kubectl delete -f test-manual-cert.yaml 
certificate.cert-manager.io "test-monkeyrun-net-cert" deleted
# kubectl apply -f test-manual-cert.yaml
certificate.cert-manager.io/test-monkeyrun-net-cert created
# kubectl get certificate
NAME                      READY   SECRET                   AGE
test-monkeyrun-net-cert   True    tls-test-monkeyrun-net   4s

！ 此问题解决，手动签发证书成功，下面进行自动签发证书！

自动签发证书

资源清单

# cat test-nginx.yaml 
##########################################################################
#Author:                     zisefeizhu
#QQ:                         2********0
#Date:                       2020-08-10
#FileName:                   test-nginx.yaml
#URL:                        https://www.cnblogs.com/zisefeizhu/
#Description:                The test script
#Copyright (C):              2020 All rights reserved
###########################################################################
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      run: test-nginx
  template:
    metadata:
      labels:
        run: test-nginx
    spec:
      containers:
      - name: test-nginx
        image: nginx
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: test-nginx
  labels:
    app: test-nginx
spec:
  ports:
  - port: 80
    protocol: TCP
    name: http
  selector:
    run: test-nginx
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test-nginx
  annotations:
    kubernetes.io/ingress.class: "nginx"
    kubernetes.io/tls-acme: "true"
    certmanager.k8s.io/cluster-issuer: "letsencrypt-prod"
spec:
  rules:
  - host: test.axxx.rxxxox.com
    http:
      paths:
      - backend:
          serviceName: test-nginx
          servicePort: 80
        path: /
  tls:
  - secretName: tls-test-monkeyrun-net
    hosts:
    - test.axxxxe.rexxxox.com

测试

# kubectl apply -f test-nginx.yaml 
deployment.apps/test-nginx created
service/test-nginx created
ingress.extensions/test-nginx created
# kubectl get pods,ingress -n test 
NAME                              READY   STATUS    RESTARTS   AGE
pod/test-nginx-6dcd7c6dc5-fd5xx   1/1     Running   0          31s

NAME                            HOSTS                       ADDRESS         PORTS     AGE
ingress.extensions/test-nginx   test.xxxce.rxxxxxxbox.com   xxxxxxx   80, 443   31s

# kubectl get secrets -n test 
NAME                                              TYPE                                  DATA   AGE
acr-credential-560b66540f01e51c18524b09ad7f575f   kubernetes.io/dockerconfigjson        1      87m
acr-credential-5dee66918cdf5d93de4aa5cd90247f73   kubernetes.io/dockerconfigjson        1      87m
acr-credential-6731ef77d88edc24b279ebf20860f30f   kubernetes.io/dockerconfigjson        1      87m
acr-credential-bab42ef118a2913b05cd8cdb95441d70   kubernetes.io/dockerconfigjson        1      87m
acr-credential-be55512166dd26eda658d0706de5a06a   kubernetes.io/dockerconfigjson        1      87m
default-token-22cwv                               kubernetes.io/service-account-token   3      87m
tls-test-monkeyrun-net                            kubernetes.io/tls                     3      10m

登陆aliyun 查看证书信息

浏览器访问

！