k8s部署持续集成环境

k8s部署持续集成环境

安装前的准备：
关闭防火墙
关闭selinux
安装docker-ce
1、安装harbor参考前面笔记
2、安装并配置git，参考前面笔记
3、在git这台部署nfs服务器，并且在各个节点安装nfs-utils客户端
4、创建nfs-client-provisioner客户端
cat class.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: managed-nfs-storage
provisioner: fuseim.pri/ifs # or choose another name, must match deployment's env PROVISIONER_NAME'
parameters:
archiveOnDelete: "true"

cat deployment.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: nfs-client-provisioner

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: nfs-client-provisioner
spec:
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
app: nfs-client-provisioner
spec:
serviceAccountName: nfs-client-provisioner
containers:

• name: nfs-client-provisioner
  image: lizhenliang/nfs-client-provisioner:latest
  volumeMounts:
  • name: nfs-client-root
    mountPath: /persistentvolumes
    env:
  • name: PROVISIONER_NAME
    value: fuseim.pri/ifs
  • name: NFS_SERVER
    value: 192.168.31.64
  • name: NFS_PATH
    value: /ifs/kubernetes
    volumes:

• name: nfs-client-root
  nfs:
  server: 192.168.31.64
  path: /ifs/kubernetes
  cat rbac.yaml
  kind: ServiceAccount
  apiVersion: v1
  metadata:
  name: nfs-client-provisioner

  kind: ClusterRole
  apiVersion: rbac.authorization.k8s.io/v1
  metadata:
  name: nfs-client-provisioner-runner
  rules:

  • apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  • apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  • apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]

  • apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]

    kind: ClusterRoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
    name: run-nfs-client-provisioner
    subjects:

  • kind: ServiceAccount
    name: nfs-client-provisioner
    namespace: default
    roleRef:
    kind: ClusterRole
    name: nfs-client-provisioner-runner
    apiGroup: rbac.authorization.k8s.io

    kind: Role
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
    name: leader-locking-nfs-client-provisioner
    rules:

  • apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]

    kind: RoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
    name: leader-locking-nfs-client-provisioner
    subjects:

  • kind: ServiceAccount
    name: nfs-client-provisioner

    replace with namespace where provisioner is deployed

    namespace: default
    roleRef:
    kind: Role
    name: leader-locking-nfs-client-provisioner
    apiGroup: rbac.authorization.k8s.io
    5、部署jenkins服务器，前提是需要部署k8s的core-dns（安装coredns查看前面笔记），否则没法做解析就没法安装插件
    cat ingress.yml
    apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
    name: jenkins
    annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    kubernetes.io/tls-acme: "true"

    如果上传插件超出默认会报"413 Request Entity Too Large", 增加 client_max_body_size

    nginx.ingress.kubernetes.io/proxy-body-size: 50m
    nginx.ingress.kubernetes.io/proxy-request-buffering: "off"

    nginx-ingress controller版本小于 0.9.0.beta-18 的配置

    ingress.kubernetes.io/ssl-redirect: "true"
    ingress.kubernetes.io/proxy-body-size: 50m
    ingress.kubernetes.io/proxy-request-buffering: "off"
    spec:
    rules:

  • host: jenkins.example.com
    http:
    paths:

    • path: /
      backend:
      serviceName: jenkins
      servicePort: 80
      cat rbac.yml

      创建名为jenkins的ServiceAccount

      apiVersion: v1
      kind: ServiceAccount
      metadata:
      name: jenkins

---

创建名为jenkins的Role，授予允许管理API组的资源Pod

kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: jenkins
rules:

• apiGroups: [""]
  resources: ["pods"]
  verbs: ["create","delete","get","list","patch","update","watch"]
• apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["create","delete","get","list","patch","update","watch"]
• apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get","list","watch"]
• apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get"]

---

将名为jenkins的Role绑定到名为jenkins的ServiceAccount

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: jenkins
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: jenkins
subjects:

• kind: ServiceAccount
  name: jenkins

cat service.yml
apiVersion: v1
kind: Service
metadata:
name: jenkins
spec:
selector:
name: jenkins
type: NodePort
ports:

  name: http
  port: 80
  targetPort: 8080
  protocol: TCP
  nodePort: 30006
-
  name: agent
  port: 50000
  protocol: TCP

cat statefulset.yml
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: jenkins
labels:
name: jenkins
spec:
serviceName: jenkins
replicas: 1
updateStrategy:
type: RollingUpdate
template:
metadata:
name: jenkins
labels:
name: jenkins
spec:
terminationGracePeriodSeconds: 10
serviceAccountName: jenkins
containers: (如果出现dns没法解析的情况，需要在container这 一行上面加一行dnsPolicy: Default)

• name: jenkins
  image: jenkins/jenkins
  imagePullPolicy: Always
  ports:
  • containerPort: 8080
  • containerPort: 50000
    resources:
    limits:
    cpu: 1
    memory: 1Gi
    requests:
    cpu: 0.5
    memory: 500Mi
    env:
  • name: LIMITS_MEMORY
    valueFrom:
    resourceFieldRef:
    resource: limits.memory
    divisor: 1Mi
  • name: JAVA_OPTS
    value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85
    volumeMounts:
  • name: jenkins-home
    mountPath: /var/jenkins_home
    livenessProbe:
    httpGet:
    path: /login
    port: 8080
    initialDelaySeconds: 60
    timeoutSeconds: 5
    failureThreshold: 12
    readinessProbe:
    httpGet:
    path: /login
    port: 8080
    initialDelaySeconds: 60
    timeoutSeconds: 5
    failureThreshold: 12
    securityContext:
    fsGroup: 1000
    volumeClaimTemplates:
    • metadata:
      name: jenkins-home
      spec:
      storageClassName: "managed-nfs-storage"
      accessModes: [ "ReadWriteOnce" ]
      resources:
      requests:
      storage: 1Gi
      6、使用这几个yaml文件启动安装jenkins
      PS：当提示以下出错时
      'FailedCreate' create Pod jenkins-0 in StatefulSet jenkins failed error: pods "jenkins-0" is forbidden: pod.Spec.SecurityContext.FSGroup is forbidden
      修改/opt/kubernetes/cfg/kube-apiserver，删除里面的安全字段SecurityContext
      7、使用浏览器访问http://10.1.2.190:30006进行安装(使用kubectl get svc -o wide查看jenkins当前在190这个pod运行)
      8、安装时不选择任何插件，使用手动安装插件，安装git和kubernetes插件