(2) -- Openshift 3.11 安装 istio 及相关配套组件

一、Red Hat OpenShift Service Mesh 安装概览

Red Hat OpenShift Service Mesh 安装过程中会创建两个不同的 project (namespace)

• istio-operator project (1 pod)

• istio-system project (17 pods)

你首先需要创建一个Kubernetes operator。这个operator定义并监控着一个custom resource。operator用于部署、升级、删除Service Mesh的组件。

取决于你如何定义这个custom resource文件，当安装Service Mesh时，你可以选择安装以下一个或多个以下组件

• Istio - 基于开源 Istio 项目, 让你能连接、安全保护、控制、观察组成你的应用的各个微服务

• Jaeger - 基于开源 Jaeger 项目, 让你能在一个复杂的分布式系统中，通过调用链跟踪来监控和排查事务问题

• Kiali - 基于开源 Kiali 项目, Kiali 为你的 Service Mesh 提供了可观察的特性，使用 Kiali 能让你可视化地配置、监控流量、可视化地分析调用链。

• Launcher - 基于开源 fabric8 社区, fabric8是一个开源集成开发平台，为基于Kubernetes和Jenkins的微服务提供持续发布。

在 operator 的安装过程中，会创建一个 Ansible job， Ansible job 会启动一个 Ansible playbook，Ansible playbook 会自动执行以下步骤，并配置好各个组件。

1.1 创建 istio-system namespace

1.2 创建 openshift-ansible-istio-installer-job，它会安装以下组件:

Istio components: istio-citadel、istio-egressgateway、istio-galley、istio-ingressgateway、istio-pilot、istio-policy、istio-sidecar-injector、istio-statsd-prom-bridge、istio-telemetry

Elasticsearch

Grafana

Jaeger components: jaeger-agent、jaeger-collector、jaeger-query

Kiali components (如果在 custom resource 文件中配置了 Kiali 的话): Kiali

Prometheus

1.3 执行 launcher 的配置任务 (如果在 custom resource 文件中配置了 launcher 的话):

1.3.1 创建一个 devex 项目并安装 Fabric8 launcher 到 devex project中.

1.3.2 将集群管理角色添加到在自定义资源文件的启动程序参数中指定的openshift容器平台用户。

二、安装前准备

2.1 如果离线安装需要下载的镜像包括

docker pull registry.access.redhat.com/openshift-istio-tech-preview/istio-operator:0.6.0
docker pull registry.access.redhat.com/openshift-istio-tech-preview/openshift-ansible:0.6.0
docker pull registry.access.redhat.com/openshift-istio-tech-preview/citadel:0.6.0
docker pull registry.access.redhat.com/openshift-istio-tech-preview/proxyv2:0.6.0
docker pull registry.access.redhat.com/openshift-istio-tech-preview/pilot:0.6.0
docker pull registry.access.redhat.com/openshift-istio-tech-preview/mixer:0.6.0
docker pull registry.access.redhat.com/openshift-istio-tech-preview/galley:0.6.0
docker pull registry.access.redhat.com/openshift-istio-tech-preview/sidecar-injector:0.6.0
docker pull registry.access.redhat.com/openshift-istio-tech-preview/proxy-init:0.6.0
docker pull registry.access.redhat.com/openshift-istio-tech-preview/kiali:0.11.0
docker pull registry.access.redhat.com/distributed-tracing-tech-preview/jaeger-elasticsearch:5.6.10
docker pull registry.access.redhat.com/distributed-tracing-tech-preview/jaeger-agent:1.8.1
docker pull registry.access.redhat.com/distributed-tracing-tech-preview/jaeger-collector:1.8.1
docker pull registry.access.redhat.com/distributed-tracing-tech-preview/jaeger-query:1.8.1
docker pull grafana/grafana:5.4.2
docker pull docker.io/prom/prometheus:v2.3.1

2.2 更新 Openshift 各节点配置

2.2.1 在每台机器新建立一个/etc/sysctl.d/99-elasticsearch.conf文件，添加

vm.max_map_count = 262144

2.2.2 在每个 Node 上执行命令

$ sysctl vm.max_map_count=262144

三、安装 Service Mesh

3.1 创建 CUSTOM RESOURCE 文件

包含所有组件的 istio-installation.yaml：

apiVersion: "istio.openshift.com/v1alpha1"
kind: "Installation"
metadata:
  name: "istio-installation"
spec:
  deployment_type: openshift
  istio:
    authentication: true
    community: false
    prefix: openshift-istio-tech-preview/ #镜像前缀
    version: 0.6.0 #镜像tag
  jaeger:
    prefix: distributed-tracing-tech-preview/
    version: 1.8.1
    elasticsearch_memory: 1Gi
  kiali:
    username: username #kiali管理界面的登录名
    password: password #kiali管理界面的登录密码
    prefix: openshift-istio-tech-preview/
    version: 0.11.0
  launcher: #内网中干掉
    openshift:
      user: user
      password: password
    github:
      username: username
      token: token
    catalog:
      filter: booster.mission.metadata.istio
      branch: v71
      repo: https://github.com/fabric8-launcher/launcher-booster-catalog.git

注：如果在内网环境安装，由于无法访问外网的 Github，可以不用安装 launcher 组件，在 custom resource 文件中去掉 launcher 相关的所有配置即可。

最小化的 istio-installation.yaml

apiVersion: "istio.openshift.com/v1alpha1"
kind: "Installation"
metadata:
  name: "istio-installation"

3.2 安装 operator

Service Mesh 安装过程中引入了 kubernetes operator 来管理 istio-system namespace 内 control plane 的安装。此 operator 定义了监视 control plane 的部署、更新和删除相关的自定义资源。

istio_product_operator_template.yaml :

apiVersion: v1
kind: Template
metadata:
  name: istio-operator-job
parameters:
- displayName: Master Public URL
  description: The public URL for master
  name: OPENSHIFT_ISTIO_MASTER_PUBLIC_URL
  value: https://127.0.0.1:8443
- displayName: OpenShift Release
  description: The version of the OpenShift release.
  name: OPENSHIFT_RELEASE
  value: v3.11.0
  required: true
- displayName: Istio Operator Namespace
  description: The namespace for the Istio operator
  name: OPENSHIFT_ISTIO_OPERATOR_NAMESPACE
  value: istio-operator
  required: true
- displayName: Default Prefix
  description: The default image prefix for istio deployments
  name: OPENSHIFT_ISTIO_PREFIX
  value: openshift-istio-tech-preview/
- displayName: Default Version
  description: The default image version for istio deployments
  name: OPENSHIFT_ISTIO_VERSION
  value: 0.6.0
- displayName: Default Deployment Type
  description: The default deployment type for istio deployments
  name: OPENSHIFT_DEPLOYMENT_TYPE
  value: openshift
objects:
- kind: CustomResourceDefinition
  apiVersion: apiextensions.k8s.io/v1beta1
  metadata:
    name: installations.istio.openshift.com
  spec:
    group: istio.openshift.com
    names:
      kind: Installation
      plural: installations
      singular: installation
    scope: Namespaced
    version: v1alpha1
- kind: Role
  apiVersion: rbac.authorization.k8s.io/v1
  metadata:
    name: istio-operator
  rules:
  - apiGroups:
    - istio.openshift.com
    resources:
    - "*"
    verbs:
    - "*"
  - apiGroups:
    - ""
    resources:
    - pods
    - services
    - endpoints
    - persistentvolumeclaims
    - events
    - configmaps
    - secrets
    - securitycontextconstraints
    verbs:
    - "*"
  - apiGroups:
    - apps
    resources:
    - deployments
    - daemonsets
    - replicasets
    - statefulsets
    verbs:
    - "*"
- kind: RoleBinding
  apiVersion: rbac.authorization.k8s.io/v1
  metadata:
    name: default-account-istio-operator
  subjects:
  - kind: ServiceAccount
    namespace: ${OPENSHIFT_ISTIO_OPERATOR_NAMESPACE}
    name: default
  roleRef:
    kind: Role
    name: istio-operator
    apiGroup: rbac.authorization.k8s.io
- kind: ClusterRoleBinding
  apiVersion: rbac.authorization.k8s.io/v1
  metadata:
    name: default-account-istio-operator-cluster-role-binding
  subjects:
  - kind: ServiceAccount
    namespace: ${OPENSHIFT_ISTIO_OPERATOR_NAMESPACE}
    name: default
  roleRef:
    kind: ClusterRole
    name: cluster-admin
    apiGroup: rbac.authorization.k8s.io
- kind: Deployment
  apiVersion: apps/v1
  metadata:
    name: istio-operator
    namespace: ${OPENSHIFT_ISTIO_OPERATOR_NAMESPACE}
  spec:
    replicas: 1
    selector:
      matchLabels:
        name: istio-operator
    template:
      metadata:
        labels:
          name: istio-operator
      spec:
        containers:
          - name: istio-operator
            image: ${OPENSHIFT_ISTIO_PREFIX}istio-operator:${OPENSHIFT_ISTIO_VERSION}
            ports:
            - containerPort: 60000
              name: metrics
            command:
            - istio-operator
            args:
            - "--release=${OPENSHIFT_RELEASE}"
            - "--masterPublicURL=${OPENSHIFT_ISTIO_MASTER_PUBLIC_URL}"
            - "--istioPrefix=${OPENSHIFT_ISTIO_PREFIX}"
            - "--istioVersion=${OPENSHIFT_ISTIO_VERSION}"
            - "--deploymentType=${OPENSHIFT_DEPLOYMENT_TYPE}"
            imagePullPolicy: IfNotPresent
            env:
              - name: WATCH_NAMESPACE
                valueFrom:
                  fieldRef:
                    fieldPath: metadata.namespace
              - name: OPERATOR_NAME
                value: "istio-operator"

3.3 安装 operator

以下命令将安装 Service Mesh operator 到 Openshift 容器平台中。在集群任意一个节点执行。

$ oc new-project istio-operator
$ oc new-app -f istio_product_operator_template.yaml --param=OPENSHIFT_ISTIO_MASTER_PUBLIC_URL=

3.4 验证 operator 是否安装成功

先前的命令创建了一个 deployment 资源 到 istio-operator 这个 project 里，同时运行了 operator 通过 custom resource 来管理 Red Hat OpenShift Service Mesh 中 control plane 的状态。

为了验证 operator 是否正确安装，执行以下命令观察日志：

$ oc logs -n istio-operator $(oc -n istio-operator get pods -l name=istio-operator --output=jsonpath={.items..metadata.name})

执行以上命令后，如果出现类似以下的结果，说明安装正确：

time="2018-08-31T17:42:39Z" level=info msg="Go Version: go1.9.4"
time="2018-08-31T17:42:39Z" level=info msg="Go OS/Arch: linux/amd64"
time="2018-08-31T17:42:39Z" level=info msg="operator-sdk Version: 0.0.5+git"
time="2018-08-31T17:42:39Z" level=info msg="Metrics service istio-operator created"
time="2018-08-31T17:42:39Z" level=info msg="Watching resource istio.openshift.com/v1alpha1, kind Installation, namespace istio-operator, resyncPeriod 0"
time="2018-08-31T17:42:39Z" level=info msg="Installing istio for Installation istio-installation"

3.5 部署 control plane

$ oc create -f istio-installation.yaml -n istio-operator #istio-installation.yaml的内容见上

观察安装过程中 pods 的状态：

$ oc get pods -n istio-system -w

3.6 验证 control plane 的安装

$ oc get pods -n istio-system

如果出现和下面一样的内容，说明安装成功：

NAME                                          READY     STATUS      RESTARTS   AGE
elasticsearch-0                               1/1       Running     0          2m
grafana-6d5c5477-k7wrh                        1/1       Running     0          2m
istio-citadel-6f9c778bb6-q9tg9                1/1       Running     0          3m
istio-egressgateway-957857444-2g84h           1/1       Running     0          3m
istio-galley-c47f5dffc-dm27s                  1/1       Running     0          3m
istio-ingressgateway-7db86747b7-s2dv9         1/1       Running     0          3m
istio-pilot-5646d7786b-rh54p                  2/2       Running     0          3m
istio-policy-7d694596c6-pfdzt                 2/2       Running     0          3m
istio-sidecar-injector-57466d9bb-4cjrs        1/1       Running     0          3m
istio-statsd-prom-bridge-7f44bb5ddb-6vx7n     1/1       Running     0          3m
istio-telemetry-7cf7b4b77c-p8m2k              2/2       Running     0          3m
jaeger-agent-5mswn                            1/1       Running     0          2m
jaeger-collector-9c9f8bc66-j7kjv              1/1       Running     0          2m
jaeger-query-fdc6dcd74-99pnx                  1/1       Running     0          2m
kiali-779bcc566f-qqt65                        1/1       Running     0          2m
openshift-ansible-istio-installer-job-f8n9g   0/1       Completed   0          7m
prometheus-84bd4b9796-2vcpc                   1/1       Running     0          3m

如果在 custom resource 文件中配置了 launcher，查看 devex project 中的 容器状态会出现下面内容：

$ oc get pods -n devex

NAME                          READY     STATUS    RESTARTS   AGE
configmapcontroller-1-8rr6w   1/1       Running   0          1m
launcher-backend-2-2wg86      1/1       Running   0          1m
launcher-frontend-2-jxjsd     1/1       Running   0          1m

四、在 Openshift Service Mesh 部署应用需要满足的要求

4.1 为应用的 service account 配置 SCC （Security Context Constraints，安全上下文 ）

oc adm policy add-scc-to-user anyuid -z  -n 
oc adm policy add-scc-to-user privileged -z  -n 

4.2 更新 Openshift Master节点的配置

保证以下操作在每个 Openshift Container Platform 的 master 节点中都执行：

4.2.1 切换目录到包含 master configuration 文件的路径下(for example, /etc/origin/master/master-config.yaml).

$ cd  /etc/origin/master

4.2.2 创建master-config.patch，包含以下内容：

admissionConfig:
  pluginConfig:
    MutatingAdmissionWebhook:
      configuration:
        apiVersion: apiserver.config.k8s.io/v1alpha1
        kubeConfigFile: /dev/null
        kind: WebhookAdmission
    ValidatingAdmissionWebhook:
      configuration:
        apiVersion: apiserver.config.k8s.io/v1alpha1
        kubeConfigFile: /dev/null
        kind: WebhookAdmission

4.2.3 在该目录下执行以下命令，来修改 master-config.yaml 文件：

$ cp -p master-config.yaml master-config.yaml.prepatch
$ oc ex config patch master-config.yaml.prepatch -p "$(cat master-config.patch)" > master-config.yaml
$ /usr/local/bin/master-restart api && /usr/local/bin/master-restart controllers

4.3 如何为应用配置 sidecar 的自动注入

相当简单，只需要在应用的部署yaml文件中，为 annotation 添加 sidecar.istio.io/inject 属性，并置为 true

例如：

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: sleep
spec:
  replicas: 1
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: "true"
      labels:
        app: sleep
    spec:
      containers:
      - name: sleep
        image: tutum/curl
        command: ["/bin/sleep","infinity"]
        imagePullPolicy: IfNotPresent

至此，Openshift 3.11 环境下安装 istio 及相关配套组件的步骤全部结束，下一章进入 Service Mesh 示例工程 -- bookInfo 的安装教程

参考：https://docs.openshift.com/container-platform/3.11/servicemesh-install/servicemesh-install.html