基于K8s 1.15部署kubeflow 0.7.0 by kfctl

一、环境介绍

centos7系统机器三台：
192.168.3.100: kubernetes master
192.168.3.101: kubernetes node1
192.168.3.102: kubernetes node2

二、系统要求

官方说明
https://www.kubeflow.org/docs/started/k8s/overview/#minimum-system-requirements
Kubernetes集群必须满足以下最低要求：
集群必须至少包含一个工作节点，且最少为：
4个CPU
50 GB的存储空间
12 GB内存
兼容性说明：

三、下载docker镜像

使用kfctl默认安装的话，很多镜像都需要从gcr.io下载，但是国内无法访问gcr.io，因此需要从国内源下载需要的镜像
kubeflow 0.7.0需要的镜像整理如下

docker pull gcr.azk8s.cn/kubeflow-images-public/ingress-setup:latest
docker pull gcr.azk8s.cn/kubeflow-images-public/admission-webhook:v20190520-v0-139-gcee39dbc-dirty-0d8f4c
docker pull gcr.azk8s.cn/kubeflow-images-public/kubernetes-sigs/application:1.0-beta
docker pull gcr.azk8s.cn/kubeflow-images-public/centraldashboard@sha256:4299297b8390599854aa8f77e9eb717db684b32ca9a94a0ab0e73f3f73e5d8b5
docker pull gcr.azk8s.cn/kubeflow-images-public/jupyter-web-app:9419d4d
docker pull gcr.azk8s.cn/kubeflow-images-public/katib/v1alpha3/katib-controller:v0.7.0
docker pull gcr.azk8s.cn/kubeflow-images-public/katib/v1alpha3/katib-manager:v0.7.0
docker pull gcr.azk8s.cn/kubeflow-images-public/katib/v1alpha3/katib-ui:v0.7.0
docker pull gcr.azk8s.cn/kubebuilder/kube-rbac-proxy:v0.4.0
docker pull gcr.azk8s.cn/kubeflow-images-public/metadata:v0.1.11
docker pull gcr.azk8s.cn/ml-pipeline/envoy:metadata-grpc
docker pull gcr.azk8s.cn/tfx-oss-public/ml_metadata_store_server:0.15.1
docker pull gcr.azk8s.cn/kubeflow-images-public/metadata-frontend:v0.1.8
docker pull gcr.azk8s.cn/ml-pipeline/api-server:0.1.31
docker pull gcr.azk8s.cn/ml-pipeline/visualization-server:0.1.27
docker pull gcr.azk8s.cn/ml-pipeline/persistenceagent:0.1.31
docker pull gcr.azk8s.cn/ml-pipeline/scheduledworkflow:0.1.31
docker pull gcr.azk8s.cn/ml-pipeline/frontend:0.1.31
docker pull gcr.azk8s.cn/ml-pipeline/viewer-crd-controller:0.1.31
docker pull gcr.azk8s.cn/kubeflow-images-public/notebook-controller@sha256:6490f737000bd1d2520ac4b8cbde2b09749cdb291b1967ddda95d05131db49db
docker pull gcr.azk8s.cn/kubeflow-images-public/profile-controller@sha256:9b834c9d44f9e4e8d5b9f8f01645afb84d5b3debcb177f3e49eac8ee46ae239d
docker pull gcr.azk8s.cn/kubeflow-images-public/kfam@sha256:3b0d4be7e59a3fa5ed1d80dccc832312caa94f3b2d36682524d3afc4e45164f0
docker pull gcr.azk8s.cn/kubeflow-images-public/pytorch-operator:v0.7.0
docker pull gcr.azk8s.cn/kubeflow-images-public/tf_operator:v0.7.0
docker pull gcr.azk8s.cn/kfserving/kfserving-controller@sha256:180d06026c4dd6c2d3ce4748efc896751b9bb6108b67a9eaa0e50158d6e10f1e
docker pull gcr.azk8s.cn/google_containers/spartakus-amd64:v1.1.0

docker pull docker.io/argoproj/workflow-controller:v2.3.0
docker pull docker.io/tensorflow/tensorflow:1.8.0
docker pull docker.io/seldonio/seldon-core-operator:0.4.1
docker pull docker.io/metacontroller/metacontroller:v0.3.0
docker pull docker.io/argoproj/argoui:v2.3.0
docker pull minio/minio:RELEASE.2018-02-09T22-40-05Z
docker pull mysql:5.6
docker pull mysql:8.0.3

重新打tag

docker tag 8fb2e7d74d1e gcr.azk8s.cn/kubeflow-images-public/centraldashboard
docker tag f4f4177b0a09 gcr.azk8s.cn/kubeflow-images-public/profile-controller
docker tag 89782fa49fbc gcr.azk8s.cn/kubeflow-images-public/kfam
docker tag 0359b650ea45 gcr.azk8s.cn/kubeflow-images-public/notebook-controller
docker tag 221420aa74f7 gcr.azk8s.cn/kfserving/kfserving-controller

docker tag gcr.azk8s.cn/kubeflow-images-public/ingress-setup:latest gcr.io/kubeflow-images-public/ingress-setup:latest
docker tag gcr.azk8s.cn/kubeflow-images-public/admission-webhook:v20190520-v0-139-gcee39dbc-dirty-0d8f4c gcr.io/kubeflow-images-public/admission-webhook:v20190520-v0-139-gcee39dbc-dirty-0d8f4c
docker tag gcr.azk8s.cn/kubeflow-images-public/kubernetes-sigs/application:1.0-beta gcr.io/kubeflow-images-public/kubernetes-sigs/application:1.0-beta
docker tag gcr.azk8s.cn/kubeflow-images-public/centraldashboard:latest gcr.io/kubeflow-images-public/centraldashboard:latest
docker tag gcr.azk8s.cn/kubeflow-images-public/jupyter-web-app:9419d4d gcr.io/kubeflow-images-public/jupyter-web-app:9419d4d
docker tag gcr.azk8s.cn/kubeflow-images-public/katib/v1alpha3/katib-controller:v0.7.0 gcr.io/kubeflow-images-public/katib/v1alpha3/katib-controller:v0.7.0
docker tag gcr.azk8s.cn/kubeflow-images-public/katib/v1alpha3/katib-manager:v0.7.0 gcr.io/kubeflow-images-public/katib/v1alpha3/katib-manager:v0.7.0
docker tag gcr.azk8s.cn/kubeflow-images-public/katib/v1alpha3/katib-ui:v0.7.0 gcr.io/kubeflow-images-public/katib/v1alpha3/katib-ui:v0.7.0
docker tag gcr.azk8s.cn/kubebuilder/kube-rbac-proxy:v0.4.0 gcr.io/kubebuilder/kube-rbac-proxy:v0.4.0
docker tag gcr.azk8s.cn/kubeflow-images-public/metadata:v0.1.11 gcr.io/kubeflow-images-public/metadata:v0.1.11
docker tag gcr.azk8s.cn/ml-pipeline/envoy:metadata-grpc gcr.io/ml-pipeline/envoy:metadata-grpc
docker tag gcr.azk8s.cn/tfx-oss-public/ml_metadata_store_server:0.15.1 gcr.io/tfx-oss-public/ml_metadata_store_server:0.15.1
docker tag gcr.azk8s.cn/kubeflow-images-public/metadata-frontend:v0.1.8 gcr.io/kubeflow-images-public/metadata-frontend:v0.1.8
docker tag gcr.azk8s.cn/ml-pipeline/api-server:0.1.31 gcr.io/ml-pipeline/api-server:0.1.31
docker tag gcr.azk8s.cn/ml-pipeline/visualization-server:0.1.27 gcr.io/ml-pipeline/visualization-server:0.1.27
docker tag gcr.azk8s.cn/ml-pipeline/persistenceagent:0.1.31 gcr.io/ml-pipeline/persistenceagent:0.1.31
docker tag gcr.azk8s.cn/ml-pipeline/scheduledworkflow:0.1.31 gcr.io/ml-pipeline/scheduledworkflow:0.1.31
docker tag gcr.azk8s.cn/ml-pipeline/frontend:0.1.31 gcr.io/ml-pipeline/frontend:0.1.31
docker tag gcr.azk8s.cn/ml-pipeline/viewer-crd-controller:0.1.31 gcr.io/ml-pipeline/viewer-crd-controller:0.1.31
docker tag gcr.azk8s.cn/kubeflow-images-public/notebook-controller:latest gcr.io/kubeflow-images-public/notebook-controller:latest
docker tag gcr.azk8s.cn/kubeflow-images-public/profile-controller:latest gcr.io/kubeflow-images-public/profile-controller:latest
docker tag gcr.azk8s.cn/kubeflow-images-public/kfam:latest gcr.io/kubeflow-images-public/kfam:latest
docker tag gcr.azk8s.cn/kubeflow-images-public/pytorch-operator:v0.7.0 gcr.io/kubeflow-images-public/pytorch-operator:v0.7.0
docker tag gcr.azk8s.cn/kubeflow-images-public/tf_operator:v0.7.0 gcr.io/kubeflow-images-public/tf_operator:v0.7.0
docker tag gcr.azk8s.cn/kfserving/kfserving-controller:latest gcr.io/kfserving/kfserving-controller:latest
docker tag gcr.azk8s.cn/google_containers/spartakus-amd64:v1.1.0 gcr.io/google_containers/spartakus-amd64:v1.1.0

删除gcr.azk8s.cn的镜像

docker rmi gcr.azk8s.cn/kubeflow-images-public/ingress-setup:latest
docker rmi gcr.azk8s.cn/kubeflow-images-public/admission-webhook:v20190520-v0-139-gcee39dbc-dirty-0d8f4c
docker rmi gcr.azk8s.cn/kubeflow-images-public/kubernetes-sigs/application:1.0-beta
docker rmi gcr.azk8s.cn/kubeflow-images-public/centraldashboard:latest
docker rmi gcr.azk8s.cn/kubeflow-images-public/jupyter-web-app:9419d4d
docker rmi gcr.azk8s.cn/kubeflow-images-public/katib/v1alpha3/katib-controller:v0.7.0
docker rmi gcr.azk8s.cn/kubeflow-images-public/katib/v1alpha3/katib-manager:v0.7.0
docker rmi gcr.azk8s.cn/kubeflow-images-public/katib/v1alpha3/katib-ui:v0.7.0
docker rmi gcr.azk8s.cn/kubebuilder/kube-rbac-proxy:v0.4.0
docker rmi gcr.azk8s.cn/kubeflow-images-public/metadata:v0.1.11
docker rmi gcr.azk8s.cn/ml-pipeline/envoy:metadata-grpc
docker rmi gcr.azk8s.cn/tfx-oss-public/ml_metadata_store_server:0.15.1
docker rmi gcr.azk8s.cn/kubeflow-images-public/metadata-frontend:v0.1.8
docker rmi gcr.azk8s.cn/ml-pipeline/api-server:0.1.31
docker rmi gcr.azk8s.cn/ml-pipeline/visualization-server:0.1.27
docker rmi gcr.azk8s.cn/ml-pipeline/persistenceagent:0.1.31
docker rmi gcr.azk8s.cn/ml-pipeline/scheduledworkflow:0.1.31
docker rmi gcr.azk8s.cn/ml-pipeline/frontend:0.1.31
docker rmi gcr.azk8s.cn/ml-pipeline/viewer-crd-controller:0.1.31
docker rmi gcr.azk8s.cn/kubeflow-images-public/notebook-controller:latest
docker rmi gcr.azk8s.cn/kubeflow-images-public/profile-controller:latest
docker rmi gcr.azk8s.cn/kubeflow-images-public/kfam:latest
docker rmi gcr.azk8s.cn/kubeflow-images-public/pytorch-operator:v0.7.0
docker rmi gcr.azk8s.cn/kubeflow-images-public/tf_operator:v0.7.0
docker rmi gcr.azk8s.cn/kfserving/kfserving-controller:latest
docker rmi gcr.azk8s.cn/google_containers/spartakus-amd64:v1.1.0

四、下载kfctl v0.7.0

从Kubeflow版本页面下载kfctl v0.7.0版本
https://github.com/kubeflow/kubeflow/releases/tag/v0.7.0

解压，我的环境解压到/ssd目录

五、安装kubeflow 0.7.0

[root@master ~]#export PATH=$PATH:"/ssd"
[root@master ~]#export KF_NAME= kubeflow
[root@master ~]#export BASE_DIR=/ssd
[root@master ~]#export KF_DIR=${BASE_DIR}/${KF_NAME}
[root@master ~]#export CONFIG_URI="https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_k8s_istio.0.7.0.yaml"
[root@master ~]#mkdir -p ${KF_DIR}
[root@master ~]#cd ${KF_DIR}
[root@master ~]#kfctl apply -V -f ${CONFIG_URI}

也可以把kfctl_k8s_istio.0.7.0.yaml文件下载下来，比如

[root@ssd]#wget https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_k8s_istio.0.7.0.yaml

安装的时候执行

[root@ssd]# kfctl apply -V -f kfctl_k8s_istio.0.7.0.yaml

六、访问kubeflow

根据官方说明，kubeflow通过istio-ingressgateway访问Kubeflow Dashboard
https://www.kubeflow.org/docs/started/k8s/kfctl-k8s-istio/

查看istio-ingressgateway的服务配置

[root@master ~]# kubectl get pod -n istio-system  -o wide | grep istio-ingressgateway

可以看到istio-ingressgateway在node1，但没有配置外网地址

设置外网地址，修改istio-ingressgateway的服务配置

[root@master ~]# kubectl edit svc istio-ingressgateway -n istio-system

增加externalIPs，填node1的IP（如果有内外网IP，填写内网IP）
修改Type，将ClusterIP的值NodePort修改为LoadBalancer
:wq保存退出


访问地址http:// EXTERNAL-IP: 31380 (上图可以看到80端口映射到了31380)
EXTERNAL-IP为任意node的EXTERNAL-IP

七、单独访问pipeline

查看ml-pipeline-ui的服务配置

[root@master ~]# kubectl -n kubeflow get svc ml-pipeline-ui

修改ml-pipeline-ui的服务配置

[root@master ~]# kubectl -n kubeflow edit svc ml-pipeline-ui

增加externalIPs，填node1的IP（如果有内外网IP，填写内网IP）
修改Type，将ClusterIP修改为LoadBalancer
保存退出

[root@master ~]# kubectl -n kubeflow get svc ml-pipeline-ui

访问地址http:// EXTERNAL-IP:32172(上图80端口映射到了32172)

八、删除kubeflow

[root@master ~]#export PATH=$PATH:"/ssd"
[root@master ~]#export KF_NAME=mlopskubeflow
[root@master ~]#export BASE_DIR=/ssd/kubeflow
[root@master ~]#export KF_DIR=${BASE_DIR}/${KF_NAME}
[root@master ~]#export CONFIG_FILE=${KF_DIR}/kfctl_k8s_istio.0.7.0.yaml
[root@master ~]#kfctl delete -f ${CONFIG_FILE}

如果还要使用以上方式去安装kubeflow，需要删除掉原有的istio-system、knative-serving下的所有资源

[root@master ~]#kubectl delete ns istio-system
[root@master ~]#kubectl delete ns knative-serving

九、问题集锦

待续《基于K8s 1.15部署kubeflow 0.7.0 by kfctl–问题集锦》