要求
懂点K8S，会简单的命令，需要梯子。
简述
这里主要是按照官网的流程,根据要修改的地方,再一步步安装spinnaker.
[官网安装文档链接]
Step 1: 安装Halyard
此处选择使用locally而非docker安装方式.
1.1 环境准备
- 虚机一台：2C4G
- 操作系统：Ubuntu16+
1.2 设置代理
此步骤服务器需要先设置代理,需要先有一台代理服务器在GFW后,我这边使用的是本地Shadowsocks，配置为远程可以使用方式
ShadowsocksX-NG的HTTP代理设置
通过ifconfig获得到本机的IP为，比如是192.168.0.9
Ubuntu服务器设置如下:
export https_proxy=http://192.168.0.9:1087
export http_proxy=http://192.168.0.9:1087
export no_proxy="localhost"
根据实际情况修改即可。
通过curl google.com 测试是否生效！
1.3安装Halyard
创建一个用户 adduser spinnaker
下载脚本和安装,这里需要输入安装(运行Halyard程序)的用户,可以使用root用户运行
curl -O https://raw.githubusercontent.com/spinnaker/halyard/master/install/debian/InstallHalyard.sh
bash InstallHalyard.sh
确认安装,这里的安装的用户是spinnaker用户,所以才去到它的home目录的source它的.bashrc,这里不同用户不用home路径
hal -v
. /home/spinnaker/.bashrc
1.4设置java代理
执行这个步骤之前,Halyard应该已经启动且能正确显示hal -v的返回.在这个步骤执行之前,可以把上面的代理设置清除(由于上述配置是临时生效，退出终端重新进入即可)，因为Halyard的安装包已经能从google下载回来安装了。
为java设置代理,因为后期的设置Halyard需要在墙外获取信息.
修改halyard的DEFAULT_JVM_OPTS参数
vim /opt/halyard/bin/halyard
DEFAULT_JVM_OPTS='"-Djava.security.egd=file:/dev/./urandom" "-Dspring.config.location=/opt/spinnaker/config/" "-Dhttp.proxyHost=192.168.0.9" "-Dhttps.proxyHost=192.168.0.9" "-Dhttp.proxyPort=1087" "-Dhttps.proxyPort=1087" "-Dhttp.nonProxyHosts=\"localhost|*.spinnaker.com\""'
其中spinnaker.com给deck和gate两个组件使用。
关闭hal服务 hal shutdown
执行任何命令会启动hal服务
检测结果,能显示则说明设置没问题了
root@ubuntu:~# hal shutdown
Halyard Daemon Response: Shutting down, bye...
root@ubuntu:~# hal version list
The halyard daemon isn't running yet... starting it manually...
+ Get current deployment
Success
+ Get Spinnaker version
Success
+ Get released versions
Success
+ You are on version "1.12.5", and the following are available:
- 1.10.14 (Maniac):
Changelog: https://gist.github.com/spinnaker-release/a47efe1884b4611f57f0966173bf4f51
Published: Sat Mar 02 01:43:40 HKT 2019
(Requires Halyard >= 1.11)
- 1.11.11 (Cobra Kai):
Changelog: https://gist.github.com/spinnaker-release/12abde4a1f722164b50a2c77fb898cc0
Published: Sat Mar 02 01:46:38 HKT 2019
(Requires Halyard >= 1.11)
- 1.12.6 (Unbreakable):
Changelog: https://gist.github.com/spinnaker-release/8f0d6e084c19bde32bb84ec810863a43
Published: Thu Mar 14 10:37:19 HKT 2019
(Requires Halyard >= 1.11)
Step 2: 选择应用部署环境
Spinnaker可支持的云提供商有很多,这里只以[Kubernetes Provider V2 (Manifest Based)]作为实例 ，其中
[Kubernetes (Legacy)]这个是指老版本的k8s。
2.1 安装K8S测试集群
推荐安装的脚本：yonyoucloud
基本可以很简单的安装完成。可以单台安装，比较简单。只是为了测试。
特别注意下私有仓库的IP:PORT ，端口一般是5000
[root@osnode011238 ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
172.28.11.238:5000/esn-containers/web_test 20190312.104615 f3fe8cc926d4 3 days ago 49MB
172.28.11.238:5000/k8s.gcr.io/kubernetes-dashboard-amd64 v1.10.1 f9aed6605b81 2 months ago 122MB
172.28.11.238:5000/k8s.gcr.io/coredns 1.2.6 f59dcacceff4 4 months ago 40MB
registry latest 2e2f252f3c88 6 months ago 33.3MB
172.28.11.238:5000/quay.io/calico/node v3.1.3 7eca10056c8e 9 months ago 248MB
172.28.11.238:5000/quay.io/calico/kube-controllers v3.1.3 240a82836573 9 months ago 55MB
172.28.11.238:5000/quay.io/calico/cni v3.1.3 9f355e076ea7 9 months ago 68.8MB
172.28.11.238:5000/google_containers/heapster-amd64 v1.5.3 f57c75cd7b0a 10 months ago 75.3MB
172.28.11.238:5000/google-containers/pause-amd64 3.1 da86e6ba6ca1 14 months ago 742kB
172.28.11.238:5000/esn-containers/esn_base 1.0 3f4cf9b985de 17 months ago 39.5MB
172.28.11.238:5000/esn-containers/alpine latest 76da55c8019d 18 months ago 3.97MB
172.28.11.238:5000/google_containers/heapster-influxdb-amd64 v1.3.3 577260d221db 18 months ago 12.5MB
172.28.11.238:5000/google_containers/heapster-grafana-amd64 v4.4.3 8cb3de219af7 18 months ago 152MB
Ubuntu的服务器需要能访问这个k8s，首先把k8s-master节点的~/.kube/config 文件拷贝到对应的Ubuntu的服务器目录下（若存在文件，需要手工合并），然后安装kubectl
root@ubuntu:~# snap install kubectl --classic
root@ubuntu:~# kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1", GitTreeState:"clean", BuildDate:"2019-02-28T13:37:52Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-18T01:20:20Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
2.2 配置k8s支持
#开启K8s支持
hal config provider kubernetes enable
#使spinnaker关联k8s,account名字为my-k8s-v2-account,一会要用到
CONTEXT=$(kubectl config current-context)
hal config provider kubernetes account add my-k8s-v2-account \
--provider-version v2 \
--context $CONTEXT
hal config features edit --artifacts true
为了一步到位,此处还额外设置支持jenkins,避免后期再设置
这里没设置认证,详细看 文档
hal config ci jenkins master add my-jenkins-master --address http://jenkins.com
Step 3: 选择Spinnaker部署环境
spinnaker部署环境一共有三种:
- 在k8s分布式安装(推荐使用)
- 本地单台机器安装
- 本地git安装
选择部署环境,然后选择与上面k8s关联了的账号my-k8s-v2-account作为认证方式部署
hal config deploy edit --type distributed --account-name my-k8s-v2-account
Step 4: 选择存储设备
这里选择Redis作为存储设备,但是文档说不建议这个,只是为了测试
hal config storage edit --type redis
Step 5: 安装Spinnaker
5.1 设置安装spinnaker的版本
root@ubuntu:~# hal version list
The halyard daemon isn't running yet... starting it manually...
+ Get current deployment
Success
+ Get Spinnaker version
Success
+ Get released versions
Success
+ You are on version "1.12.5", and the following are available:
- 1.10.14 (Maniac):
Changelog: https://gist.github.com/spinnaker-release/a47efe1884b4611f57f0966173bf4f51
Published: Sat Mar 02 01:43:40 HKT 2019
(Requires Halyard >= 1.11)
- 1.11.11 (Cobra Kai):
Changelog: https://gist.github.com/spinnaker-release/12abde4a1f722164b50a2c77fb898cc0
Published: Sat Mar 02 01:46:38 HKT 2019
(Requires Halyard >= 1.11)
- 1.12.6 (Unbreakable):
Changelog: https://gist.github.com/spinnaker-release/8f0d6e084c19bde32bb84ec810863a43
Published: Thu Mar 14 10:37:19 HKT 2019
(Requires Halyard >= 1.11)
root@ubuntu:~# hal config version edit --version 1.12.5
5.2 spinnaker的组件镜像拉取
简单处理使用Step 2中安装的私有镜仓库。
Ubuntu安装docker
apt-get update
apt-get install \
apt-transport-https \
ca-certificates \
curl \
gnupg-agent \
software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo apt-key fingerprint 0EBFCD88
add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) \
stable"
apt-get update
apt-get install docker-ce docker-ce-cli containerd.io
docker info
配置docker使用私有仓库
在/etc/docker/下创建daemon.json 内容如下
{
"registry-mirrors": ["http://172.28.11.238:5000"]
}
然后拷贝k8s-master下/etc/docker/下的certs.d和key.json文件到这个目录下。为了等会从google拉取镜像，打tag重新push到这个私有仓库。
获得spinnaker组件的版本
root@ubuntu:~# hal version bom 1.12.5
+ Get BOM for 1.12.5
Success
version: 1.12.5
timestamp: '2019-03-08 23:26:14'
services:
echo:
version: 2.3.1-20190214121429
commit: 5db9d437ca7f2fa374dcada17f77bbbb2965bd67
clouddriver:
version: 4.3.5-20190307172446
commit: f87eb66fd55cd4df7497ef22528a11709745075d
deck:
version: 2.7.5-20190308182538
commit: e9b899d63cb6ea15dc2d6c99a810c8b48886c6a5
fiat:
version: 1.3.2-20190128153726
commit: daf21b24330a5f22866601559aa0f7ac99590274
front50:
version: 0.15.2-20190222161456
commit: 3105e86b8c084ad6ad78507e3a5e5a427f290b99
gate:
version: 1.5.2-20190301030607
commit: b238ab993ab25381ce907260879548ed74a4953f
igor:
version: 1.1.1-20190213190226
commit: 63d06a5c5d55f07443dd60a81035b35cf96238e7
kayenta:
version: 0.6.1-20190221030610
commit: 81d906bf8307143f40fe88f8554baa318de25ef1
orca:
version: 2.4.0-20190308182538
commit: 5e911ff1c29bbc443ce48bcaefbc45f27d389edd
rosco:
version: 0.9.0-20190123170846
commit: 42f81a2501de6d40676d47661579a6106b5c3e8a
defaultArtifact: {}
monitoring-third-party:
version: 0.11.2-20190222030609
commit: 232c84a8a87cecbc17f157dd180643a8b2e6067a
monitoring-daemon:
version: 0.11.2-20190222030609
commit: 232c84a8a87cecbc17f157dd180643a8b2e6067a
dependencies:
redis:
version: 2:2.8.4-2
consul:
version: 0.7.5
vault:
version: 0.7.0
artifactSources:
debianRepository: https://dl.bintray.com/spinnaker-releases/debians
dockerRegistry: gcr.io/spinnaker-marketplace
googleImageProject: marketplace-spinnaker-release
gitPrefix: https://github.com/spinnaker
获得到镜像和版本
gcr.io/spinnaker-marketplace/echo:2.3.1-20190214121429
gcr.io/spinnaker-marketplace/clouddriver:4.3.5-20190307172446
gcr.io/spinnaker-marketplace/deck:2.7.5-20190308182538
gcr.io/spinnaker-marketplace/fiat:1.3.2-20190128153726
gcr.io/spinnaker-marketplace/front50:0.15.2-20190222161456
gcr.io/spinnaker-marketplace/gate:1.5.2-20190301030607
gcr.io/spinnaker-marketplace/igor:1.1.1-20190213190226
gcr.io/spinnaker-marketplace/kayenta:0.6.1-20190221030610
gcr.io/spinnaker-marketplace/orca:2.4.0-20190308182538
gcr.io/spinnaker-marketplace/rosco:0.9.0-20190123170846
gcr.io/kubernetes-spinnaker/redis-cluster:v2
保存为spinnaker-list.txt文件，为脚本读取使用。
docker配置代理
mkdir -p /etc/systemd/system/docker.service.d
vim /etc/systemd/system/docker.service.d/http-proxy.conf
[Service]
Environment="HTTPS_PROXY=http://192.168.0.9:1087/" "NO_PROXY=localhost,127.0.0.1,172.28.11.238,registry.docker-cn.com,hub-mirror.c.163.com"
systemctl daemon-reload
systemctl restart docker
脚本拉取-Tag-push
import os
if __name__ == '__main__':
images = open("spinnaker-list.txt", 'r', encoding='utf-8')
for line in images.readlines():
name = line.strip()
tag = name.replace("gcr.io", "172.28.11.238:5000")
os.system('docker pull %s' % name)
os.system('docker tag %s %s' % (name, tag))
os.system('docker push %s' % tag)
5.3 配置hal的自定义参数
在.hal的目录下找到对应的目录,修改artifactId和overrideBaseUrl。其他更多设置可以参考这里 Custom Configuration
cd /home/spinnaker/.hal/default
mkdir service-settings
创建如下文件，文件内容如下
root@ubuntu:/home/spinnaker/.hal/default/service-settings# ls
clouddriver.yml deck.yml echo.yml fiat.yml front50.yml gate.yml igor.yml kayenta.yml orca.yml redis.yml rosco.yml
root@ubuntu:/home/spinnaker/.hal/default/service-settings# cat *
artifactId: 172.28.11.238:5000/spinnaker-marketplace/clouddriver:4.3.5-20190307172446
artifactId: 172.28.11.238:5000/spinnaker-marketplace/deck:2.7.5-20190308182538
overrideBaseUrl: http://deck.spinnaker.com
artifactId: 172.28.11.238:5000/spinnaker-marketplace/echo:2.3.1-20190214121429
artifactId: 172.28.11.238:5000/spinnaker-marketplace/fiat:1.3.2-20190128153726
artifactId: 172.28.11.238:5000/spinnaker-marketplace/front50:0.15.2-20190222161456
artifactId: 172.28.11.238:5000/spinnaker-marketplace/gate:1.5.2-20190301030607
overrideBaseUrl: http://gate.spinnaker.com
artifactId: 172.28.11.238:5000/spinnaker-marketplace/igor:1.1.1-20190213190226
artifactId: 172.28.11.238:5000/spinnaker-marketplace/kayenta:0.6.1-20190221030610
artifactId: 172.28.11.238:5000/spinnaker-marketplace/orca:2.4.0-20190308182538
artifactId: 172.28.11.238:5000/kubernetes-spinnaker/redis-cluster:v2
artifactId: 172.28.11.238:5000/spinnaker-marketplace/rosco:0.9.0-20190123170846
完成设置后,使用命令部署
部署spinnaker
[[email protected] service-settings]$hal deploy apply
+ Get current deployment
Success
+ Prep deployment
Success
Problems in default.security:
- WARNING Your UI or API domain does not have override base URLs
set even though your Spinnaker deployment is a Distributed deployment on a
remote cloud provider. As a result, you will need to open SSH tunnels against
that deployment to access Spinnaker.
? We recommend that you instead configure an authentication
mechanism (OAuth2, SAML2, or x509) to make it easier to access Spinnaker
securely, and then register the intended Domain and IP addresses that your
publicly facing services will be using.
+ Preparation complete... deploying Spinnaker
+ Get current deployment
Success
+ Apply deployment
Success
+ Deploy spin-redis
Success
+ Deploy spin-clouddriver
Success
+ Deploy spin-front50
Success
+ Deploy spin-orca
Success
+ Deploy spin-deck
Success
+ Deploy spin-echo
Success
+ Deploy spin-gate
Success
+ Deploy spin-rosco
Success
+ Deploy spin-igor
Success
+ Run `hal deploy connect` to connect to Spinnaker.
查看k8s中部署情况
[root@osnode011238 ~]# kubectl get pods -n spinnaker
NAME READY STATUS RESTARTS AGE
spin-clouddriver-849fd8f6c9-55ps6 1/1 Running 0 27h
spin-deck-85bc568c74-flqj2 1/1 Running 0 27h
spin-echo-69d7565658-dpz7t 1/1 Running 0 27h
spin-front50-d46888f79-q8dxr 1/1 Running 0 27h
spin-gate-6b7465474c-pqgkn 1/1 Running 0 27h
spin-orca-679dc6ff45-r2h2w 1/1 Running 0 27h
spin-redis-8489d5f685-d6j5m 1/1 Running 0 27h
spin-rosco-7c874fcf8c-xbznj 1/1 Running 0 27h
[root@osnode011238 ~]#
如果没有成功，可以删除namespace，重新使用hal deploy apply部署。
Step 6: 配置Ingress访问Spinnaker
使用kubectl apply -f 文件名
kind: ConfigMap
apiVersion: v1
metadata:
name: nginx-configuration
namespace: spinnaker
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
kind: ConfigMap
apiVersion: v1
metadata:
name: tcp-services
namespace: spinnaker
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
kind: ConfigMap
apiVersion: v1
metadata:
name: udp-services
namespace: spinnaker
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: nginx-ingress-serviceaccount
namespace: spinnaker
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: nginx-ingress-clusterrole
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
rules:
- apiGroups:
- ""
resources:
- configmaps
- endpoints
- nodes
- pods
- secrets
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- apiGroups:
- ""
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- "extensions"
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- apiGroups:
- "extensions"
resources:
- ingresses/status
verbs:
- update
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
name: nginx-ingress-role
namespace: spinnaker
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
rules:
- apiGroups:
- ""
resources:
- configmaps
- pods
- secrets
- namespaces
verbs:
- get
- apiGroups:
- ""
resources:
- configmaps
resourceNames:
# Defaults to "-"
# Here: "-"
# This has to be adapted if you change either parameter
# when launching the nginx-ingress-controller.
- "ingress-controller-leader-nginx"
verbs:
- get
- update
- apiGroups:
- ""
resources:
- configmaps
verbs:
- create
- apiGroups:
- ""
resources:
- endpoints
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: nginx-ingress-role-nisa-binding
namespace: spinnaker
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: nginx-ingress-role
subjects:
- kind: ServiceAccount
name: nginx-ingress-serviceaccount
namespace: spinnaker
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: nginx-ingress-clusterrole-nisa-binding
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: nginx-ingress-clusterrole
subjects:
- kind: ServiceAccount
name: nginx-ingress-serviceaccount
namespace: spinnaker
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-ingress-controller
namespace: spinnaker
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
template:
metadata:
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
annotations:
prometheus.io/port: "10254"
prometheus.io/scrape: "true"
spec:
serviceAccountName: nginx-ingress-serviceaccount
containers:
- name: nginx-ingress-controller
image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.23.0
args:
- /nginx-ingress-controller
- --configmap=$(POD_NAMESPACE)/nginx-configuration
- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
- --udp-services-configmap=$(POD_NAMESPACE)/udp-services
- --publish-service=$(POD_NAMESPACE)/ingress-nginx
- --annotations-prefix=nginx.ingress.kubernetes.io
securityContext:
allowPrivilegeEscalation: true
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
# www-data -> 33
runAsUser: 33
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
ports:
- name: http
containerPort: 80
- name: https
containerPort: 443
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 10
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
name: ingress-nginx
namespace: spinnaker
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
externalIPs:
- k8s的node节点IP1
- k8s的node节点IP2
type: NodePort
ports:
- name: http
port: 80
targetPort: 80
protocol: TCP
- name: https
port: 443
targetPort: 443
protocol: TCP
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: name-virtual-host-ingress
namespace: spinnaker
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
rules:
- host: deck.spinnaker.com
http:
paths:
- backend:
serviceName: spin-deck
servicePort: 9000
- host: gate.spinnaker.com
http:
paths:
- backend:
serviceName: spin-gate
servicePort: 8084
本地配置Host文件添加记录：
任意k8s的node节点IP deck.spinnaker.com
任意k8s的node节点IP gate.spinnaker.com
可以在浏览器使用deck.spinnaker.com访问了。deck只是前端，需要通过gate调用api访问数据。