概要
statefulset应用是有状态并且数据有强关联性，多副本之间不能共用一个存储盘，所以每个副本需要一个单独的存储盘；通常情况下为多副本的statefulset设置持久化存储需要使用到Dynamic Provisioning，可以理解为需要一个StorageClass 存储；在openshift3.11 prometheus的管理由operator管理，也就是会使用到statefulset；Dynamic Provisioning对存储的要求较高，有些客户那里并没有支持Dynamic Provisioning的存储，其实NFS也是支持Dynamic Provisioning的，但是这个需要openshift对接到NFS存储管理层面，意味着openshift自己去NFS存储上创建NFS卷，但是这个在大部分客户那里是无法接受提供这么高的管理权限给openshift；通常情况下客户的存储负责人那里提供一个比较大的NFS卷给到openshift，openshift自行规划使用。基于上述原因我要给operator 管理 statefulset (prometheus) 多副本配置持久化存储只能使用客户提供好的nfs盘，在里面自己创建应用目录进行使用了。
通常情况下定义statefulset多副本的yml文件
创建了这个statefulset后，会自动的去找能够使用的storageclass创建PVC/PV，然后挂载至对应的pod上。
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: web
spec:
serviceName: "nginx"
replicas: 2
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.9.1
ports:
- containerPort: 80
name: web
volumeMounts:
- name: www
mountPath: /usr/share/nginx/html
volumeClaimTemplates:
- metadata:
name: www
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
默认openshift未对prometheus进行持久化statefulset的yml文件
可以看到prometheus的volume字段是空的，未进行持久化。
apiVersion: apps/v1
kind: StatefulSet
metadata:
creationTimestamp: null
generation: 2
labels:
prometheus: k8s
name: prometheus-k8s
ownerReferences:
- apiVersion: monitoring.coreos.com/v1
blockOwnerDeletion: true
controller: true
kind: Prometheus
name: k8s
spec:
podManagementPolicy: OrderedReady
replicas: 2
revisionHistoryLimit: 10
selector:
matchLabels:
app: prometheus
prometheus: k8s
serviceName: prometheus-operated
template:
metadata:
creationTimestamp: null
labels:
app: prometheus
prometheus: k8s
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchExpressions:
- key: prometheus
operator: In
values:
- k8s
namespaces:
- openshift-monitoring
topologyKey: kubernetes.io/hostname
weight: 100
containers:
- args:
- --web.console.templates=/etc/prometheus/consoles
- --web.console.libraries=/etc/prometheus/console_libraries
- --config.file=/etc/prometheus/config_out/prometheus.env.yaml
- --storage.tsdb.path=/prometheus
- --storage.tsdb.retention=15d
- --web.enable-lifecycle
- --storage.tsdb.no-lockfile
- --web.external-url=https://prometheus-k8s.xxxxxx/
- --web.route-prefix=/
- --web.listen-address=127.0.0.1:9090
image: registry.access.redhat.com/openshift3/prometheus:v3.11
imagePullPolicy: IfNotPresent
name: prometheus
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/prometheus/config_out
name: config-out
readOnly: true
- mountPath: /prometheus
name: prometheus-k8s-db
- mountPath: /etc/prometheus/rules/prometheus-k8s-rulefiles-0
name: prometheus-k8s-rulefiles-0
- mountPath: /etc/prometheus/secrets/prometheus-k8s-tls
name: secret-prometheus-k8s-tls
readOnly: true
- mountPath: /etc/prometheus/secrets/prometheus-k8s-proxy
name: secret-prometheus-k8s-proxy
readOnly: true
- mountPath: /etc/prometheus/secrets/prometheus-k8s-htpasswd
name: secret-prometheus-k8s-htpasswd
readOnly: true
- args:
- --log-format=logfmt
- --reload-url=http://localhost:9090/-/reload
- --config-file=/etc/prometheus/config/prometheus.yaml
- --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
command:
- /bin/prometheus-config-reloader
env:
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
image: registry.access.redhat.com/openshift3/ose-prometheus-config-reloader:v3.11
imagePullPolicy: IfNotPresent
name: prometheus-config-reloader
resources:
limits:
cpu: 10m
memory: 50Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/prometheus/config
name: config
- mountPath: /etc/prometheus/config_out
name: config-out
- args:
- -provider=openshift
- -https-address=:9091
- -http-address=
- -email-domain=*
- -upstream=http://localhost:9090
- -htpasswd-file=/etc/proxy/htpasswd/auth
- -openshift-service-account=prometheus-k8s
- '-openshift-sar={"resource": "namespaces", "verb": "get"}'
- '-openshift-delegate-urls={"/": {"resource": "namespaces", "verb": "get"}}'
- -tls-cert=/etc/tls/private/tls.crt
- -tls-key=/etc/tls/private/tls.key
- -client-secret-file=/var/run/secrets/kubernetes.io/serviceaccount/token
- -cookie-secret-file=/etc/proxy/secrets/session_secret
- -openshift-ca=/etc/pki/tls/cert.pem
- -openshift-ca=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
- -skip-auth-regex=^/metrics
image: registry.access.redhat.com/openshift3/oauth-proxy:v3.11
imagePullPolicy: IfNotPresent
name: prometheus-proxy
ports:
- containerPort: 9091
name: web
protocol: TCP
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/tls/private
name: secret-prometheus-k8s-tls
- mountPath: /etc/proxy/secrets
name: secret-prometheus-k8s-proxy
- mountPath: /etc/proxy/htpasswd
name: secret-prometheus-k8s-htpasswd
- args:
- --webhook-url=http://localhost:9090/-/reload
- --volume-dir=/etc/prometheus/rules/prometheus-k8s-rulefiles-0
image: registry.access.redhat.com/openshift3/ose-configmap-reloader:v3.11
imagePullPolicy: IfNotPresent
name: rules-configmap-reloader
resources:
limits:
cpu: 5m
memory: 10Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/prometheus/rules/prometheus-k8s-rulefiles-0
name: prometheus-k8s-rulefiles-0
dnsPolicy: ClusterFirst
nodeSelector:
node-role.kubernetes.io/infra: "true"
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: prometheus-k8s
serviceAccountName: prometheus-k8s
terminationGracePeriodSeconds: 600
volumes:
- name: config
secret:
defaultMode: 420
secretName: prometheus-k8s
- emptyDir: {}
name: config-out
- configMap:
defaultMode: 420
name: prometheus-k8s-rulefiles-0
name: prometheus-k8s-rulefiles-0
- name: secret-prometheus-k8s-tls
secret:
defaultMode: 420
secretName: prometheus-k8s-tls
- name: secret-prometheus-k8s-proxy
secret:
defaultMode: 420
secretName: prometheus-k8s-proxy
- name: secret-prometheus-k8s-htpasswd
secret:
defaultMode: 420
secretName: prometheus-k8s-htpasswd
- emptyDir: {}
name: prometheus-k8s-db
updateStrategy:
type: RollingUpdate
status:
collisionCount: 0
currentReplicas: 2
currentRevision: prometheus-k8s-68fb6c678
observedGeneration: 2
readyReplicas: 2
replicas: 2
updateRevision: prometheus-k8s-68fb6c678
updatedReplicas: 2
对prometheus多副本进行持久化的
- yml文件
可以看到volume字段的prometheus-k8s-db没了，多了个volumeClaimTemplates字段。
apiVersion: apps/v1
kind: StatefulSet
metadata:
creationTimestamp: 2019-06-26T08:07:55Z
generation: 1
labels:
prometheus: k8s
name: prometheus-k8s
namespace: openshift-monitoring
ownerReferences:
- apiVersion: monitoring.coreos.com/v1
blockOwnerDeletion: true
controller: true
kind: Prometheus
name: k8s
resourceVersion: "20759155"
selfLink: /apis/apps/v1/namespaces/openshift-monitoring/statefulsets/prometheus-k8s
spec:
podManagementPolicy: OrderedReady
replicas: 2
revisionHistoryLimit: 10
selector:
matchLabels:
app: prometheus
prometheus: k8s
serviceName: prometheus-operated
template:
metadata:
creationTimestamp: null
labels:
app: prometheus
prometheus: k8s
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchExpressions:
- key: prometheus
operator: In
values:
- k8s
namespaces:
- openshift-monitoring
topologyKey: kubernetes.io/hostname
weight: 100
containers:
- args:
- --web.console.templates=/etc/prometheus/consoles
- --web.console.libraries=/etc/prometheus/console_libraries
- --config.file=/etc/prometheus/config_out/prometheus.env.yaml
- --storage.tsdb.path=/prometheus
- --storage.tsdb.retention=15d
- --web.enable-lifecycle
- --storage.tsdb.no-lockfile
- --web.external-url=https://prometheus-k8s.xxxx/
- --web.route-prefix=/
- --web.listen-address=127.0.0.1:9090
image: registry.access.redhat.com/openshift3/prometheus:v3.11
imagePullPolicy: IfNotPresent
name: prometheus
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/prometheus/config_out
name: config-out
readOnly: true
- mountPath: /prometheus
name: prometheus-k8s-db
- mountPath: /etc/prometheus/rules/prometheus-k8s-rulefiles-0
name: prometheus-k8s-rulefiles-0
- mountPath: /etc/prometheus/secrets/prometheus-k8s-tls
name: secret-prometheus-k8s-tls
readOnly: true
- mountPath: /etc/prometheus/secrets/prometheus-k8s-proxy
name: secret-prometheus-k8s-proxy
readOnly: true
- mountPath: /etc/prometheus/secrets/prometheus-k8s-htpasswd
name: secret-prometheus-k8s-htpasswd
readOnly: true
- args:
- --log-format=logfmt
- --reload-url=http://localhost:9090/-/reload
- --config-file=/etc/prometheus/config/prometheus.yaml
- --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
command:
- /bin/prometheus-config-reloader
env:
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
image: registry.access.redhat.com/openshift3/ose-prometheus-config-reloader:v3.11
imagePullPolicy: IfNotPresent
name: prometheus-config-reloader
resources:
limits:
cpu: 10m
memory: 50Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/prometheus/config
name: config
- mountPath: /etc/prometheus/config_out
name: config-out
- args:
- -provider=openshift
- -https-address=:9091
- -http-address=
- -email-domain=*
- -upstream=http://localhost:9090
- -htpasswd-file=/etc/proxy/htpasswd/auth
- -openshift-service-account=prometheus-k8s
- '-openshift-sar={"resource": "namespaces", "verb": "get"}'
- '-openshift-delegate-urls={"/": {"resource": "namespaces", "verb": "get"}}'
- -tls-cert=/etc/tls/private/tls.crt
- -tls-key=/etc/tls/private/tls.key
- -client-secret-file=/var/run/secrets/kubernetes.io/serviceaccount/token
- -cookie-secret-file=/etc/proxy/secrets/session_secret
- -openshift-ca=/etc/pki/tls/cert.pem
- -openshift-ca=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
- -skip-auth-regex=^/metrics
image: registry.access.redhat.com/openshift3/oauth-proxy:v3.11
imagePullPolicy: IfNotPresent
name: prometheus-proxy
ports:
- containerPort: 9091
name: web
protocol: TCP
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/tls/private
name: secret-prometheus-k8s-tls
- mountPath: /etc/proxy/secrets
name: secret-prometheus-k8s-proxy
- mountPath: /etc/proxy/htpasswd
name: secret-prometheus-k8s-htpasswd
- args:
- --webhook-url=http://localhost:9090/-/reload
- --volume-dir=/etc/prometheus/rules/prometheus-k8s-rulefiles-0
image: registry.access.redhat.com/openshift3/ose-configmap-reloader:v3.11
imagePullPolicy: IfNotPresent
name: rules-configmap-reloader
resources:
limits:
cpu: 5m
memory: 10Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/prometheus/rules/prometheus-k8s-rulefiles-0
name: prometheus-k8s-rulefiles-0
dnsPolicy: ClusterFirst
nodeSelector:
node-role.kubernetes.io/infra: "true"
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: prometheus-k8s
serviceAccountName: prometheus-k8s
terminationGracePeriodSeconds: 600
volumes:
- name: config
secret:
defaultMode: 420
secretName: prometheus-k8s
- emptyDir: {}
name: config-out
- configMap:
defaultMode: 420
name: prometheus-k8s-rulefiles-0
name: prometheus-k8s-rulefiles-0
- name: secret-prometheus-k8s-tls
secret:
defaultMode: 420
secretName: prometheus-k8s-tls
- name: secret-prometheus-k8s-proxy
secret:
defaultMode: 420
secretName: prometheus-k8s-proxy
- name: secret-prometheus-k8s-htpasswd
secret:
defaultMode: 420
secretName: prometheus-k8s-htpasswd
updateStrategy:
type: RollingUpdate
volumeClaimTemplates:
- metadata:
creationTimestamp: null
name: prometheus-k8s-db
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
- 在规划好的nfs目录上创建两个pv，因为副本是两个，pv yml
apiVersion: v1
kind: PersistentVolume
metadata:
creationTimestamp: null
finalizers:
- kubernetes.io/pv-protection
name: prometheus-0
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 10Gi
nfs:
path: /prometheus-0
server: xxxx
persistentVolumeReclaimPolicy: Retain
apiVersion: v1
kind: PersistentVolume
metadata:
creationTimestamp: null
finalizers:
- kubernetes.io/pv-protection
name: prometheus-1
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 10Gi
nfs:
path: /prometheus-1
server: xxxx
persistentVolumeReclaimPolicy: Retain
- 删除原有的statefulset的prometheus
oc delete sts prometheus-k8s
- 使用两个pv文件创建两个pv
oc create -f xxx yyyy
- 使用配置好持久化的statefulset prometheus文件进行创建
oc create -f zzz
- 查看pv状态
oc get pv
prometheus-0 10Gi RWO Retain Bound openshift-monitoring/prometheus-k8s-db-prometheus-k8s-0 9d
prometheus-1 10Gi RWO Retain Bound openshift-monitoring/prometheus-k8s-db-prometheus-k8s-1 9d
- 查看pvc
可以看到会自动创建两个pvc，并且绑定到之前创建的两个pv上面。
oc get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
prometheus-k8s-db-prometheus-k8s-0 Bound prometheus-0 10Gi RWO 9d
prometheus-k8s-db-prometheus-k8s-1 Bound prometheus-1 10Gi RWO 9d
- 查看pod的yaml文件
oc get pod prometheus-k8s-0 -oyaml
apiVersion: v1
kind: Pod
metadata:
annotations:
openshift.io/scc: restricted
creationTimestamp: 2019-06-26T08:07:55Z
generateName: prometheus-k8s-
labels:
app: prometheus
controller-revision-hash: prometheus-k8s-69779b749
prometheus: k8s
statefulset.kubernetes.io/pod-name: prometheus-k8s-0
name: prometheus-k8s-0
namespace: openshift-monitoring
ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: StatefulSet
name: prometheus-k8s
uid: 80ba5cd2-97e9-11e9-a7ad-00163e095b6c
resourceVersion: "20759046"
selfLink: /api/v1/namespaces/openshift-monitoring/pods/prometheus-k8s-0
uid: 80cb7a88-97e9-11e9-8c89-00163e095c2a
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchExpressions:
- key: prometheus
operator: In
values:
- k8s
namespaces:
- openshift-monitoring
topologyKey: kubernetes.io/hostname
weight: 100
containers:
- args:
- --web.console.templates=/etc/prometheus/consoles
- --web.console.libraries=/etc/prometheus/console_libraries
- --config.file=/etc/prometheus/config_out/prometheus.env.yaml
- --storage.tsdb.path=/prometheus
- --storage.tsdb.retention=15d
- --web.enable-lifecycle
- --storage.tsdb.no-lockfile
- --web.external-url=https://prometheus-k8s.xxxx/
- --web.route-prefix=/
- --web.listen-address=127.0.0.1:9090
image: registry.access.redhat.com/openshift3/prometheus:v3.11
imagePullPolicy: IfNotPresent
name: prometheus
resources: {}
securityContext:
capabilities:
drop:
- KILL
- MKNOD
- SETGID
- SETUID
runAsUser: 1000140000
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/prometheus/config_out
name: config-out
readOnly: true
- mountPath: /prometheus
name: prometheus-k8s-db
- mountPath: /etc/prometheus/rules/prometheus-k8s-rulefiles-0
name: prometheus-k8s-rulefiles-0
- mountPath: /etc/prometheus/secrets/prometheus-k8s-tls
name: secret-prometheus-k8s-tls
readOnly: true
- mountPath: /etc/prometheus/secrets/prometheus-k8s-proxy
name: secret-prometheus-k8s-proxy
readOnly: true
- mountPath: /etc/prometheus/secrets/prometheus-k8s-htpasswd
name: secret-prometheus-k8s-htpasswd
readOnly: true
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: prometheus-k8s-token-l66z9
readOnly: true
- args:
- --log-format=logfmt
- --reload-url=http://localhost:9090/-/reload
- --config-file=/etc/prometheus/config/prometheus.yaml
- --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
command:
- /bin/prometheus-config-reloader
env:
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
image: registry.access.redhat.com/openshift3/ose-prometheus-config-reloader:v3.11
imagePullPolicy: IfNotPresent
name: prometheus-config-reloader
resources:
limits:
cpu: 10m
memory: 50Mi
requests:
cpu: 10m
memory: 50Mi
securityContext:
capabilities:
drop:
- KILL
- MKNOD
- SETGID
- SETUID
runAsUser: 1000140000
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/prometheus/config
name: config
- mountPath: /etc/prometheus/config_out
name: config-out
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: prometheus-k8s-token-l66z9
readOnly: true
- args:
- -provider=openshift
- -https-address=:9091
- -http-address=
- -email-domain=*
- -upstream=http://localhost:9090
- -htpasswd-file=/etc/proxy/htpasswd/auth
- -openshift-service-account=prometheus-k8s
- '-openshift-sar={"resource": "namespaces", "verb": "get"}'
- '-openshift-delegate-urls={"/": {"resource": "namespaces", "verb": "get"}}'
- -tls-cert=/etc/tls/private/tls.crt
- -tls-key=/etc/tls/private/tls.key
- -client-secret-file=/var/run/secrets/kubernetes.io/serviceaccount/token
- -cookie-secret-file=/etc/proxy/secrets/session_secret
- -openshift-ca=/etc/pki/tls/cert.pem
- -openshift-ca=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
- -skip-auth-regex=^/metrics
image: registry.access.redhat.com/openshift3/oauth-proxy:v3.11
imagePullPolicy: IfNotPresent
name: prometheus-proxy
ports:
- containerPort: 9091
name: web
protocol: TCP
resources: {}
securityContext:
capabilities:
drop:
- KILL
- MKNOD
- SETGID
- SETUID
runAsUser: 1000140000
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/tls/private
name: secret-prometheus-k8s-tls
- mountPath: /etc/proxy/secrets
name: secret-prometheus-k8s-proxy
- mountPath: /etc/proxy/htpasswd
name: secret-prometheus-k8s-htpasswd
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: prometheus-k8s-token-l66z9
readOnly: true
- args:
- --webhook-url=http://localhost:9090/-/reload
- --volume-dir=/etc/prometheus/rules/prometheus-k8s-rulefiles-0
image: registry.access.redhat.com/openshift3/ose-configmap-reloader:v3.11
imagePullPolicy: IfNotPresent
name: rules-configmap-reloader
resources:
limits:
cpu: 5m
memory: 10Mi
requests:
cpu: 5m
memory: 10Mi
securityContext:
capabilities:
drop:
- KILL
- MKNOD
- SETGID
- SETUID
runAsUser: 1000140000
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/prometheus/rules/prometheus-k8s-rulefiles-0
name: prometheus-k8s-rulefiles-0
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: prometheus-k8s-token-l66z9
readOnly: true
dnsPolicy: ClusterFirst
hostname: prometheus-k8s-0
imagePullSecrets:
- name: prometheus-k8s-dockercfg-gwmqm
nodeName: xxxx
nodeSelector:
node-role.kubernetes.io/infra: "true"
priority: 0
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
fsGroup: 1000140000
seLinuxOptions:
level: s0:c12,c4
serviceAccount: prometheus-k8s
serviceAccountName: prometheus-k8s
subdomain: prometheus-operated
terminationGracePeriodSeconds: 600
tolerations:
- effect: NoSchedule
key: node.kubernetes.io/memory-pressure
operator: Exists
volumes:
- name: prometheus-k8s-db
persistentVolumeClaim:
claimName: prometheus-k8s-db-prometheus-k8s-0
- name: config
secret:
defaultMode: 420
secretName: prometheus-k8s
- emptyDir: {}
name: config-out
- configMap:
defaultMode: 420
name: prometheus-k8s-rulefiles-0
name: prometheus-k8s-rulefiles-0
- name: secret-prometheus-k8s-tls
secret:
defaultMode: 420
secretName: prometheus-k8s-tls
- name: secret-prometheus-k8s-proxy
secret:
defaultMode: 420
secretName: prometheus-k8s-proxy
- name: secret-prometheus-k8s-htpasswd
secret:
defaultMode: 420
secretName: prometheus-k8s-htpasswd
- name: prometheus-k8s-token-l66z9
secret:
defaultMode: 420
secretName: prometheus-k8s-token-l66z9
status:
conditions:
- lastProbeTime: null
lastTransitionTime: 2019-06-26T08:07:56Z
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: 2019-06-26T08:08:03Z
status: "True"
type: Ready
- lastProbeTime: null
lastTransitionTime: null
status: "True"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: 2019-06-26T08:07:56Z
status: "True"
type: PodScheduled
containerStatuses:
- containerID: docker://514552885b6f8754023a4ba561531746bce627bc1b014661b9e1a7384991ea3c
image: registry.access.redhat.com/openshift3/prometheus:v3.11
imageID: docker-pullable://registry.access.redhat.com/openshift3/prometheus@sha256:1a77f9a6e27fcc3bb7a0b8b5b48a64be374991623e707559cad671ddaa059bb4
lastState:
terminated:
containerID: docker://40f9884c98cf43162958bdac71fa0251b3b840e3bcea8b3b0b0b2c6ae0061e45
exitCode: 1
finishedAt: 2019-06-26T08:07:58Z
reason: Error
startedAt: 2019-06-26T08:07:58Z
name: prometheus
ready: true
restartCount: 1
state:
running:
startedAt: 2019-06-26T08:08:02Z
- containerID: docker://b9b483c70d6f160db502377016ff5a37757c7ae9a26957038295b12ced38ec27
image: registry.access.redhat.com/openshift3/ose-prometheus-config-reloader:v3.11
imageID: docker-pullable://registry.access.redhat.com/openshift3/ose-prometheus-config-reloader@sha256:59ba60ec6d4bb690937ade48d7fa86425b3a0d46d34277d7998f9d3d0909c686
lastState: {}
name: prometheus-config-reloader
ready: true
restartCount: 0
state:
running:
startedAt: 2019-06-26T08:08:00Z
- containerID: docker://89f3bf9ba851da30ab5c9e70a1255c4ad3d22a3a8527f03915287ad23c75c620
image: registry.access.redhat.com/openshift3/oauth-proxy:v3.11
imageID: docker-pullable://registry.access.redhat.com/openshift3/oauth-proxy@sha256:3e50aaa617cf65b890796992bc3f66932ef30a3c06ca39454b0b5b0412d2a7be
lastState: {}
name: prometheus-proxy
ready: true
restartCount: 0
state:
running:
startedAt: 2019-06-26T08:08:00Z
- containerID: docker://cbfe5b66646c6413835228404902d42d6b54326c6be59c093e645957252ef701
image: registry.access.redhat.com/openshift3/ose-configmap-reloader:v3.11
imageID: docker-pullable://registry.access.redhat.com/openshift3/ose-configmap-reloader@sha256:d8efe959e815f765e6f9f3cfca121986be68a9ae7ab0ed8078dae7878e6e7c07
lastState: {}
name: rules-configmap-reloader
ready: true
restartCount: 0
state:
running:
startedAt: 2019-06-26T08:08:02Z
hostIP: xxxx
phase: Running
podIP: 172.34.3.4
qosClass: Burstable
startTime: 2019-06-26T08:07:56Z
- 进入两个prometheus的pod能看到挂载上了对应的nfs存储
oc rsh prometheus-k8s-0
Defaulting container name to prometheus.
Use 'oc describe pod/prometheus-k8s-0 -n openshift-monitoring' to see all of the containers in this pod.
sh-4.2$ df -h
Filesystem Size Used Avail Use% Mounted on
overlay 99G 15G 80G 16% /
tmpfs 3.9G 0 3.9G 0% /dev
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
xxxxxxxxxxxxxx:/prometheus-0 10P 13G 10P 1% /prometheus
/dev/vda1 99G 15G 80G 16% /etc/hosts
shm 64M 0 64M 0% /dev/shm
tmpfs 3.9G 8.0K 3.9G 1% /etc/prometheus/secrets/prometheus-k8s-tls
tmpfs 3.9G 4.0K 3.9G 1% /etc/prometheus/secrets/prometheus-k8s-proxy
tmpfs 3.9G 4.0K 3.9G 1% /etc/prometheus/secrets/prometheus-k8s-htpasswd
tmpfs 3.9G 16K 3.9G 1% /run/secrets/kubernetes.io/serviceaccount
tmpfs 3.9G 0 3.9G 0% /proc/acpi
tmpfs 3.9G 0 3.9G 0% /proc/scsi
tmpfs 3.9G 0 3.9G 0% /sys/firmware