Today, CoreOS introduced a new class of software called Operators and are also introducing two Operators as open source projects, one for etcd and another for Prometheus. In this post, we'll outline the importance of an Operator for Prometheus, the monitoring system for K8S.
An Operator builds upon the basic K8S resource and controller concepts but it includes application domain knowledge to take care of common tasks.They ultimately help you focus on a desired configuration, not the details of mannual deployment and lifecycle management.
Prometheus is a close cousin of K8S:Google introduced K8S as an open source descendent of their Borg cluster system and Promethues shares fundamental design concepts with Borgmon, the monitoring system paired with Borg.Today, both Prometheus and K8S are governed by the CNCF.And at a technical level K8S exports all of its internal metrics in the native Prometheus format.
The Prometheus Operator: The best way to intergate K8S and Prometheus
The Prometheus Operator is simple to install with a single command line, and enables users to configure and manage instances of Prometheus using simple declarative configuration that will, in response, create, configure, and manage Prometheus monitoring instances.
Once installed the Prometheus Operator provides the following features:
Creator/Desotry
Easily launch a Prometheus instance for your K8S namespace, a specific application or team easily using the Operator.
Simple Configuration
Configure the fundamentals of Prometheus like versions, pesistence, retention policies, and replicas from a native Kubernetes reource.
Target Services via Labels
Automatically generate monitoring target configurations based on familiar K8S label queries;no need to learn of learning a Prometheus specific configuration language.
How it works
The core idea of the Operator is to decouple deployment of Prometheus instances from the configuration of which entiies they are monitoring.For the purpose two TPRS are defined:Prometheus and ServiceMonitor.
The Operaotr ensures at all times that for each Prometheus resource in the cluster a set of Prometheus servers with the desired configuration are running.This entails aspects like the data rentention time, persistent volume claims, number of replicas, the Promethes version, and AlertManager instances to send alerts to.Each Prometheus instance is paired with a respective configuration that specifices which monitoring targets to scrape for metrics and with which parameters.
The user can either manully specify this configuration or let the Operator generate it based on the TPR, the ServiceMonitor. The ServiceMonitor resource specifies how metrics can be retrieved from a set of services exposing them in a common way. A Prometheus resouce object can dynamically include ServiceMonitor objects by their labels. The Operator configures the Prometheus instance to monitor all services convered by included ServiceMonitors and keeps this configuration synchronized with any changes happening in the cluster.
The Operator encapsulates a large part of the Prometheus domain knowledge and only surfaces aspects meaningful to the monitoring system's end user.It's a powerful approach that enables engineers across all teams of an organization to be autonomous and flexible in the way they run their monitoring.
Prometheus Operator in Action
We are going to walk through a full demonstration of the Prometheus Operator by creating a Prometheus instance and some service to monitor.Let's start by deploying our first Prometheus instance.
First, you need running K8S cluster with aplha APIS enabled.If you don't already have a cluster, follow the minikube instructions to quickly get a local cluster up and running.
Managed Deployments
Let's start by deploying the Prometheus Operator in our cluster.
创建失败，没有权限：
thirdpartyresources.extensions is forbidden: User "system:serviceaccount:default:default" cannot create resource "thirdpartyresources" in API group "extensions" at the cluster scope
Verify that it is up and running and has registerd the TPR types with the K8S API server.
A simple definition of a Prometheus TPR that deploys a single Prometheus instance looks like this
This also creates service to make the Prometheus UI accessible for the user. For the purpose of this demo, a service exposing it on NodePort 30900 is created.
We can now reach the Prometheus UI by going to http://:30900
In the same manner we can easily deploy further Prometheus servers and use advanced options in our Prometheus TPR to let the Operator handle version upgrades, pesistent volume claims, and connecting Prometheus to Alertmanger instances.
You can read more on the full capabilities of the managed Prometheus deployments in the repository's documentation.
Cluster Monitoring
We successfully created a managed Prometheus server.However, it is not monitoring anything yet as we did not provide any configuration.Each Prometheus deployment mounts a Kubernetes ConfigMap named after itself,i.e. our Prometheus server mounts the configuration provided in the "prometheus-k8s" ConfigMap in its namespace.
We want our Prometheus server to monitor all aspects of our cluster itself like container resource usage,cluster nodes,and kubelets.Kubernetes chose the Prometheus metric format as the canonical way to expose metrics for all its components.So,we only need to point Prometheus to the right endpoints to retrieve those metrics.This works the same across virtualy any cluster and we can use the predefined manifests in our kube-prometheus repository.
Service Monitoring
On top of monitoring our cluster components,we also want to monitor our own services.Using the regular Prometheus configuration,we have to deal with the concept of relabeling to dicover and configure monitoring targets properly.It is a powerful approach allowing Prometheus to integrate with a variety of service discovery mechanisms and arbitrary operational models.However,it is very verbose and repetitive and thus not generally suitable to be written manually.
The Prometheus Operator solves this problem by defining a second TPR to express how to monitor our custom services in a way that is fully idiomatic to Kubernetes.
Suppose all our services with the label 'tier = frontend' serve metrics on the named port 'web' under the standard '/metrics' path.The ServiceMonitor TPR allow us to declaratively express a monitoring configuration that applies to all those services,selecting them by the 'tier' label.
This merely defines how a set of services should be monitored.We now need define Prometheus instance that include this ServiceMonitor into its configuration.ServiceMonitors belonging to a Prometheus setup are selected,once again,based on labels.When deploying said Prometheus instance,the Operator configures it accoring to the matching service monitors.
Future Directions
With Operators introduced today we showcase the power of the Kubernetes platform.The Prometheus Operator extends the Kubernetes API with new monitoring capabilities.We have seen how the Prometheus Operator helps us with dynamically deploying Prometheus instances and managing their life cycle.Additionally,it provides a way to define custom service monitoring purely expressed in Kubernetes idioms.Monitoring truly becomes part of the cluster itself and all implementation details of a distince system being used are abstracted away.
While it's still in an early stage of development,the Operator already hanles serveral aspects of a Prometheus setup that are beyond the scope of this post,such as persistent storage,replication,alerting,and version updates.