9-4 Prometheus监控案例Tomcat, Redis, Mysql, Haproxy, Nginx, Ingress

前言

Prometheus除了使用node-exporter监控系统指标外，还支持多种多样的应用服务监控。包括数据库，硬件，消息服务，存储，网站服务，APIs，日志等等。下面我们介绍最常用几种应用监控：Tomcat, Redis, Mysql, Haproxy, Nginx, Ingress。

---

Tomcat

监控 tomcat 的活跃连接数、堆栈内存等信息。

https://github.com/nlighten/tomcat_exporter

tomcat镜像

镜像在tomcat:8.5.73基础上，添加tomcat_exporter相关jar包和war包：

vim Dockerfile

FROM tomcat:8.5.73-jdk11-corretto 
RUN mkdir -p /data/tomcat/webapps

ADD server.xml /usr/local/tomcat/conf/server.xml 
ADD myapp /data/tomcat/webapps/myapp
ADD metrics.war /data/tomcat/webapps 
ADD simpleclient-0.8.0.jar  /usr/local/tomcat/lib/
ADD simpleclient_common-0.8.0.jar /usr/local/tomcat/lib/
ADD simpleclient_hotspot-0.8.0.jar /usr/local/tomcat/lib/
ADD simpleclient_servlet-0.8.0.jar /usr/local/tomcat/lib/
ADD tomcat_exporter_client-0.0.12.jar /usr/local/tomcat/lib/

EXPOSE 8080 8443 8009

构建，上传，测试 tomcat镜像：

docker build -t easzlab.io.local:5000/prom/tomcat-exporter:v2 .
docker push easzlab.io.local:5000/prom/tomcat-exporter:v2
docker run -d -p 8081:8080 easzlab.io.local:5000/prom/tomcat-exporter:v2

tomcat-deployment

跟平时部署tomcat一样，只是镜像添加了tomcat_exporter：

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tomcat-deployment
  namespace: default
spec:
  selector: 
    matchLabels: 
     app: tomcat
  replicas: 1
  template:
    metadata:
      labels:
        app: tomcat
      annotations:
        prometheus.io/scrape: 'true'
    spec:
      containers:
      - name: tomcat
        image: easzlab.io.local:5000/prom/tomcat-exporter:v2
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
        securityContext: 
          privileged: true

---
kind: Service
apiVersion: v1
metadata:
  annotations:
    prometheus.io/scrape: 'true'
  name: tomcat-service
spec:
  selector:
    app: tomcat
  ports:
  - nodePort: 31080
    port: 80
    protocol: TCP
    targetPort: 8080
  type: NodePort

访问网页：http://192.168.100.194:31080/myapp/
访问指标：http://192.168.100.194:31080/metrics/

Prometheus采集tomcat

在Prometheus server添加job：

vim /apps/prometheus/prometheus.yml

  - job_name: 'tomcat-exporter'
    static_configs:
      - targets: ['192.168.100.194:31080']

重载后发现tomcat-exporter：

Grafana展示tomcat

tomcat-exporter模板下载地址：

https://github.com/nlighten/tomcat_exporter/tree/master/dashboard

Redis

通过 redis_exporter 监控 redis 服务状态。

https://github.com/oliver006/redis_exporter

redis_exporter镜像

redis_exporter是基于go的Prometheus Exporter ，支持的版本 Redis 2.x, 3.x, 4.x, 5.x, 6.x, and 7.x。

镜像可在hub.docker.com直接下载：

docker pull oliver006/redis_exporter

redis-deployment

创建1个Pod内含2个容器，一个容器是redis，另一个就是redis_exporter。

apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis
  namespace: studylinux-net 
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      containers:
      - name: redis
        image: redis:4.0.14 
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
        ports:
        - containerPort: 6379
      - name: redis-exporter
        image: oliver006/redis_exporter:latest
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
        ports:
        - containerPort: 9121

redis-service

redis-service针对redis的服务，scrape注释要false：

kind: Service
apiVersion: v1
metadata:
  annotations:
    prometheus.io/scrape: 'false'
  name: redis-redis-service
  namespace: studylinux-net 
spec:
  selector:
    app: redis
  ports:
  - nodePort: 31081
    name: redis
    port: 6379
    protocol: TCP
    targetPort: 6379
  type: NodePort

exporter-service针对redis-exporter的服务，scrape注释要true，同时标注port为9121。

kind: Service
apiVersion: v1
metadata:
  annotations:
    prometheus.io/scrape: 'true'
    prometheus.io/port: "9121"
  name: redis-exporter-service
  namespace: studylinux-net 
spec:
  selector:
    app: redis
  ports:
  - nodePort: 31082
    name: prom
    port: 9121
    protocol: TCP
    targetPort: 9121
  type: NodePort

连接redis服务端口： redis-cli -h 192.168.100.196 -p 31081
redis-exporter指标： http://192.168.100.196:31082/metrics

Prometheus采集redis

目标选择上述的URL地址：

vim /apps/prometheus/prometheus.yml

  - job_name: 'redis-exporter'
    static_configs:
      - targets: ['192.168.100.196:31082']

Grafana展示redis

可在官网查找redis相关的模板

https://grafana.com/grafana/dashboards/

推荐模板ID：11835，14615

Mysql

通过 mysqld_exporter 监控 MySQL 服务的运行状态

https://github.com/prometheus/mysqld_exporter

mysqld_exporter下载

mysqld_exporter 采用二进制安装方式，在官网下载文件：

wget https://github.com/prometheus/mysqld_exporter/releases/download/v0.14.0/mysqld_exporter-0.14.0.linux-amd64.tar.gz

安装mysql

使用yum安装mariadb，创建给exporter的账号密码，并赋予权限：

yum install mariadb-server
systemctl enable --now mariadb.service
mysql

CREATE USER 'mysql_exporter'@'localhost' IDENTIFIED BY 'pwisnotweak';
GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'mysql_exporter'@'localhost';

mysql -umysql_exporter -ppwisnotweak -hlocalhost

安装mysqld_exporter

# 解压下载好的安装包
tar zxvf mysqld_exporter-0.14.0.linux-amd64.tar.gz

# 复制二进制文件到PATH目录
cp mysqld_exporter-0.14.0.linux-amd64/mysqld_exporter /usr/local/bin/

# mysql免密登录
vim /root/.my.cnf
[client]
user=mysql_exporter
password=pwisnotweak

# 登录查看用户状态
mysql
MariaDB [(none)]> status

创建服务，免密登录运行

vim /etc/systemd/system/mysqld_exporter.service

[Unit]
Description=Prometheus Node Exporter
After=network.target

[Service]
ExecStart=/usr/local/bin/mysqld_exporter --config.my-cnf=/root/.my.cnf

[Install]
WantedBy=multi-user.target

应用服务，开机自启动：

systemctl daemon-reload
systemctl restart mysqld_exporter
systemctl enable mysqld_exporter
systemctl status mysqld_exporter

查出mysqld_exporter服务端口为9104：

# 注意进程名太长会切掉，搜索时删除末尾r
netstat -lntp | grep mysqld_exporte
tcp6       0      0 :::9104                 :::*                    LISTEN      2653/mysqld_exporte

mysql连接本地端口： mysql -uroot -proot -P 3306
mysql-exporter指标： http://192.168.100.195:9104/metrics

Prometheus采集mysql

添加metrics的地址：

vim /apps/prometheus/prometheus.yml

  - job_name: 'mysql-exporter'
    static_configs:
      - targets: ['192.168.100.195:9104']

Grafana展示mysql

可在官网查找redis相关的模板

https://grafana.com/grafana/dashboards/

推荐模板ID：13106，11323

Haproxy

通过 haproxy_exporter 监控 haproxy。

https://github.com/prometheus/haproxy_exporter

haproxy_exporter 采用二进制安装方式，在官网下载文件：

wget https://github.com/prometheus/haproxy_exporter/releases/download/v0.13.0/haproxy_exporter-0.13.0.linux-amd64.tar.gz

安装haproxy

安装haproxy软件，启用status状态监控：

yum install haproxy

vim /etc/haproxy/haproxy.cfg

listen stats
  bind :9999	# haproxy-dashboard端口
  stats enable	# 启用状态监控
  stats uri /haproxy-status  # 监控页url
  stats realm HAPorxy\ Stats\ Page	# 监控内容
  stats auth admin:123456	# 登录账密

如果服务起不来，先修改selinux的haproxy设置：

setsebool -P haproxy_connect_any=1
systemctl restart haproxy.service

输入预定的账密admin和123456

haproxy-dashboard监控页面：

安装haproxy_exporter

二进制安装：

wget https://github.com/prometheus/haproxy_exporter/releases/download/v0.13.0/haproxy_exporter-0.13.0.linux-amd64.tar.gz
tar zxvf haproxy_exporter-0.13.0.linux-amd64.tar.gz
cp haproxy_exporter-0.13.0.linux-amd64/haproxy_exporter /usr/local/bin/

查找socket文件，启动应用：

# 找出socket文件路径
cat /etc/haproxy/haproxy.cfg | grep socket
# turn on stats unix socket
stats socket /var/lib/haproxy/stats

# 启动应用，使用socket文件连接
haproxy_exporter --web.listen-address=":9101" --haproxy.scrape-uri=unix:/var/lib/haproxy/stats

haproxy_exporter默认端口为9101：

Prometheus采集haproxy

添加haproxy的工作：

vim /apps/prometheus/prometheus.yml

  - job_name: 'haproxy-exporter'
    static_configs:
      - targets: ['192.168.100.180:9101']

Grafana展示haproxy

可在官网查找haproxy相关的模板：

https://grafana.com/grafana/dashboards/

推荐模板ID：367，2428

Nginx

通过 prometheus 监控 nginx，需要在编译安装 nginx 的时候添加 nginx-module-vts 模块：

https://github.com/vozlt/nginx-module-vts

编译安装Nginx

安装c语言编译器，和相关组件依赖。

yum -y install gcc
yum -y install make zlib-devel gcc-c++ libtool openssl openssl-devel

github克隆vts模块，官网下载源码包，编译时添加vts模块：

git clone https://github.com/vozlt/nginx-module-vts.git

wget http://nginx.org/download/nginx-1.20.2.tar.gz
tar zxvf nginx-1.20.2.tar.gz
cd nginx-1.20.2

# 编译设置
./configure \
--prefix=/apps/nginx \
--with-http_ssl_module \
--with-http_v2_module \
--with-http_realip_module \
--with-http_stub_status_module \
--with-http_gzip_static_module \
--with-pcre \
--with-file-aio \
--with-stream \
--with-stream_ssl_module \
--with-stream_realip_module \
--add-module=/root/nginx-module-vts/

# 编译安装
make
make install

配置与启动

修改配置文件，vim /apps/nginx/conf/nginx.conf

# 核心数
worker_processes  1;

events {
	# 最大连接数
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;

	# 启用状态页
    vhost_traffic_status_zone;

    server {
    	# 本地80端口
        listen       80;
        server_name  localhost;
		
		# 首页设置
        location / {
            root   html;
            index  index.html index.htm;
        }
       
       	# 状态页设置
        location /status {
            vhost_traffic_status_display;
            vhost_traffic_status_display_format html;    
        }
		
		# 错误页设置
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}

语法检测与配置测试

/apps/nginx/sbin/nginx -t

nginx: the configuration file /apps/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /apps/nginx/conf/nginx.conf test is successful

# 启动程序，自动后台运行
/apps/nginx/sbin/nginx

# 查看后台进程
ps aux | grep nginx
root       6903  0.0  0.1  47396  1208 ?        Ss   20:02   0:00 nginx: master process /apps/nginx/sbin/nginx
nobody     6904  0.0  0.1  47796  1956 ?        S    20:02   0:00 nginx: worker process

Nginx状态页面：

安装nginx_exporter

二进制安装，设定抓取uri和格式：

wget https://github.com/hnlq715/nginx-vts-exporter/releases/download/v0.10.3/nginx-vts-exporter-0.10.3.linux-amd64.tar.gz

tar zxvf nginx-vts-exporter-0.10.3.linux-amd64.tar.gz 
cp nginx-vts-exporter /usr/local/bin/

# 后台运行，抓取源为status页面，格式为json，正常日志std.log，错误日志err.log 
nohup nginx-vts-exporter -nginx.scrape_uri http://192.168.100.180/status/format/json 1>std.log 2>err.log &

nginx_exporter默认端口9913

Prometheus采集nginx

vim /apps/prometheus/prometheus.yml

  - job_name: 'nginx-exporter'
    static_configs:
      - targets: ['192.168.100.180:9913']     

systemctl reload prometheus.service

Grafana展示nginx

可在官网查找nginx相关的模板：

https://grafana.com/grafana/dashboards/

推荐模板ID：2949

Ingress

通过 ingress-nginx-controller 监控K8s集群的Ingress状态。

K8s部署ingress，端口为10254：

vim ingress-nginx-controller-v1.3.0_deployment.yaml
...
...
...
    # kubernetes v1.20 stable，appProtocol字段提供了一种为每个Service端口指定应用协议的方式,此字段的取值会被映射到对应的Endpoints
  - appProtocol: http 
    name: prometheus-metrics-port
    port: 10254
    protocol: TCP
    # ingress-nginx-controller内置的指标数据采集端口10254
    targetPort: 10254 
    nodePort: 31254
...
...
...

访问任一节点IP和31254端口的metrics页面：

Prometheus采集ingress

vim /apps/prometheus/prometheus.yml

  - job_name: 'ingress-exporter'
    static_configs:
      - targets: ['192.168.100.195:31254']   

systemctl reload prometheus.service   

Grafana展示ingress

首先，创建2条规则，再频繁访问这2个域名，丰富数据：

kubectl apply -f ingress_multi-host.yaml

curl www.app1.com:35080/myapp/
curl www.app2.com:35080/myapp/

可在官网查找nginx相关的模板：

https://grafana.com/grafana/dashboards/

推荐模板ID：9614