Prometheus数据采集

1. Prometheus

1.1 Exporter

启动prometheus后，会启动一个3000端口的exporter，使用如下命令测试exporter是否正常

curl -I http://22.22.3.244:3000/metrics

1.2 Targets

在prometheus的scrape_configs中增加job_name='prometheus'

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
    - targets: ['localhost:9090']

重启prometheus

 ps -ef|grep prometheus|grep -v grep|awk -F' ' '{print $2}'|xargs kill -1

1.3 Grafana

已经存在dashboard

image.png

2. Linux

2.1 Exporter

安装并启动node_exporter,缺省端口为9100，可以使用--web.listen-address修改监听地址及端口

/root/node_exporter-0.18.1.linux-amd64/node_exporter --web.listen-address=:9200

使用如下命令测试exporter是否正常

curl -I http://node01:9200/metrics

2.2 Targets

在prometheus的scrape_configs中增加job_name='nodes'

  - job_name: 'nodes'
    static_configs:
#    - targets: ['mgmt01:9200']
    - targets: ['etcd01:9200']
    - targets: ['etcd02:9200']
    - targets: ['etcd03:9200']
    - targets: ['master01:9200']
    - targets: ['master02:9200']

重启prometheus

 ps -ef|grep prometheus|grep -v grep|awk -F' ' '{print $2}'|xargs kill -1

2.3 Grafana

下载并导入dashboard模板
https://grafana.com/grafana/dashboards/8919

image.png

3. Etcd

3.1 Exporter

编辑 /etc/etcd/etcd.conf ，去掉ETCD_METRICS="basic"前面的注释，重新启动etcd
测试etcd exporter工作是否正常，由于etcd是https的，需要携带client证书

curl -I --cert /etc/kubernetes/pki/etcd/healthcheck-client.crt --key /etc/kubernetes/pki/etcd/healthcheck-client.key -k   https://etcd01:2379/metrics

3.2 Targets

在prometheus的scrape_configs中增加job_name='etcd'

  - job_name: 'etcd'
    static_configs:
    - targets: ["22.22.3.231:2379","22.22.3.232:2379","22.22.3.233:2379"]
      scheme: https
    tls_config:
      - ca_file: /etc/kubernetes/pki/etcd/ca.crt
      - key_file: /etc/kubernetes/pki/etcd/healthcheck-client.key
      - cert_file: /etc/kubernetes/pki/etcd/healthcheck-client.crt

重启prometheus

 ps -ef|grep prometheus|grep -v grep|awk -F' ' '{print $2}'|xargs kill -1

3.3 Grafana

下载并导入dashboard模板
https://grafana.com/api/dashboards/3070/revisions/3/download

image.png

4. Mysql

4.1 Exporter

到项目主页
https://github.com/prometheus/mysqld_exporter
下载mysqld_exporter-0.12.1.linux-amd64.tar.gz

CREATE USER 'exporter'@'localhost' IDENTIFIED BY 'exporter123' WITH MAX_USER_CONNECTIONS 10;
GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'exporter'@'localhost';

解压并启动exporter，可使用--web.listen-address=":9105" 修改监听地址及端口

tar -zxvf mysqld_exporter-0.12.1.linux-amd64.tar.gz
export DATA_SOURCE_NAME='exporter:exporter123@(localhost:3306)/'
cd mysqld_exporter-0.12.1.linux-amd64
nohup ./mysqld_exporter  --web.listen-address=":9105"   2>&1 &

测试mysqld exporter工作是否正常，mgmt01为exporter所在机器

 curl -I  http://mgmt01:9105/metrics

4.2 Targets

在prometheus的scrape_configs中增加job_name='mysqld'

  - job_name: 'mysqld'
    static_configs:
    - targets: ["22.22.3.244:9105"]

重启prometheus

 ps -ef|grep prometheus|grep -v grep|awk -F' ' '{print $2}'|xargs kill -1

4.3 Grafana

下载并导入dashboard模板
https://grafana.com/api/dashboards/6239/revisions/1/download

image.png

5. Ceph

5.1 Exporter

从ceph主机上复制配置及key文件到监控机上,并验证访问正常

scp ceph01:/etc/ceph/ceph.client.admin.keyring  /etc/ceph/
scp ceph01:/etc/ceph/ceph.conf  /etc/ceph/
ceph -s
  cluster:
    id:     0cd78d03-771a-4c45-99eb-49b200ae7338
    health: HEALTH_WARN
            application not enabled on 1 pool(s)

下载exporter镜像digitalocean/ceph_exporter，使用docker的方式运行ceph exporter

docker pull digitalocean/ceph_exporter
docker run -v /etc/ceph:/etc/ceph --net=host -p=9128:9128  -d  digitalocean/ceph_exporter  

测试ceph exporter工作是否正常，mgmt01为exporter所在机器

 curl -I  http://mgmt01:9128/metrics

5.2 Targets

在prometheus的scrape_configs中增加job_name='mysqld'

  - job_name: 'ceph'
    static_configs:
    - targets: ["22.22.3.244:9128"]

重启prometheus

 ps -ef|grep prometheus|grep -v grep|awk -F' ' '{print $2}'|xargs kill -1

5.3 Grafana

下载并导入dashboard模板
https://grafana.com/api/dashboards/917/revisions/1/download

image.png

6. SNMP

6.1 Exporter

下载并启动snmp_exporter
https://github.com/prometheus/snmp_exporter/releases

wget https://github.com/prometheus/snmp_exporter/releases/download/v0.15.0/snmp_exporter-0.15.0.linux-amd64.tar.gz
 tar -zxvf  snmp_exporter-0.15.0.linux-amd64.tar.gz
cd snmp_exporter-0.15.0.linux-amd64

snmp的认证信息是通过snmp_exporter的配置文件snmp.yml来配置的，在module if_mib下增加如下配置，

if_mib:
  version: 2
  auth:
    community: TEST_RO

启动snmp_exporter

nohup ./snmp_exporter 2>&1 &

测试SNMP exporter是否工作正常，mgmt01为exporter所在机器

curl "http://mgmt01:9116/snmp?target=30.3.1.86" -I

5.2 Targets

在prometheus的scrape_configs中增加job_name='snmp'

  - job_name: 'snmp'
    static_configs:
      - targets:
        - 30.3.1.86
    metrics_path: /snmp
    params:
      module: [if_mib]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 22.22.3.244:9116

重启prometheus

 ps -ef|grep prometheus|grep -v grep|awk -F' ' '{print $2}'|xargs kill -1

image.png

5.3 Grafana

下载并导入dashboard模板
https://grafana.com/grafana/dashboards/1124

image.png