SpringBoot 2.x Prometheus + Grafana 整合 Consul实现微服务监控

1、搭建基础环境

1. 安装Consul
2. Docker安装 Prometheus Grafana实现应用可视化监控
3. 配置的 prometheus.yml

global:
  scrape_interval: 10s
  scrape_timeout: 10s
  evaluation_interval: 10m
scrape_configs:
- job_name: prometheus # 直接连接对应的服务
  honor_timestamps: true
  scrape_interval: 5s
  scrape_timeout: 5s
  metrics_path: /monitoring/actuator/prometheus
  scheme: http
  static_configs:
  - targets:
    - 192.168.10.24:8090
- job_name: consul # 连接consul配置
  honor_timestamps: true
  scrape_interval: 10s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  consul_sd_configs:
  - server: 192.168.10.6:8500
    tag_separator: ','
    scheme: http
    allow_stale: true
    refresh_interval: 30s
  relabel_configs: #这一段必须配:不配桩体是DOWN
  - source_labels: [__metrics_path__]
    separator: ;
    regex: /metrics
    target_label: __metrics_path__
    replacement: /actuator/prometheus # 必须这样否则会404
    action: replace

---

1. 问题1： springboot和promethues整合，暴露的metrics是通过/promethues路径访问的，而promethues默认的metrics访问路径（即metrics_path配置项）是/metrics，需要修改。
   如下图：当把鼠标放在某个label上时，显示了Before relabeling的配置，可以看到__metrics_path__=’/metrics’，所以必须通过relabel_configs方式修改为‘/promethues’后，才能让此微服务的状态为UP，不然会因为不符合格式错误而使Endpoint的状态为DOWN。
2. 问题2 ：待补充

---

参考

• 问题1参考：prometheus获取Consul上注册的服务
• 待补充

2、编写测试Demo

2.1 引入pom依赖

	<dependency>
        <groupId>org.springframework.bootgroupId>
        <artifactId>spring-boot-starter-actuatorartifactId>
    dependency>
    <dependency>
        <groupId>org.springframework.bootgroupId>
        <artifactId>spring-boot-starter-webartifactId>
    dependency>
    
    <dependency>
        <groupId>io.micrometergroupId>
        <artifactId>micrometer-registry-prometheusartifactId>
    dependency>
    <dependency>
        <groupId>org.springframework.cloudgroupId>
        <artifactId>spring-cloud-starter-consul-discoveryartifactId>
    dependency>
    <dependency>
        <groupId>org.projectlombokgroupId>
        <artifactId>lombokartifactId>
        <optional>trueoptional>
    dependency>
    <dependency>
        <groupId>org.springframework.bootgroupId>
        <artifactId>spring-boot-starter-testartifactId>
        <scope>testscope>
    dependency>

2.2 添加配置文件application.yml

management:
  endpoint:
    health:
      enabled: true
    metrics:
      enabled: true
  endpoints:
    web:
      exposure:
        include: '*'
  server:
    servlet:
      context-path: /actuator
server:
  port: 9001
spring:
  application:
    name: monitor-test
  cloud:
    consul:
      discovery:
        health-check-interval: 10s
        health-check-path: ${management.server.servlet.context-path}/health
        instance-id: ${spring.application.name}
        prefer-ip-address: true
        register: true
      enabled: true
      host: ${consul-host:192.168.10.6}
      port: ${consul-port:8500}

2.3 创建SpringBoot启动类

@EnableDiscoveryClient
@SpringBootApplication
public class ServiceMonitorApplication {

    public static void main(String[] args) {
        SpringApplication.run(ServiceMonitorApplication.class, args);
    }

}

2.4 启动访问

1. 访问 http://xxx:9001/actuator/prometheus
   结果如下 ：

# HELP jvm_memory_used_bytes The amount of used memory
# TYPE jvm_memory_used_bytes gauge
jvm_memory_used_bytes{area="nonheap",id="Code Cache",} 2.3640256E7
jvm_memory_used_bytes{area="nonheap",id="Metaspace",} 5.0193328E7
jvm_memory_used_bytes{area="nonheap",id="Compressed Class Space",} 6376960.0
jvm_memory_used_bytes{area="heap",id="PS Eden Space",} 1.64805792E8
jvm_memory_used_bytes{area="heap",id="PS Survivor Space",} 1.1772016E7
jvm_memory_used_bytes{area="heap",id="PS Old Gen",} 1.9487024E7
# HELP process_cpu_usage The "recent cpu usage" for the Java Virtual Machine process
# TYPE process_cpu_usage gauge
process_cpu_usage 4.921259842519685E-4
# HELP tomcat_sessions_created_total  
# TYPE tomcat_sessions_created_total counter
tomcat_sessions_created_total 0.0
# HELP tomcat_servlet_request_max_seconds  
# TYPE tomcat_servlet_request_max_seconds gauge
tomcat_servlet_request_max_seconds{name="default",} 0.0
# HELP logback_events_total Number of error level events that made it to the logs
# TYPE logback_events_total counter
logback_events_total{level="error",} 0.0
logback_events_total{level="warn",} 2.0
logback_events_total{level="info",} 60.0
logback_events_total{level="debug",} 0.0
logback_events_total{level="trace",} 0.0
# HELP jvm_buffer_memory_used_bytes An estimate of the memory that the Java virtual machine is using for this buffer pool
# TYPE jvm_buffer_memory_used_bytes gauge
jvm_buffer_memory_used_bytes{id="direct",} 81920.0
jvm_buffer_memory_used_bytes{id="mapped",} 0.0
# HELP process_start_time_seconds Start time of the process since unix epoch.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.557991820293E9
# HELP tomcat_sessions_rejected_total  
# TYPE tomcat_sessions_rejected_total counter
tomcat_sessions_rejected_total 0.0
# HELP jvm_gc_pause_seconds Time spent in GC pause
# TYPE jvm_gc_pause_seconds summary
jvm_gc_pause_seconds_count{action="end of minor GC",cause="Allocation Failure",} 2.0
jvm_gc_pause_seconds_sum{action="end of minor GC",cause="Allocation Failure",} 0.029
# HELP jvm_gc_pause_seconds_max Time spent in GC pause
# TYPE jvm_gc_pause_seconds_max gauge
jvm_gc_pause_seconds_max{action="end of minor GC",cause="Allocation Failure",} 0.0
# HELP tomcat_global_request_seconds  
# TYPE tomcat_global_request_seconds summary
tomcat_global_request_seconds_count{name="http-nio-9001",} 359.0
tomcat_global_request_seconds_sum{name="http-nio-9001",} 2.894
# HELP tomcat_sessions_alive_max_seconds  
# TYPE tomcat_sessions_alive_max_seconds gauge
tomcat_sessions_alive_max_seconds 0.0
# HELP system_load_average_1m The sum of the number of runnable entities queued to available processors and the number of runnable entities running on the available processors averaged over a period of time
# TYPE system_load_average_1m gauge
system_load_average_1m 0.16
# HELP system_cpu_usage The "recent cpu usage" for the whole system
# TYPE system_cpu_usage gauge
system_cpu_usage 0.005412054120541205
# HELP process_files_max The maximum file descriptor count
# TYPE process_files_max gauge
process_files_max 4096.0
# HELP tomcat_global_received_bytes_total  
# TYPE tomcat_global_received_bytes_total counter
tomcat_global_received_bytes_total{name="http-nio-9001",} 0.0
# HELP tomcat_sessions_expired_total  
# TYPE tomcat_sessions_expired_total counter
tomcat_sessions_expired_total 0.0
# HELP system_cpu_count The number of processors available to the Java virtual machine
# TYPE system_cpu_count gauge
system_cpu_count 13.0
# HELP process_files_open The open file descriptor count
# TYPE process_files_open gauge
process_files_open 35.0
# HELP process_uptime_seconds The uptime of the Java virtual machine
# TYPE process_uptime_seconds gauge
process_uptime_seconds 1805.083
# HELP tomcat_cache_access_total  
# TYPE tomcat_cache_access_total counter
tomcat_cache_access_total 0.0
# HELP jvm_memory_committed_bytes The amount of memory in bytes that is committed for the Java virtual machine to use
# TYPE jvm_memory_committed_bytes gauge
jvm_memory_committed_bytes{area="nonheap",id="Code Cache",} 2.3855104E7
jvm_memory_committed_bytes{area="nonheap",id="Metaspace",} 5.300224E7
jvm_memory_committed_bytes{area="nonheap",id="Compressed Class Space",} 6864896.0
jvm_memory_committed_bytes{area="heap",id="PS Eden Space",} 7.07264512E8
jvm_memory_committed_bytes{area="heap",id="PS Survivor Space",} 1.2058624E7
jvm_memory_committed_bytes{area="heap",id="PS Old Gen",} 5.17472256E8
# HELP jvm_threads_daemon The current number of live daemon threads
# TYPE jvm_threads_daemon gauge
jvm_threads_daemon 22.0
# HELP jvm_classes_loaded The number of classes that are currently loaded in the Java virtual machine
# TYPE jvm_classes_loaded gauge
jvm_classes_loaded 9647.0
# HELP jvm_gc_memory_allocated_bytes_total Incremented for an increase in the size of the young generation memory pool after one GC to before the next
# TYPE jvm_gc_memory_allocated_bytes_total counter
jvm_gc_memory_allocated_bytes_total 1.108344832E9
# HELP jvm_memory_max_bytes The maximum amount of memory in bytes that can be used for memory management
# TYPE jvm_memory_max_bytes gauge
jvm_memory_max_bytes{area="nonheap",id="Code Cache",} 2.5165824E8
jvm_memory_max_bytes{area="nonheap",id="Metaspace",} -1.0
jvm_memory_max_bytes{area="nonheap",id="Compressed Class Space",} 1.073741824E9
jvm_memory_max_bytes{area="heap",id="PS Eden Space",} 4.179099648E9
jvm_memory_max_bytes{area="heap",id="PS Survivor Space",} 1.2058624E7
jvm_memory_max_bytes{area="heap",id="PS Old Gen",} 8.413773824E9
# HELP tomcat_cache_hit_total  
# TYPE tomcat_cache_hit_total counter
tomcat_cache_hit_total 0.0
# HELP jvm_threads_peak The peak live thread count since the Java virtual machine started or peak was reset
# TYPE jvm_threads_peak gauge
jvm_threads_peak 27.0
# HELP tomcat_servlet_error_total  
# TYPE tomcat_servlet_error_total counter
tomcat_servlet_error_total{name="default",} 0.0
# HELP tomcat_sessions_active_current  
# TYPE tomcat_sessions_active_current gauge
tomcat_sessions_active_current 0.0
# HELP tomcat_global_error_total  
# TYPE tomcat_global_error_total counter
tomcat_global_error_total{name="http-nio-9001",} 0.0
# HELP jvm_buffer_total_capacity_bytes An estimate of the total capacity of the buffers in this pool
# TYPE jvm_buffer_total_capacity_bytes gauge
jvm_buffer_total_capacity_bytes{id="direct",} 81920.0
jvm_buffer_total_capacity_bytes{id="mapped",} 0.0
# HELP tomcat_threads_current  
# TYPE tomcat_threads_current gauge
tomcat_threads_current{name="http-nio-9001",} 10.0
# HELP jvm_gc_memory_promoted_bytes_total Count of positive increases in the size of the old generation memory pool before GC to after GC
# TYPE jvm_gc_memory_promoted_bytes_total counter
jvm_gc_memory_promoted_bytes_total 16384.0
# HELP jvm_threads_live The current number of live threads including both daemon and non-daemon threads
# TYPE jvm_threads_live gauge
jvm_threads_live 25.0
# HELP tomcat_sessions_active_max  
# TYPE tomcat_sessions_active_max gauge
tomcat_sessions_active_max 0.0
# HELP tomcat_global_request_max_seconds  
# TYPE tomcat_global_request_max_seconds gauge
tomcat_global_request_max_seconds{name="http-nio-9001",} 0.134
# HELP jvm_buffer_count An estimate of the number of buffers in the pool
# TYPE jvm_buffer_count gauge
jvm_buffer_count{id="direct",} 10.0
jvm_buffer_count{id="mapped",} 0.0
# HELP tomcat_threads_config_max  
# TYPE tomcat_threads_config_max gauge
tomcat_threads_config_max{name="http-nio-9001",} 200.0
# HELP tomcat_servlet_request_seconds  
# TYPE tomcat_servlet_request_seconds summary
tomcat_servlet_request_seconds_count{name="default",} 0.0
tomcat_servlet_request_seconds_sum{name="default",} 0.0
# HELP jvm_gc_max_data_size_bytes Max size of old generation memory pool
# TYPE jvm_gc_max_data_size_bytes gauge
jvm_gc_max_data_size_bytes 0.0
# HELP http_server_requests_seconds  
# TYPE http_server_requests_seconds summary
http_server_requests_seconds_count{exception="None",method="GET",status="200",uri="/actuator/prometheus",} 180.0
http_server_requests_seconds_sum{exception="None",method="GET",status="200",uri="/actuator/prometheus",} 0.941796958
http_server_requests_seconds_count{exception="None",method="GET",status="200",uri="/actuator/health",} 179.0
http_server_requests_seconds_sum{exception="None",method="GET",status="200",uri="/actuator/health",} 1.48098511
# HELP http_server_requests_seconds_max  
# TYPE http_server_requests_seconds_max gauge
http_server_requests_seconds_max{exception="None",method="GET",status="200",uri="/actuator/prometheus",} 0.005677743
http_server_requests_seconds_max{exception="None",method="GET",status="200",uri="/actuator/health",} 0.009210879
# HELP jvm_classes_unloaded_total The total number of classes unloaded since the Java virtual machine has started execution
# TYPE jvm_classes_unloaded_total counter
jvm_classes_unloaded_total 0.0
# HELP tomcat_global_sent_bytes_total  
# TYPE tomcat_global_sent_bytes_total counter
tomcat_global_sent_bytes_total{name="http-nio-9001",} 1674898.0
# HELP jvm_gc_live_data_size_bytes Size of old generation memory pool after a full GC
# TYPE jvm_gc_live_data_size_bytes gauge
jvm_gc_live_data_size_bytes 0.0
# HELP tomcat_threads_busy  
# TYPE tomcat_threads_busy gauge
tomcat_threads_busy{name="http-nio-9001",} 1.0

2. 访问 Prometheus
   http://xxx:9090/targets
   结果如下：
   
3. 访问 Grafana
   http://xxx:3000 默认用户名/密码 admin/admin
   
   结果图，如下：
   

仅供参考：代码service-monitor

3、参考

1. SpringBoot 2.x Prometheus Grafana实现应用可视化监控
2. 官方Consul配置文档
3. Prometheus 通过consul动态修改Targets接入