APM监控--（四）Pinpoint扩展报警--超详细，各种问题解决之后的顺畅版本
部署完成pinpoint之后，并没有开放报警功能，而报警是每个公司都必要的功能，以下就跟着我一起开始开发报警功能并部署到服务器。
1，下载源码包
地址：https://github.com/naver/pinpoint/releases
选择对应版本源码包，本次为1.7.1，单击下载
2，编译
源码在maven端增加了很多检查和限制，保证jdk版本1.8，maven为3.1.1就可以去除这些限制，修改后pom文件为：
4.0.0
Naver Corporation
http://www.navercorp.com/
com.navercorp.pinpoint
pinpoint
1.7.1
pinpoint
pom
cloudera
https://repository.cloudera.com/artifactory/cloudera-repos/
spring-maven-repository
Spring Framework Maven Release Repository
http://maven.springframework.org/release/
spring-maven-release-remote
Spring Framework Maven Release Remote Repository
http://repo.spring.io/libs-release-remote/
bintray
bintray
http://jcenter.bintray.com
jboss-3rd-party-releases
Jboss Third Party Repository
https://repository.jboss.org/nexus/content/repositories/thirdparty-releases/
central
Maven central plugin repository
http://repo1.maven.org/maven2/
annotations
agent
bootstrap-core
bootstrap-core-optional
bootstrap
collector
commons
commons-hbase
commons-server
tools
plugins
profiler
profiler-optional
profiler-test
rpc
thrift
test
web
hbase
flink
UTF-8
1.8
1.7.21
3.1.2
8.1.12.v20130726
2.6.7
4.3
2.4.2
${basedir}/clover.license
4.3.8.RELEASE
3.0.7.RELEASE
4.2.2.RELEASE
0.4.3
2.1.7.1
java16
junit
junit
test
org.hamcrest
hamcrest-library
test
org.mockito
mockito-core
test
com.navercorp.pinpoint
pinpoint-annotations
${project.version}
com.navercorp.pinpoint
pinpoint-commons
${project.version}
com.navercorp.pinpoint
pinpoint-commons-server
${project.version}
com.navercorp.pinpoint
pinpoint-commons-hbase
${project.version}
com.navercorp.pinpoint
pinpoint-rpc
${project.version}
com.navercorp.pinpoint
pinpoint-thrift
${project.version}
com.navercorp.pinpoint
pinpoint-bootstrap-core
${project.version}
com.navercorp.pinpoint
pinpoint-bootstrap-core-optional
${project.version}
com.navercorp.pinpoint
pinpoint-bootstrap
${project.version}
com.navercorp.pinpoint
pinpoint-profiler
${project.version}
com.navercorp.pinpoint
pinpoint-profiler-optional
pom
${project.version}
com.navercorp.pinpoint
pinpoint-profiler-test
${project.version}
com.navercorp.pinpoint
pinpoint-collector
war
${project.version}
com.navercorp.pinpoint
pinpoint-collector
classes
${project.version}
com.navercorp.pinpoint
pinpoint-web
${project.version}
war
com.navercorp.pinpoint
pinpoint-web
classes
${project.version}
com.navercorp.pinpoint
pinpoint-plugins
pom
${project.version}
com.navercorp.pinpoint
pinpoint-plugins
zip
${project.version}
com.navercorp.pinpoint
pinpoint-tools
${project.version}
com.navercorp.pinpoint
pinpoint-test
${project.version}
com.navercorp.pinpoint
pinpoint-flink
${project.version}
org.springframework
spring-core
${spring.version}
org.springframework
spring-beans
${spring.version}
org.springframework
spring-context
${spring.version}
org.springframework
spring-orm
${spring.version}
org.springframework
spring-test
${spring.version}
com.jayway.jsonpath
json-path
1.2.0
org.springframework
spring-web
${spring.version}
org.springframework
spring-webmvc
${spring.version}
org.springframework
spring-websocket
${spring.version}
org.springframework
spring-jdbc
${spring.version}
org.springframework
spring-tx
${spring.version}
org.springframework
spring-oxm
${spring.version}
org.springframework
spring-instrument
${spring.version}
org.springframework
spring-context-support
${spring.version}
org.springframework
spring-aop
${spring.version}
org.springframework
spring-aspects
${spring.version}
io.dropwizard.metrics
metrics-core
${metrics.version}
io.dropwizard.metrics
metrics-jvm
${metrics.version}
io.dropwizard.metrics
metrics-servlets
${metrics.version}
com.fasterxml.jackson.core
jackson-core
${fastxml.jackson.version}
com.fasterxml.jackson.core
jackson-annotations
${fastxml.jackson.version}
com.fasterxml.jackson.core
jackson-databind
${fastxml.jackson.version}
org.apache.hbase
hbase-shaded-client
1.2.4
com.sematext.hbasewd
hbasewd
0.1.0
org.slf4j
slf4j-api
${slf4j.version}
org.slf4j
jcl-over-slf4j
${slf4j.version}
org.slf4j
slf4j-log4j12
${slf4j.version}
log4j
log4j
1.2.16
commons-dbcp
commons-dbcp
1.4
org.apache.commons
commons-dbcp2
2.1.1
org.mybatis
mybatis
3.2.7
org.mybatis
mybatis-spring
1.2.2
org.mariadb.jdbc
mariadb-java-client
1.3.4
mysql
mysql-connector-java
5.1.20
org.postgresql
postgresql
9.4.1207
net.sourceforge.jtds
jtds
1.2.8
com.ning
async-http-client
1.8.3
org.asynchttpclient
async-http-client
2.0.32
io.netty
netty
3.10.6.Final
io.netty
netty-all
4.0.18.Final
org.apache.tomcat
catalina
6.0.43
org.apache.tomcat
coyote
6.0.43
com.navercorp.arcus
arcus-java-client
1.8.1
net.sf.ehcache
ehcache-core
2.6.9
net.sf.ehcache
ehcache
1.6.2
com.datastax.cassandra
cassandra-driver-core
${cassandra.driver.version}
org.springframework.data
spring-data-redis
1.1.1.RELEASE
org.apache.httpcomponents
httpclient
${httpcomponents.version}
org.apache.httpcomponents
httpcore
${httpcomponents.version}
org.apache.httpcomponents
httpcore-nio
${httpcomponents.version}
org.apache.httpcomponents
httpasyncclient
4.0
commons-httpclient
commons-httpclient
3.1
com.google.http-client
google-http-client
1.20.0
com.squareup.okhttp
okhttp
2.5.0
com.squareup.okhttp3
okhttp
3.8.1
redis.clients
jedis
${jedis.version}
commons-io
commons-io
2.5
org.apache.commons
commons-lang3
3.5
commons-collections
commons-collections
3.2.2
commons-logging
commons-logging
1.2
provided
commons-codec
commons-codec
1.10
commons-lang
commons-lang
2.6
commons-beanutils
commons-beanutils
1.9.3
javax.transaction
jta
1.1
javax.mail
mail
1.4.7
org.javassist
javassist
3.21.0-GA
cglib
cglib-nodep
2.2
org.ow2.asm
asm-debug-all
5.2
org.aspectj
aspectjweaver
1.8.10
org.apache.tomcat
servlet-api
6.0.35
javax.servlet
javax.servlet-api
3.0.1
javax.servlet
jstl
1.1.0
javax.servlet
jsp-api
2.0
taglibs
standard
1.1.2
junit
junit
4.12
org.hamcrest
hamcrest-library
1.3
org.mockito
mockito-core
2.7.22
org.dbunit
dbunit
2.4.3
org.apache.zookeeper
zookeeper
3.4.5
org.jboss.netty
netty
org.apache.curator
curator-test
2.6.0
zookeeper
org.apache.zookeeper
org.apache.thrift
libthrift
0.10.0
org.apache.httpcomponents
httpclient
org.apache.httpcomponents
httpcore
org.slf4j
slf4j-api
com.google.guava
guava
20.0
com.google.inject
guice
4.1.0
com.alibaba
dubbo
2.5.3
org.apache.activemq
activemq-client
5.13.2
org.apache.cxf
cxf-rt-frontend-jaxws
2.7.18
org.apache.cxf
cxf-rt-transports-http
2.7.18
com.netflix.hystrix
hystrix-core
1.5.12
io.vertx
vertx-core
3.3.3
io.reactivex
rxjava
1.2.0
${basedir}/src/main/java
**/*.java
true
${basedir}/src/main/resources
${basedir}/src/test/java
**/*.java
true
${basedir}/src/test/resources
org.apache.maven.plugins
maven-war-plugin
2.6
true
classes
${project.build.directory}/deploy
org.apache.maven.plugins
maven-jar-plugin
2.6
org.apache.maven.plugins
maven-resources-plugin
2.7
${encoding}
org.apache.maven.plugins
maven-compiler-plugin
3.1
${jdk.version}
${jdk.version}
true
com.atlassian.maven.plugins
maven-clover2-plugin
4.0.6
${encoding}
${jdk.version}
true
true
false
${cloverLicenseLocation}
false
false
com/navercorp/pinpoint/thrift/dto/**/*.java
com/navercorp/pinpoint/common/util/apache/**/*.java
com/navercorp/pinpoint/common/util/jsr166/**/*.java
com/navercorp/pinpoint/profiler/util/jdk/**/*.java
org.apache.maven.plugins
maven-pmd-plugin
3.6
true
${encoding}
${jdk.version}
com/navercorp/pinpoint/thrift/dto/**/*.java
com/navercorp/pinpoint/common/util/apache/**/*.java
com/navercorp/pinpoint/common/util/jsr166/**/*.java
com/navercorp/pinpoint/profiler/util/jdk/**/*.java
com.cj.jshintmojo
jshint-maven-plugin
1.6.0
sub, boss, newcap, loopfunc
src/main/webapp
src/main/webapp/components
checkstyle
${project.build.directory}/jshint.xml
false
org.owasp
dependency-check-maven
3.0.2
false
xml
org.codehaus.mojo
animal-sniffer-maven-plugin
1.15
org.codehaus.mojo.signature
${sniffer.artifactid}
1.0
sun.*
ensure-java-${jdk.version}-class-library
test
check
org.apache.maven.plugins
maven-jxr-plugin
false
true
${encoding}
${encoding}
Pinpoint ${project.version} Cross Reference
Pinpoint ${project.version} Cross Reference
2.5
org.codehaus.mojo
findbugs-maven-plugin
findbugs-exclude.xml
true
${project.build.directory}/findbugs
3.0.3
org.apache.maven.plugins
maven-pmd-plugin
3.6
true
${encoding}
${jdk.version}
com/navercorp/pinpoint/thrift/dto/**/*.java
com/navercorp/pinpoint/common/util/apache/**/*.java
com/navercorp/pinpoint/common/util/jsr166/**/*.java
com/navercorp/pinpoint/profiler/util/jdk/**/*.java
com.atlassian.maven.plugins
maven-clover2-plugin
4.0.6
true
clover.license
${encoding}
${jdk.version}
com/navercorp/pinpoint/thrift/dto/**/*.java
com/navercorp/pinpoint/common/util/apache/**/*.java
com/navercorp/pinpoint/common/util/jsr166/**/*.java
com/navercorp/pinpoint/profiler/util/jdk/**/*.java
local
true
local
release
release
klocwork
com.klocwork.ps
kwmaven
2.1.1
${KWINJECT_OUT_PATH}
validate
run
然后根目录执行 mvn clean install -Dmaven.test.skip=true
通过install命令将相关包安装进本地maven仓库之后，后期开发才没有问题。
3，新增报警类
开发工具加载web文件夹
web包下pom文件新增发送邮件依赖
com.sun.mail
javax.mail
1.5.6
然后新增AlarmMessageSenderImple类，全路径如下:com.navercorp.pinpoint.web.alarm.AlarmMessageSenderImple
源码：
package com.navercorp.pinpoint.web.alarm;
import com.navercorp.pinpoint.web.alarm.checker.AlarmChecker;
import com.navercorp.pinpoint.web.alarm.mail.MailSenderInfo;
import com.navercorp.pinpoint.web.alarm.mail.SimpleMailSender;
import com.navercorp.pinpoint.web.service.UserGroupService;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import java.util.List;
/**
* Created by 许恕 on 2018/4/3.
*/
public class AlarmMessageSenderImple implements AlarmMessageSender{
private final Logger logger = LoggerFactory.getLogger(this.getClass());
@Autowired
private UserGroupService userGroupService;
@Override
public void sendSms(AlarmChecker checker, int sequenceCount) {
List receivers = userGroupService.selectPhoneNumberOfMember(checker.getuserGroupId());
if (receivers.size() == 0) {
return;
}
List sms = checker.getSmsMessage();
for (String message : sms) {
logger.info("send SMS : {}", message);
// TODO Implement logic for sending SMS
}
}
@Override
public void sendEmail(AlarmChecker checker, int sequenceCount) {
List receivers = userGroupService.selectEmailOfMember(checker.getuserGroupId());
if (receivers.size() == 0) {
return;
}
for (String emailId : receivers) {
/* 这个类主要是设置邮件
* mailServerHost:邮箱服务器
* userName:发送方邮件用户名
* password:发送方邮件密码
* fromAddress:发送方邮件用户名
* toAddress:接收方邮件用户名
*/
MailSenderInfo mailInfo = new MailSenderInfo();
mailInfo.setMailServerHost("smtp.21cn.com");
mailInfo.setMailServerPort("25");
mailInfo.setValidate(true);
mailInfo.setUserName("[email protected]");
mailInfo.setPassword("yourpass");
mailInfo.setFromAddress("[email protected]");
mailInfo.setToAddress(emailId);
mailInfo.setSubject("Pinpoint报警");
mailInfo.setContent(checker.getEmailMessage());
// 这个类主要来发送邮件
SimpleMailSender sms = new SimpleMailSender();
boolean con = false;
try {
con = sms.sendTextMail(mailInfo);// 发送文体格式
} catch (Exception e) {
logger.error("=AlarmMessageSenderImple.sendEmail=>错误原因是:" + e.getMessage(),e);
}
if (con) {
logger.info("=AlarmMessageSenderImple.sendEmail=>"+emailId+"发送成功！邮件内容："+checker.getEmailMessage());
} else {
logger.info("=AlarmMessageSenderImple.sendEmail=>"+emailId+"发送失败！邮件内容："+checker.getEmailMessage());
}
}
}
}
修改AlarmWriter使用自定义报警类
@Autowired(required=false)
private AlarmMessageSender alarmMessageSender = new AlarmMessageSenderImple();
#batch enable config
batch.enable=true
#batch server ip to execute batch
batch.server.ip=127.0.0.13.3配置mysql
修改jdbc.properties配置
#jdbc.properties配置信息
jdbc.driverClassName=com.mysql.jdbc.Driver
jdbc.url=jdbc:mysql://192.168.2.10:3306/ipinpoint?characterEncoding=UTF-8
jdbc.username=develop
jdbc.password=develop编辑applicationContext-web.xml文件
在applicationContext-web.xml文件中添加如下代码
<bean id="AlarmMessageSenderImple" class="com.navercorp.pinpoint.web.alarm.AlarmMessageSenderImple">
bean>修改web页面，打开用户注册
修改后源码：
Pinpoint User
+ Add
-
{{"("+pinpointUser.department + ")" + pinpointUser.name}}
4，初始数据库
在mysql中新建一个ipinpoint库在项目resources/sql中找到两个sql文件并在ipinpoint中执行
CreateTableStatement-mysql.sql
SpringBatchJobRepositorySchema-mysql.sql
执行后结果：
5，部署
在web目录下执行 mvn clean package命令，进行打包，拿到pinpoint-web-1.7.1.war之后，按照正常部署方式进行部署。参考博客：
APM监控--（二）Pinpoint部署手册
6，操作
新增用户
新增组
用户与组关联
配置报警
报警类型说明：
SLOW COUNT / 慢请求数
当应用发出的慢请求数量超过配置阈值时触发。
SLOW RATE / 慢请求比例
当应用发出的慢请求百分比超过配置阈值时触发。
ERROR COUNT / 请求失败数
当应用发出的失败请求数量超过配置阈值时触发。
ERROR RATE / 请求失败率
当应用发出的失败请求百分比超过配置阈值时触发。
TOTAL COUNT / 总数量
当应用发出的所有请求数量超过配置阈值时触发。
SLOW COUNT TO CALLEE / 被调用的慢请求数量
当发送给应用的慢请求数量超过配置阈值时触发。
SLOW RATE TO CALLEE / 被调用的慢请求比例
当发送给应用的慢请求百分比超过配置阈值时触发。
ERROR COUNT TO CALLEE / 被调用的请求错误数
当发送给应用的请求失败数量超过配置阈值时触发。
ERROR RATE TO CALLEE / 被调用的请求错误率
当发送给应用的请求失败百分比超过配置阈值时触发。
TOTAL COUNT TO CALLEE / 被调用的总数量
当发送给应用的所有请求数量超过配置阈值时触发。
HEAP USAGE RATE / 堆内存使用率
当应用的堆内存使用率超过配置阈值时触发。
JVM CPU USAGE RATE / JVM CPU使用率
当应用的CPU使用率超过配置阈值时触发。
7，问题
虽然pinpoint配置便捷，但是告警功能开发简单，不能丰富的配置，需要二次开发，最坑的是配置的规则80%都是默认的，通过看源码发现，基本都是写死的
以com.navercorp.pinpoint.common.trace.BaseHistogramSchema为例
报警规则源码：
public static final HistogramSchema FAST_SCHEMA = new BaseHistogramSchema(1, (short) 100, "100ms", (short) 300, "300ms", (short) 500, "500ms", "Slow", "Error", (short) -100, "100ms", (short) -300, "300ms", (short) -500, "500ms", (short) -999, "Slow");
public static final HistogramSchema NORMAL_SCHEMA = new BaseHistogramSchema(2, (short) 1000, "1s", (short) 3000, "3s", (short) 5000, "5s", "Slow", "Error", (short) -1000, "1s", (short) -3000, "3s", (short) -5000, "5s", (short) -9999, "Slow");