【异常】记一次，排查docker容器的微服务，CPU飙高问题
项目时 以docker为容器的 微服务项目。 其中一个docker容器，CPU很高。所以需要排查一下问题。
[root@service-slave2 ~]# docker stats
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS NAMES
fa4b3663ece4 keda6-slave2-information-plan 733.45% 919MiB / 1.5GiB 59.83% 909MB / 748MB 0B / 0B 217CPU达到 733%
1、进入容器，查看 进程号
[root@service-slave2 ~]# docker exec -it fa4b3663ece4 bash
[root@slave2_information_plan project]# top
top - 11:13:50 up 11 days, 23:26, 0 users, load average: 44.96, 41.39, 39.79
Tasks: 13 total, 1 running, 12 sleeping, 0 stopped, 0 zombie
%Cpu(s): 96.8 us, 0.8 sy, 0.0 ni, 2.4 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 64757920 total, 571656 free, 28601872 used, 35584392 buff/cache
KiB Swap: 33554428 total, 29073788 free, 4480640 used. 29747760 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
8156 root 20 0 6311560 879272 13592 S 706.2 1.4 6122:00 java
1 root 20 0 43272 3072 2392 S 0.0 0.0 0:00.40 systemd
17 root 20 0 47276 9424 9308 S 0.0 0.0 0:00.81 systemd-journal
28 root 20 0 41628 1392 1276 S 0.0 0.0 0:00.15 systemd-udevd
66 root 20 0 26384 1600 1460 S 0.0 0.0 0:00.06 systemd-logind
67 dbus 20 0 58112 2008 1824 S 0.0 0.0 0:00.01 dbus-daemon
81 root 20 0 112924 3524 3332 S 0.0 0.0 0:00.13 sshd
83 root 20 0 9924 724 720 S 0.0 0.0 0:00.00 agetty
8158 root 20 0 13000 828 632 S 0.0 0.0 0:00.48 sh
8172 root 20 0 3131584 72224 12544 S 0.0 0.1 1:05.19 jstatd
15282 root 20 0 15244 2072 1656 S 0.0 0.0 0:00.01 bash
15324 root 20 0 7772 356 280 S 0.0 0.0 0:00.00 sleep
15327 root 20 0 59616 2060 1488 R 0.0 0.0 0:00.00 top 2、使用top -H -p 进程号查看异常线程、查看线程
[root@slave2_information_plan project]# top -H -p 8156
top - 11:14:20 up 11 days, 23:26, 0 users, load average: 39.80, 40.49, 39.54
Threads: 193 total, 30 running, 163 sleeping, 0 stopped, 0 zombie
%Cpu(s): 96.0 us, 0.5 sy, 0.0 ni, 3.4 id, 0.0 wa, 0.0 hi, 0.1 si, 0.0 st
KiB Mem : 64757920 total, 540364 free, 28621188 used, 35596368 buff/cache
KiB Swap: 33554428 total, 29073788 free, 4480640 used. 29728468 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14805 root 20 0 6311560 879272 13592 R 43.9 1.4 34:15.23 java
14847 root 20 0 6311560 879272 13592 R 37.5 1.4 29:11.78 java
14848 root 20 0 6311560 879272 13592 R 37.2 1.4 28:24.52 java
14857 root 20 0 6311560 879272 13592 R 35.9 1.4 27:59.42 java
14865 root 20 0 6311560 879272 13592 R 35.5 1.4 27:07.37 java
14692 root 20 0 6311560 879272 13592 R 33.2 1.4 30:21.00 java
14866 root 20 0 6311560 879272 13592 R 29.6 1.4 27:00.60 java
14834 root 20 0 6311560 879272 13592 R 28.6 1.4 28:56.27 java
14833 root 20 0 6311560 879272 13592 R 27.2 1.4 29:11.41 java
14883 root 20 0 6311560 879272 13592 R 26.6 1.4 26:18.58 java
14849 root 20 0 6311560 879272 13592 R 26.2 1.4 28:11.36 java
10125 root 20 0 6311560 879272 13592 R 25.6 1.4 56:53.44 java
14671 root 20 0 6311560 879272 13592 R 24.9 1.4 53:55.84 java
14818 root 20 0 6311560 879272 13592 R 23.9 1.4 31:27.83 java
14873 root 20 0 6311560 879272 13592 R 23.6 1.4 26:25.93 java
8592 root 20 0 6311560 879272 13592 R 23.3 1.4 59:16.66 java
8304 root 20 0 6311560 879272 13592 R 22.3 1.4 63:12.35 java
14685 root 20 0 6311560 879272 13592 R 22.3 1.4 30:09.24 java
8307 root 20 0 6311560 879272 13592 R 21.9 1.4 65:13.92 java
14819 root 20 0 6311560 879272 13592 R 21.9 1.4 30:13.75 java
8310 root 20 0 6311560 879272 13592 R 21.3 1.4 868:16.29 java
8311 root 20 0 6311560 879272 13592 R 19.3 1.4 868:50.86 java
8309 root 20 0 6311560 879272 13592 R 18.9 1.4 872:06.83 java
14864 root 20 0 6311560 879272 13592 R 18.9 1.4 27:53.39 java
8305 root 20 0 6311560 879272 13592 R 18.6 1.4 864:41.14 java
14658 root 20 0 6311560 879272 13592 R 18.3 1.4 59:20.44 java
14693 root 20 0 6311560 879272 13592 R 18.3 1.4 32:41.43 java
8306 root 20 0 6311560 879272 13592 R 14.3 1.4 868:35.26 java
8308 root 20 0 6311560 879272 13592 R 13.0 1.4 861:23.26 java
8175 root 20 0 6311560 879272 13592 S 3.0 1.4 46:57.17 java
15305 root 20 0 6311560 879272 13592 S 0.7 1.4 0:00.55 java
15318 root 20 0 6311560 879272 13592 S 0.3 1.4 0:00.53 java 3、使用printf "%x\n" 线程号将异常线程号转化为16进制
[root@slave2_information_plan project]# printf "%x\n" 14805
39d54、使用jstack 进程号|grep 16进制异常线程号 -A90来定位异常代码的位置（最后的-A90是日志行数，也可以输出为文本文件或使用其他数字）。可以看到异常代码的位置。
[root@slave2_information_plan project]# jstack 8156|grep 39d5 -A90
"http-nio-9820-exec-18" #261 daemon prio=5 os_prio=0 tid=0x00007f3e0800f800 nid=0x39d5 runnable [0x00007f3d70580000]
java.lang.Thread.State: RUNNABLE
at sun.util.calendar.ZoneInfo.getTransitionIndex(ZoneInfo.java:339)
at sun.util.calendar.ZoneInfo.getOffsets(ZoneInfo.java:259)
at sun.util.calendar.ZoneInfo.getOffsets(ZoneInfo.java:236)
at java.util.GregorianCalendar.computeFields(GregorianCalendar.java:2340)
at java.util.GregorianCalendar.computeFields(GregorianCalendar.java:2312)
at java.util.Calendar.setTimeInMillis(Calendar.java:1804)
at java.util.GregorianCalendar.add(GregorianCalendar.java:1076)
at com.keda.information.plan.service.impl.CardPlanServiceJHImpl.longOfTwoDate(CardPlanServiceJHImpl.java:1969)
at com.keda.information.plan.service.impl.CardPlanServiceJHImpl.restartCardPlanZZZ(CardPlanServiceJHImpl.java:2272)
at com.keda.information.plan.service.impl.CardPlanServiceJHImpl$$FastClassBySpringCGLIB$$26beb7de.invoke()
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:749)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:88)
at com.keda.util.log.LogRecordAspect.doAroundForService(LogRecordAspect.java:233)
at sun.reflect.GeneratedMethodAccessor374.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:644)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:633)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:93)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)
at com.keda.information.plan.service.impl.CardPlanServiceJHImpl$$EnhancerBySpringCGLIB$$115419c6.restartCardPlanZZZ()
at com.keda.information.plan.service.api.ICardPlanServiceApi.restartCardPlanZ(ICardPlanServiceApi.java:259)
at com.keda.information.plan.service.api.ICardPlanServiceApi$$FastClassBySpringCGLIB$$6b2eaa41.invoke()
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:749)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:88)
at com.keda.util.log.LogRecordAspect.doAroundForApi(LogRecordAspect.java:177)
at sun.reflect.GeneratedMethodAccessor472.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:644)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:633)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:93)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)
at com.keda.information.plan.service.api.ICardPlanServiceApi$$EnhancerBySpringCGLIB$$c22f468b.restartCardPlanZ()
at sun.reflect.GeneratedMethodAccessor867.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:189)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:800)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1038)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:942)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1005)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:897)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:634)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:882)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at brave.servlet.TracingFilter.doFilter(TracingFilter.java:65)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.alibaba.druid.support.http.WebStatFilter.doFilter(WebStatFilter.java:124)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.boot.actuate.web.trace.servlet.HttpTraceFilter.doFilterInternal(HttpTraceFilter.java:88)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:92)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.cloud.sleuth.instrument.web.ExceptionLoggingFilter.doFilter(ExceptionLoggingFilter.java:50)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) 注意事项：
如果提示 -bash: jstack: command not found，则可以找到jdk的目录，进入bin目录，
在执行 ./jstack 进程号 | grep 16进制异常线程号 -A90 即可。
如果提示 well-known file is not secure，则是因为账号权限的问题。在/tmp/hsperfdata_$USER/目录，有一个以进程号命名的文件，当我们执行jmap或者jstack出现上叙信息时，先检查执行该命令的用户是否和hsperfdata_$USER这个文件所属的用户一致，如果不一致，切换至成一致再执行。执行：sudo -u 有权限的账号 ./jstack 进程号 | grep 16进制异常线程号 -A50 即可。
5、查看代码
public static int longOfTwoDate(Date first, Date second) throws ParseException {
Calendar calendar = Calendar.getInstance();
calendar.setTime(first);
int cnt = 0;
while (calendar.getTime().compareTo(second) != 0) {
calendar.add(Calendar.DATE, 1);
cnt++;
}
return cnt;
}发现一个 while的死循环。修改如下：
public static int longOfTwoDate(Date first, Date second) throws ParseException {
Calendar calendar = Calendar.getInstance();
calendar.setTime(first);
int cnt = 0;
if(calendar.getTime().compareTo(second) <= 0){
while (calendar.getTime().compareTo(second) != 0) {
calendar.add(Calendar.DATE, 1);
cnt++;
}
}
return cnt;
}