AIX6.1上WAS服务器CPU占用高的处理步骤

阅读更多

1、问题现象

       服务器环境是AIX6.1\WAS7.0，通过topas监控CPU经常出现100%的情况，而且负荷比较轻时，CPU占用率不能也下降。

 

2、处理步骤

       1、通过topas监控可以看到当前占用CPU高的两个java进程，记录下进程号：6619258；

       2、通过ps -mp 6619258 -o THREAD 以查找当前进程正在占用 CPU 的线程信息，把输出信息拷贝到文本文件中；

       3、通过kill -3 6619258 输出ThreadDump线程执行堆栈快照信息，在当前目录中找到类似javacore.20120322.130602.6619258.0003.txt文件，拷贝到本地；

       4、下面分析这两个文件，分析tid线程信息发现有下面标红的几个线程占用CPU比较高：

    USER     PID    PPID       TID ST  CP PRI SC    WCHAN        F     TT BND COMMAND

    root 6619258       1         - A  314  60 108        *   242001      -  -1 /was/IBM/WebSphere/AppServer/java/bin/java -

       -       -       -  10616851 S    0  82  1 f1000f0a1000a240  8410400      -  -1 -

       -       -       -  11141271 S    0  82  1 f1000f0a1000aa40  8410404      -  -1 -

       -       -       -  20971659 S    0  82  1 f1000f0a10014040  8410404      -  -1 -

       -       -       -  21168201 R   55 118  0        -   400000      -  -1 -

       -       -       -  21299399 S    0  82  1 f1000f0a10014540  8410400      -  -1 -

       -       -       -  22544397 S    0  82  1 f1000f0a10015840  8410400      -  -1 -

       -       -       -  29294701 S    0  82  1 f1000f0a1001bf40  8410400      -  -1 -

       -       -       -  29819073 S    0  82  1 f1000f0a1001c740  8410400      -  -1 -

       -       -       -  29884629 R   72 130  0        -   400000      -  -1 -

       -       -       -  30212313 S    0  82  1 f1000f0a1001cd40  8410400      -  -1 -

       -       -       -  30277691 S    0  82  1 f1000f0a1001ce40  8410400      -  -1 -

       -       -       -  34013245 S    0  82  1 f1000f0a10020740  8410404      -  -1 -

       -       -       -  34078781 S    0  82  1 f1000f0a10020840  8410400      -  -1 -

       -       -       -  34209851 R   69 128  0        -   400000      -  -1 -

       -       -       -  34406643 S    1  82  1 f1000f0a10020d40  8410400      -  -1 -

       -       -       -  34537533 S    0  82  1        -   418400      -  -1 -

 

       5、取这些线程号，转换为16进制的数据，21168201 转换16进制值为0x1430049，可以使用windows的“计算器”工具，先输入10进制数值，然后切换到16进制。

       取0x1430049在javacore.20120322.130602.6619258.0003.txt中找到下面一段日志：

3XMTHREADINFO      "WebContainer : 3" TID:0x00000000E188B100, j9thread_t:0x000000011A5F0D20, state:CW, prio=5

3XMTHREADINFO1            (native thread ID:0x1430049, native priority:0x5, native policy:UNKNOWN)

4XESTACKTRACE          at org/apache/regexp/RECompiler.setNextOfEnd(RECompiler.java:209(Compiled Code))

4XESTACKTRACE          at org/apache/regexp/RECompiler.expr(RECompiler.java:1193(Compiled Code))

4XESTACKTRACE          at org/apache/regexp/RECompiler.terminal(RECompiler.java:866(Compiled Code))

4XESTACKTRACE          at org/apache/regexp/RECompiler.closure(RECompiler.java:942(Compiled Code))

4XESTACKTRACE          at org/apache/regexp/RECompiler.branch(RECompiler.java:1151(Compiled Code))

4XESTACKTRACE          at org/apache/regexp/RECompiler.expr(RECompiler.java:1203(Compiled Code))

4XESTACKTRACE          at org/apache/regexp/RECompiler.compile(RECompiler.java:1281(Compiled Code))

4XESTACKTRACE          at org/apache/regexp/RE.(RE.java:495(Compiled Code))

4XESTACKTRACE          at org/apache/regexp/RE.(RE.java:480(Compiled Code))

4XESTACKTRACE          at com/sbdcpn/utils/Stringvalidator.getRegularObj(Stringvalidator.java:173(Compiled Code))

4XESTACKTRACE          at com/sbdcpn/utils/Stringvalidator.validValue(Stringvalidator.java:150(Compiled Code))

4XESTACKTRACE          at com/sbdcpn/utils/tools/StringFormat.StringFormatToDate(StringFormat.java:40(Compiled Code))

4XESTACKTRACE          at com/sbdcpn/page/utils/PageUtilMethod.getDataValue(Unknown Source:279(Compiled Code))

4XESTACKTRACE          at com/sbdcpn/page/taglib/IteratorResultTag.convertFormat(IteratorResultTag.java:697(Compiled Code))

4XESTACKTRACE          at com/sbdcpn/page/taglib/IteratorResultTag.formatResult(IteratorResultTag.java:671(Compiled Code))

4XESTACKTRACE          at com/sbdcpn/page/taglib/IteratorResultTag.doAfterBody(IteratorResultTag.java:372(Compiled Code))

4XESTACKTRACE          at com/ibm/_jsp/_grid_5F_plan_5F_query._jspService(_grid_5F_plan_5F_query.java:1889(Compiled Code))

      

       29884629 转换为16进制值为0x1C800D5，查找日志为：

3XMTHREADINFO      "WebContainer : 8" TID:0x00000000E22E7600, j9thread_t:0x000000011FBE1500, state:CW, prio=5

3XMTHREADINFO1            (native thread ID:0x1C800D5, native priority:0x5, native policy:UNKNOWN)

4XESTACKTRACE          at org/apache/regexp/RECompiler.setNextOfEnd(RECompiler.java:207(Compiled Code))

4XESTACKTRACE          at org/apache/regexp/RECompiler.expr(RECompiler.java:1194(Compiled Code))

4XESTACKTRACE          at org/apache/regexp/RECompiler.terminal(RECompiler.java:866(Compiled Code))

4XESTACKTRACE          at org/apache/regexp/RECompiler.closure(RECompiler.java:942(Compiled Code))

4XESTACKTRACE          at org/apache/regexp/RECompiler.branch(RECompiler.java:1151(Compiled Code))

4XESTACKTRACE          at org/apache/regexp/RECompiler.expr(RECompiler.java:1203(Compiled Code))

4XESTACKTRACE          at org/apache/regexp/RECompiler.compile(RECompiler.java:1281(Compiled Code))

4XESTACKTRACE          at org/apache/regexp/RE.(RE.java:495(Compiled Code))

4XESTACKTRACE          at org/apache/regexp/RE.(RE.java:480(Compiled Code))

4XESTACKTRACE          at com/sbdcpn/utils/Stringvalidator.getRegularObj(Stringvalidator.java:173(Compiled Code))

4XESTACKTRACE          at com/sbdcpn/utils/Stringvalidator.validValue(Stringvalidator.java:150(Compiled Code))

4XESTACKTRACE          at com/sbdcpn/utils/tools/StringFormat.StringFormatToDate(StringFormat.java:40(Compiled Code))

4XESTACKTRACE          at com/sbdcpn/page/utils/PageUtilMethod.getDataValue(Unknown Source:279(Compiled Code))

4XESTACKTRACE          at com/sbdcpn/page/taglib/IteratorResultTag.convertFormat(IteratorResultTag.java:697(Compiled Code))

4XESTACKTRACE          at com/sbdcpn/page/taglib/IteratorResultTag.formatResult(IteratorResultTag.java:671(Compiled Code))

4XESTACKTRACE          at com/sbdcpn/page/taglib/IteratorResultTag.doStartTag(IteratorResultTag.java:289)

4XESTACKTRACE          at com/ibm/_jsp/_grid_5F_plan_5F_query._jspService(_grid_5F_plan_5F_query.java:1499)

      

       跟踪其它几个占用CPU高的几个线程，发现都是这段代码有问题。

 

       6、优化程序后，不使用org/apache/regexp/RECompiler包，而采用java自带的

Pattern p = Pattern.compile(regex);

              ret = p.matcher(value).lookingAt();

              问题排除。