nGrinder对监控机器收集自定义数据及源码分析
0.背景
性能测试工具nGrinder支持在无需修改源码的情况下，对目标服务器收集自定义数据，最多支持5类；
在性能测试详细报告页，目标服务器->你的机器ip便签页下，默认只收集CPU, Memory, Received Byte/s, Sent Byte Per Secode/s等4类数据；
可能你还需要监控其它的性能统计数据，用于分析(比如load, Full Gc)；本文先介绍实现方法；再分析nGrinder源码，看它是怎么实现的。
1.实现
1-1. 安装monitor
在你的nGrinder系统下，下载监控
安装到你测试服务所在的机器，解压tar包，执行sh run_monitor_bg.sh;
其实脚本是启了个java服务，以monitor模式启动；
之前介绍过Agent有2种模式：
gent mode: 运行进程和线程，压测目标服务；
monitor mode: 监控目标系统性能(cpu/memory)。
[root@10 ngrinder-monitor]# cat run_monitor.sh
#!/bin/sh
curpath=`dirname $0`
cd ${curpath}
java -server -cp "lib/*" org.ngrinder.NGrinderAgentStarter --mode monitor --command run $@
Agent的home路径为/root/.ngrinder_agent，你在执行sh run_monitor_bg.sh默认获取的配置信息为/root/.ngrinder_agent/agent.conf； 如果加上-o，sh run_monitor_bg.sh 读取你安装monitor目录下的__agent.conf， 该配置文件定义了Agent的模式，ip, 端口。
[root@10 .ngrinder_agent]# cat agent.conf
common.start_mode=monitor
#If you want to monitor bind to the different local ip not automatically selected ip. Specify below field.
#monitor.binding_port=hostname_or_ip
monitor.binding_port=13243
自定义数据需放在/root/.ngrinder_agent/monitor/custom.data文件里，格式如下：
类型1数据,类型2数据,类型3数据,类型4数据,类型5数据最多支持5类，每类数据用“,”分隔，注意的是： 数据是实时的写文件，不是累积数据到文件中(类似shell中的>, 不是>>)，即同一时刻，只有一行数据。
1-2. 定制收集脚本
以收集load和full GC为例：
[root@10 bin]# cat updateCustomData.sh
#!/bin/sh
#@author hugang
customDataRoot=/root/.ngrinder_agent/monitor/custom.data;
# 获取load信息
load=`/bin/cat /proc/loadavg | awk '{print $1}'`;
# 获取full gc count
if [[ $1 -gt 0 ]]; then
fgc=`jstat -gcutil $1 | tail -1 | awk '{print $8}'`;
echo $load,$fgc > $customDataRoot;
else
echo $load > $customDataRoot;
fi;
开始性能测试时，每秒去执行该脚本，收集数据到custom.data中：
watch -n 1 sh updateCustomData.sh 55285528为需监控java服务进程pid;
当你性能测试结束后，monitor收集的数据会放到/root/.ngrinder/perftest/0_999/{test_id}/report/monitor_system_{ip}.data文件中:
[root@10 report]# cat monitor_system_10.13.1.139.data
ip,system,collectTime,freeMemory,totalMemory,cpuUsedPercentage,receivedPerSec,sentPerSec,customValues
10.13.1.139,LINUX,20160302151441,97102768,132112072,26.895683,32954,27897,4.93,49
10.13.1.139,LINUX,20160302151443,97075896,132112072,30.513468,45702,32306,4.93,49
10.13.1.139,LINUX,20160302151445,97034772,132112072,30.411074,110306,65391,5.02,49
10.13.1.139,LINUX,20160302151447,96972504,132112072,22.073017,84813,57503,5.02,49
...
1-3.结果展示
2.源码分析
nGrinder使用Sigar工具（https://support.hyperic.com/display/SIGAR/Home）收集系统信息，该工具可以收集以下数据：
System memory, swap, cpu, load average, uptime, logins
Per-process memory, cpu, credential info, state, arguments, environment, open files
File system detection and metrics
Network interface detection, configuration info and metrics
TCP and UDP connection tables
Network route table
sigar工具（http://download.csdn.net/download/neven7/9450930）示例：
[root@10 testsigar]# ls
libsigar-amd64-linux.so sigar-1.6.4.jar sigar-1.6.4.jar.zip
[root@10 testsigar]#
[root@10 testsigar]# java -jar ./sigar-1.6.4.jar
sigar> free
total used free
Mem: 132112072 96855372 35256700
-/+ buffers/cache: 34855500 97256572
Swap: 8388600 264980 8123620
RAM: 129016MB
sigar> 收集系统数据的java文件为：
ngrinder-core/src/main/java/org/ngrinder/monitor/collector/SystemDataCollector.java
继承和实现关系：
SystemDataCollector extends DataCollector
DataCollector implements Runnable
SystemDataCollector的线程执行体：
public void run() {
// 初始化sigar
initSigar();
SystemMonitoringData systemMonitoringData = (SystemMonitoringData) getMXBean(SYSTEM);
// execute()通过sigar api获取系统信息
systemMonitoringData.setSystemInfo(execute());
}
execute()获取系统信息SystemInfo(System info object to save date collected by monitor)：
/** * Execute the collector to get the system info model. * * @return SystemInfo in current time */
public synchronized SystemInfo execute() {
SystemInfo systemInfo = new SystemInfo();
systemInfo.setCollectTime(System.currentTimeMillis());
try {
BandWidth networkUsage = getNetworkUsage();
BandWidth bandWidth = networkUsage.adjust(prev.getBandWidth());
systemInfo.setBandWidth(bandWidth);
systemInfo.setCPUUsedPercentage((float) sigar.getCpuPerc().getCombined() * 100);
Cpu cpu = sigar.getCpu();
systemInfo.setTotalCpuValue(cpu.getTotal());
systemInfo.setIdleCpuValue(cpu.getIdle());
Mem mem = sigar.getMem();
systemInfo.setTotalMemory(mem.getTotal() / 1024L);
systemInfo.setFreeMemory(mem.getActualFree() / 1024L);
systemInfo.setSystem(OperatingSystem.IS_WIN32 ? SystemInfo.System.WINDOW : SystemInfo.System.LINUX);
systemInfo.setCustomValues(getCustomMonitorData());
} catch (Throwable e) {
LOGGER.error("Error while getting system perf data:{}", e.getMessage());
LOGGER.debug("Error trace is ", e);
}
prev = systemInfo;
return systemInfo;
}
其中：getCustomMonitorData()获取自定义数据，读取custom.data文件中一行数据
private String getCustomMonitorData() {
if (customDataFile != null && customDataFile.exists()) {
BufferedReader customDataFileReader = null;
try {
customDataFileReader = new BufferedReader(new FileReader(customDataFile));
return customDataFileReader.readLine(); // these data will be parsed at
// monitor client side.
} catch (IOException e) {
// Error here is very natural
LOGGER.debug("Error to read custom monitor data", e);
} finally {
IOUtils.closeQuietly(customDataFileReader);
}
}
return prev.getCustomValues();
}
综上：类SystemDataCollector作用就是作为线程执行体，线程每次执行通过sigar获取系统信息：SystemInfo，赋值给SystemMonitoringData成员变量SystemInfo。
前面介绍启动monitor时，其实是执行了org.ngrinder.NGrinderAgentStarter类，我们再分析下该文件，ngrinder-core/src/main/java/org/ngrinder/NGrinderAgentStarter.java
/** * Agent starter. * * @param args arguments */
public static void main(String[] args) {
NGrinderAgentStarter starter = new NGrinderAgentStarter();
final NGrinderAgentStarterParam param = new NGrinderAgentStarterParam();
checkJavaVersion();
JCommander commander = new JCommander(param);
commander.setProgramName("ngrinder-agent");
commander.setAcceptUnknownOptions(true);
try {
commander.parse(args);
} catch (Exception e) {
LOG.error(e.getMessage());
return;
}
final List<String> unknownOptions = commander.getUnknownOptions();
modeParam = param.getModeParam();
modeParam.parse(unknownOptions.toArray(new String[unknownOptions.size()]));
if (modeParam.version != null) {
System.out.println("nGrinder v" + getStaticVersion());
return;
}
if (modeParam.help != null) {
modeParam.usage();
return;
}
System.getProperties().putAll(modeParam.params);
starter.init();
final String startMode = modeParam.name();
if ("stop".equalsIgnoreCase(param.command)) {
starter.stopProcess(startMode);
System.out.println("Stop the " + startMode);
return;
}
starter.checkDuplicatedRun(startMode);
if (startMode.equalsIgnoreCase("agent")) {
starter.startAgent();
} else if (startMode.equalsIgnoreCase("monitor")) {
starter.startMonitor();
} else {
staticPrintHelpAndExit("Invalid agent.conf, '--mode' must be set as 'monitor' or 'agent'.");
}
}
monitor模式执行该方法：starter.startMonitor()
/** * Start the performance monitor. */
public void startMonitor() {
printLog("***************************************************");
printLog("* Start nGrinder Monitor... ");
printLog("***************************************************");
try {
MonitorServer.getInstance().init(agentConfig);
MonitorServer.getInstance().start();
} catch (Exception e) {
LOG.error("ERROR: {}", e.getMessage());
printHelpAndExit("Error while starting Monitor", e);
}
}
MonitorServer.getInstance().start():
/** * Start monitoring. * * @throws IOException exception */
public void start() throws IOException {
if (!isRunning()) {
jmxServer.start();
DataCollectManager.getInstance().init(agentConfig);
DataCollectManager.getInstance().start();
isRunning = true;
}
}DataCollectManager.getInstance().start();
/** * start a scheduler for the data collector jobs. */
public void start() {
int collectorCount = MXBeanStorage.getInstance().getSize();
scheduler = Executors.newScheduledThreadPool(collectorCount);
if (!isRunning()) {
Collection<MXBean> mxBeans = MXBeanStorage.getInstance().getMXBeans();
for (MXBean mxBean : mxBeans) {
DataCollector collector = mxBean.gainDataCollector(agentConfig.getHome().getDirectory());
scheduler.scheduleWithFixedDelay(collector, 0L, getInterval(), TimeUnit.SECONDS);
LOG.info("{} started.", collector.getClass().getSimpleName());
}
LOG.info("Collection interval : {}s).", getInterval());
isRunning = true;
}
}scheduler.scheduleWithFixedDelay(collector, 0L, getInterval(), TimeUnit.SECONDS);
线程池周期地执行SystemDataCollector中run()去获取系统数据。
@Override
public void run() {
initSigar();
SystemMonitoringData systemMonitoringData = (SystemMonitoringData) getMXBean(SYSTEM);
systemMonitoringData.setSystemInfo(execute());
}3.总结：
后台启动的monitor， 运行的是一个java服务:
java -server -cp lib/* org.ngrinder.NGrinderAgentStarter --mode monitor --command run通过线程池周期获取系统性信息(sigar工具获取)，存放在SystemInfo;
ngrinder-controller/src/main/java/org/ngrinder/perftest/service/samplinglistener/MonitorCollectorPlugin.java中startSampling():
@Override
public void startSampling(final ISingleConsole singleConsole, PerfTest perfTest,
IPerfTestService perfTestService) {
final List<String> targetHostIP = perfTest.getTargetHostIP();
final Integer samplingInterval = perfTest.getSamplingInterval();
for (final String target : targetHostIP) {
scheduledTaskService.runAsync(new Runnable() {
@Override
public void run() {
LOGGER.info("Start JVM monitoring for IP:{}", target);
MonitorClientService client = new MonitorClientService(target, MonitorCollectorPlugin.this.port);
client.init();
if (client.isConnected()) {
File testReportDir = singleConsole.getReportPath();
File dataFile = null;
try {
dataFile = new File(testReportDir, MONITOR_FILE_PREFIX + target + ".data");
FileWriter fileWriter = new FileWriter(dataFile, false);
BufferedWriter bw = new BufferedWriter(fileWriter);
// write header info
bw.write(SystemInfo.HEADER);
bw.newLine();
bw.flush();
clientMap.put(client, bw);
} catch (IOException e) {
LOGGER.error("Error to write to file:{}, Error:{}", dataFile.getPath(), e.getMessage());
}
}
}
});
}
assignScheduledTask(samplingInterval);
}
根据SystemInfo写到/root/.ngrinder/perftest/0_999/{test_id}/report/monitor_system_{ip}.data文件中；
ngrinder-controller/src/main/java/org/ngrinder/perftest/PerfTestService.java中getMonitorGraph()根据/root/.ngrinder/perftest/0_999/{test_id}/report/monitor_system_{ip}.data获取系统信息数据
/** * Get system monitor data and wrap the data as a string value like "[22,11,12,34,....]", which can be used directly * in JS as a vector. * * @param testId test id * @param targetIP ip address of the monitor target * @param dataInterval interval value to get data. Interval value "2" means, get one record for every "2" records. * @return return the data in map */
public Map<String, String> getMonitorGraph(long testId, String targetIP, int dataInterval) {
Map<String, String> returnMap = Maps.newHashMap();
File monitorDataFile = new File(config.getHome().getPerfTestReportDirectory(String.valueOf(testId)),
MONITOR_FILE_PREFIX + targetIP + ".data");
BufferedReader br = null;
try {
StringBuilder sbUsedMem = new StringBuilder("[");
StringBuilder sbCPUUsed = new StringBuilder("[");
StringBuilder sbNetReceived = new StringBuilder("[");
StringBuilder sbNetSent = new StringBuilder("[");
StringBuilder customData1 = new StringBuilder("[");
StringBuilder customData2 = new StringBuilder("[");
StringBuilder customData3 = new StringBuilder("[");
StringBuilder customData4 = new StringBuilder("[");
StringBuilder customData5 = new StringBuilder("[");
br = new BufferedReader(new FileReader(monitorDataFile));
br.readLine(); // skip the header.
// "ip,system,collectTime,freeMemory,totalMemory,cpuUsedPercentage,receivedPerSec,sentPerSec"
String line = br.readLine();
int skipCount = dataInterval;
// to be compatible with previous version, check the length before
// adding
while (StringUtils.isNotBlank(line)) {
if (skipCount < dataInterval) {
skipCount++;
} else {
skipCount = 1;
String[] datalist = StringUtils.split(line, ",");
if ("null".equals(datalist[4]) || "undefined".equals(datalist[4])) {
sbUsedMem.append("null").append(",");
} else {
sbUsedMem.append(Long.valueOf(datalist[4]) - Long.valueOf(datalist[3])).append(",");
}
addCustomData(sbCPUUsed, 5, datalist);
addCustomData(sbNetReceived, 6, datalist);
addCustomData(sbNetSent, 7, datalist);
addCustomData(customData1, 8, datalist);
addCustomData(customData2, 9, datalist);
addCustomData(customData3, 10, datalist);
addCustomData(customData4, 11, datalist);
addCustomData(customData5, 12, datalist);
line = br.readLine();
}
}
completeCustomData(returnMap, "cpu", sbCPUUsed);
completeCustomData(returnMap, "memory", sbUsedMem);
completeCustomData(returnMap, "received", sbNetReceived);
completeCustomData(returnMap, "sent", sbNetSent);
completeCustomData(returnMap, "customData1", customData1);
completeCustomData(returnMap, "customData2", customData2);
completeCustomData(returnMap, "customData3", customData3);
completeCustomData(returnMap, "customData4", customData4);
completeCustomData(returnMap, "customData5", customData5);
} catch (IOException e) {
LOGGER.info("Error while getting monitor {} data file at {}", targetIP, monitorDataFile);
} finally {
IOUtils.closeQuietly(br);
}
return returnMap;
}
数据提供给Controller端：
ngrinder-controller/src/man/java/org/ngrinder/perftest/controller/PerfTestController.java
private Map<String, String> getMonitorGraphData(long id, String targetIP, int imgWidth) {
int interval = perfTestService.getMonitorGraphInterval(id, targetIP, imgWidth);
Map<String, String> sysMonitorMap = perfTestService.getMonitorGraph(id, targetIP, interval);
PerfTest perfTest = perfTestService.getOne(id);
sysMonitorMap.put("interval", String.valueOf(interval * (perfTest != null ? perfTest.getSamplingInterval() : 1)));
return sysMonitorMap;
}
/** * Get the monitor data of the target having the given IP. * * @param id test Id * @param targetIP targetIP * @param imgWidth image width * @return json message */
@RestAPI
@RequestMapping("/api/{id}/monitor")
public HttpEntity<String> getMonitorGraph(@PathVariable("id") long id,
@RequestParam("targetIP") String targetIP, @RequestParam int imgWidth) {
return toJsonHttpEntity(getMonitorGraphData(id, targetIP, imgWidth));
}