zookeeper

一、zookeeper监控方式

根据官网介绍主要有两种监控方式，
1：使用ssh建立远程连接，然后使用"echo stat | nc ip port"命令返回结果。开源项目代表taokeeper。
2：使用JMX监控，文档[http://zookeeper.apache.org/doc/r3.4.6/zookeeperJMX.html]
我们考虑使用JMX监控是因为运维不希望在服务器之间配置ssh连接，或者openSSH，服务器太多难以管理用户密码和公钥，容易混乱。

二、zookeeper服务开启JMX监控

修改zookeeper的启动脚本vim zkServer.sh。找到启动参数ZOOMAIN，修改为下面值。
其中local.only=false，设为false才能在远程建立连接。

ZOOMAIN="-Dcom.sun.management.jmxremote  -Dcom.sun.management.jmxremote.local.only=false
 -Djava.rmi.server.hostname=127.0.0.1
 -Dcom.sun.management.jmxremote.port=9991
 -Dcom.sun.management.jmxremote.ssl=true
 -Dcom.sun.management.jmxremote.authenticate=true
 -Dcom.sun.management.jmxremote.access.file=/data/zookeeper/conf/jmxremote.access
 -Dcom.sun.management.jmxremote.password.file=/data/zookeeper/conf/jmxremote.password
 -Dzookeeper.jmx.log4j.disable=true
 org.apache.zookeeper.server.quorum.QuorumPeerMain"

在/data/zookeeper/conf目录下建立2个访问授权文件, 修改文件权限chmod 600 jmxremote.*

-rw-------  1 deploy deploy  149 Aug  6 13:44 jmxremote.access
-rw-------  1 deploy deploy   40 Aug  6 13:46 jmxremote.password
[deploy@liutp conf]$ pwd
/data/zookeeper/conf
[deploy@liutp conf]$ cat jmxremote.access
monitorRole   readonly
controlRole   readwrite \
              create javax.management.monitor.*,javax.management.timer.* \
              unregister
[deploy@liutp conf]$ cat jmxremote.password
monitorRole  1234567
controlRole  1234567
[deploy@liutp conf]$

关于权限文件内容的设置可以参考jdk里的文件：C:\Java\jdk1.7.0_79\jre\lib\management\jmxremote.password.template
重启zk服务 ：zkServer.sh restart /data/zookeeper/conf/zk1.cfg

三、使用Java自带的JConsole

在命令行输入JConsole，再回车。
在弹出的界面选择“远程进程”，输入“服务器IP:9991”(zookeeper服务器的IP和端口)
用户名：controlRole密码：1234567就能查看节点数，连接数，watch数

一、背景

上一篇通过Java自带的JConsole来获取zookeeper状态。主要有几个不方便的地方，zk集群一般会部署3或者5台，在多个JConsole窗口中切换比较麻烦，各个zk服务及历史数据之间，不能直观比较。一般会做一个WEB管理页面来展示集群状态，设置报警阀值来做报警。

二、JVM平台提供Mbeans

在Java5.0以上版本，有一组API可以让Java应用程序和允许的工具监视和管理Java虚拟机（JVM）和虚拟机所在的本机操作系统。该组API在 java.lang.management包。可以通过这些API可以监控local端JVM，同时也可以监控远端JVM。
通过静态工厂方法获取MXBean实例，从本地访问正在运行的虚拟机的MXBean接口。这些Bean我们从ManagementFactory类中定义的静态方法获取；如ManagementFactory.getOperatingSystemMXBean();其中不足就是只能获取本地的JVM状态。无法获取远程的虚拟机数据。

• ClassLoadingMXBean Java虚拟机的类加载系统
• CompilationMXBean Java虚拟机的编译系统
• MemoryMXBean Java虚拟机的内存系统
• RuntimeMXBean Java虚拟机的运行时系统
• OperatingSystemMXBean Java虚拟机在其上运行的操作系统
• GarbageCollectorMXBean Java虚拟机中的垃圾回收器
• MemoryManagerMXBean Java虚拟机中的内存管理器
• MemoryPoolMXBean Java虚拟机中的内存池

三、Zookeeper提供出来的Mbeans

构造MXBean代理实例，通过代理将方法调用转发到给定的MBeanServe。JConsole能够监控的项目，通过API都能获取到。
具体代码如下：

 
       
       import java.io.IOException;   
       import java.lang.management.ClassLoadingMXBean;   
       import java.lang.management.CompilationMXBean;   
       import java.lang.management.ManagementFactory;   
       import java.lang.management.OperatingSystemMXBean;   
       import java.lang.management.ThreadMXBean;   
       import java.util.ArrayList;   
       import java.util.Arrays;   
       import java.util.Collections;   
       import java.util.HashMap;   
       import java.util.Iterator;   
       import java.util.List;   
       import java.util.Map;   
       import java.util.Set;   
          
       import javax.management.InstanceNotFoundException;   
       import javax.management.IntrospectionException;   
       import javax.management.JMX;   
       import javax.management.MBeanServerConnection;   
       import javax.management.MalformedObjectNameException;   
       import javax.management.ObjectName;   
       import javax.management.ReflectionException;   
       import javax.management.remote.JMXConnector;   
       import javax.management.remote.JMXConnectorFactory;   
       import javax.management.remote.JMXServiceURL;   
          
       import org.apache.zookeeper.server.ConnectionMXBean;   
       import org.apache.zookeeper.server.DataTreeMXBean;   
       import org.apache.zookeeper.server.ZooKeeperServerMXBean;   
          
       public class ZkJMXTest {   
           static JMXConnector connector;   
          
           /**  
            * @param args  
            * @throws IOException  
            * @throws MalformedObjectNameException  
            * @throws InstanceNotFoundException  
            * @throws ReflectionException  
            * @throws IntrospectionException  
            */   
           public static void main(String[] args) throws IOException, MalformedObjectNameException,   
               InstanceNotFoundException, IntrospectionException, ReflectionException {   
          
               OperatingSystemMXBean osbean = ManagementFactory.getOperatingSystemMXBean();   
               System.out.println("体系结构:" + osbean.getArch());//操作系统体系结构   
               System.out.println("处理器核数:" + osbean.getAvailableProcessors());///核数   
               System.out.println("名字:" + osbean.getName());//名字   
          
               System.out.println(osbean.getVersion());//操作系统版本   
               ThreadMXBean threadBean=ManagementFactory.getThreadMXBean();   
               System.out.println("活动线程:" + threadBean.getThreadCount());//总线程数   
          
               ClassLoadingMXBean classLoadingMXBean = ManagementFactory.getClassLoadingMXBean();   
               CompilationMXBean compilationMXBean = ManagementFactory.getCompilationMXBean();   
               System.out.println("===========");   
          
               // 通过 MBeanServer间接地访问 MXBean 接口   
               MBeanServerConnection mbsc = createMBeanServer("192.168.1.100", "9991", "controlRole", "123456");   
          
               // 操作系统   
               ObjectName os = new ObjectName("java.lang:type=OperatingSystem");   
               System.out.println("体系结构:" + getAttribute(mbsc, os, "Arch"));//体系结构   
               System.out.println("处理器核数:" + getAttribute(mbsc, os, "AvailableProcessors"));//核数   
               System.out.println("总物理内存:" + getAttribute(mbsc, os, "TotalPhysicalMemorySize"));//总物理内存   
               System.out.println("空闲物理内存:" + getAttribute(mbsc, os, "FreePhysicalMemorySize"));//空闲物理内存   
               System.out.println("总交换空间:" + getAttribute(mbsc, os, "TotalSwapSpaceSize"));//总交换空间   
               System.out.println("空闲交换空间:" + getAttribute(mbsc, os, "FreeSwapSpaceSize"));//空闲交换空间   
          
               System.out.println("操作系统:" + getAttribute(mbsc, os, "Name")+ getAttribute(mbsc, os, "Version"));//操作系统   
               System.out.println("提交的虚拟内存:" + getAttribute(mbsc, os, "CommittedVirtualMemorySize"));//提交的虚拟内存   
               System.out.println("系统cpu使用率:" + getAttribute(mbsc, os, "SystemCpuLoad"));//系统cpu使用率   
               System.out.println("进程cpu使用率:" + getAttribute(mbsc, os, "ProcessCpuLoad"));//进程cpu使用率   
          
               System.out.println("============");//   
               // 线程   
               ObjectName Threading = new ObjectName("java.lang:type=Threading");   
               System.out.println("活动线程:" + getAttribute(mbsc, Threading, "ThreadCount"));// 活动线程   
               System.out.println("守护程序线程:" + getAttribute(mbsc, Threading, "DaemonThreadCount"));// 守护程序线程   
               System.out.println("峰值:" + getAttribute(mbsc, Threading, "PeakThreadCount"));// 峰值   
               System.out.println("启动的线程总数:" + getAttribute(mbsc, Threading, "TotalStartedThreadCount"));// 启动的线程总数   
               ThreadMXBean threadBean2 = ManagementFactory.newPlatformMXBeanProxy   
                       (mbsc, ManagementFactory.THREAD_MXBEAN_NAME, ThreadMXBean.class);   
               System.out.println("活动线程:" + threadBean2.getThreadCount());// 活动线程   
               ThreadMXBean threadBean3 = ManagementFactory.getThreadMXBean();   
               System.out.println("本地活动线程:" + threadBean3.getThreadCount());// 本地活动线程   
          
               System.out.println("============");//   
               ObjectName Compilation = new ObjectName("java.lang:type=Compilation");   
               System.out.println("总编译时间 毫秒:" + getAttribute(mbsc, Compilation, "TotalCompilationTime"));// 总编译时间 毫秒   
          
               System.out.println("============");//   
               ObjectName ClassLoading = new ObjectName("java.lang:type=ClassLoading");   
               System.out.println("已加载类总数:" + getAttribute(mbsc, ClassLoading, "TotalLoadedClassCount"));// 已加载类总数   
               System.out.println("已加装当前类:" + getAttribute(mbsc, ClassLoading, "LoadedClassCount"));// 已加装当前类   
               System.out.println("已卸载类总数:" + getAttribute(mbsc, ClassLoading, "UnloadedClassCount"));// 已卸载类总数   
          
          
               System.out.println("==========================================================");//   
               // http://zookeeper.apache.org/doc/r3.4.6/zookeeperJMX.html   
               // org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.1,name2=Follower   
               ObjectName replica = new ObjectName("org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.1");   
               System.out.println("replica.1运行状态:" + getAttribute(mbsc, replica, "State"));// 运行状态   
          
               mbsc = createMBeanServer("192.168.1.100", "9992", "controlRole", "123456");   
               System.out.println("==============节点树对象===========");   
               ObjectName dataTreePattern = new ObjectName("org.apache.ZooKeeperService:name0=ReplicatedServer_id?,name1=replica.?,name2=*,name3=InMemoryDataTree");   
               Set dataTreeSets = mbsc.queryNames(dataTreePattern, null);   
               Iterator dataTreeIterator = dataTreeSets.iterator();   
               // 只有一个   
               while (dataTreeIterator.hasNext()) {   
                   ObjectName dataTreeObjectName = dataTreeIterator.next();   
                   DataTreeMXBean dataTree = JMX.newMBeanProxy(mbsc, dataTreeObjectName, DataTreeMXBean.class);   
                   System.out.println("节点总数:" + dataTree.getNodeCount());// 节点总数   
                   System.out.println("Watch总数:" + dataTree.getWatchCount());// Watch总数   
                   System.out.println("临时节点总数:" + dataTree.countEphemerals());// Watch总数   
                   System.out.println("节点名及字符总数:" + dataTree.approximateDataSize());// 节点全路径和值的总字符数   
          
                   Map dataTreeMap = dataTreeObjectName.getKeyPropertyList();   
                   String replicaId = dataTreeMap.get("name1").replace("replica.", "");   
                   String role = dataTreeMap.get("name2");// Follower,Leader,Observer,Standalone   
                   String canonicalName = dataTreeObjectName.getCanonicalName();   
                   int roleEndIndex = canonicalName.indexOf(",name3");   
          
                   ObjectName roleObjectName = new ObjectName(canonicalName.substring(0, roleEndIndex));   
                   System.out.println("==============zk服务状态===========");   
                   ZooKeeperServerMXBean ZooKeeperServer = JMX.newMBeanProxy(mbsc, roleObjectName, ZooKeeperServerMXBean.class);   
                   System.out.println(role + " 的IP和端口:" + ZooKeeperServer.getClientPort());// IP和端口   
                   System.out.println(role + " 活着的连接数:" + ZooKeeperServer.getNumAliveConnections());// 连接数   
                   System.out.println(role + " 未完成请求数:" + ZooKeeperServer.getOutstandingRequests());// 未完成的请求数   
                   System.out.println(role + " 接收的包:" + ZooKeeperServer.getPacketsReceived());// 收到的包   
                   System.out.println(role + " 发送的包:" + ZooKeeperServer.getPacketsSent());// 发送的包   
                   System.out.println(role + " 平均延迟（毫秒）:" + ZooKeeperServer.getAvgRequestLatency());   
                   System.out.println(role + " 最大延迟（毫秒）:" + ZooKeeperServer.getMaxRequestLatency());   
          
                   System.out.println(role + " 每个客户端IP允许的最大连接数:" + ZooKeeperServer.getMaxClientCnxnsPerHost());   
                   System.out.println(role + " 最大Session超时（毫秒）:" + ZooKeeperServer.getMaxSessionTimeout());   
                   System.out.println(role + " 心跳时间（毫秒）:" + ZooKeeperServer.getTickTime());   
                   System.out.println(role + " 版本:" + ZooKeeperServer.getVersion());// 版本   
                   // 三个重置操作   
       //            ZooKeeperServer.resetLatency(); //重置min/avg/max latency statistics   
       //            ZooKeeperServer.resetMaxLatency(); //重置最大延迟统计   
       //            ZooKeeperServer.resetStatistics(); // 重置包和延迟所有统计   
          
          
                   System.out.println("==============所有客户端的连接信息===========");   
                   ObjectName connectionPattern = new ObjectName("org.apache.ZooKeeperService:name0=ReplicatedServer_id?,name1=replica.?,name2=*,name3=Connections,*");   
                   Set connectionSets = mbsc.queryNames(connectionPattern, null);   
                   List connectionList = new ArrayList(connectionSets.size());   
                   connectionList.addAll(connectionSets);   
                   Collections.sort(connectionList);   
                   for (ObjectName connectionON : connectionList) {   
                       System.out.println("=========================");   
                       ConnectionMXBean connectionBean = JMX.newMBeanProxy(mbsc, connectionON, ConnectionMXBean.class);   
                       System.out.println(" IP+Port:" + connectionBean.getSourceIP());//   
                       System.out.println(" SessionId:" + connectionBean.getSessionId());//   
                       System.out.println(" PacketsReceived:" + connectionBean.getPacketsReceived());// 收到的包   
                       System.out.println(" PacketsSent:" + connectionBean.getPacketsSent());// 发送的包   
                       System.out.println(" MinLatency:" + connectionBean.getMinLatency());//   
                       System.out.println(" AvgLatency:" + connectionBean.getAvgLatency());//   
                       System.out.println(" MaxLatency:" + connectionBean.getMaxLatency());//   
                       System.out.println(" StartedTime:" + connectionBean.getStartedTime());//   
                       System.out.println(" EphemeralNodes:" + connectionBean.getEphemeralNodes().length);//   
                       System.out.println(" EphemeralNodes:" + Arrays.asList(connectionBean.getEphemeralNodes()));//   
                       System.out.println(" OutstandingRequests:" + connectionBean.getOutstandingRequests());//   
                          
                       //connectionBean.resetCounters();   
                       //connectionBean.terminateConnection();   
                       //connectionBean.terminateSession();   
                   }   
               }   
               // close connection   
               if (connector != null) {   
                   connector.close();   
               }   
           }   
          
           /**  
            * 建立连接  
            *  
            * @param ip  
            * @param jmxport  
            * @return  
            */   
           public static MBeanServerConnection createMBeanServer(String ip,   
                   String jmxport, String userName, String password) {   
               try {   
                   String jmxURL = "service:jmx:rmi:///jndi/rmi://" + ip + ":"   
                           + jmxport + "/jmxrmi";   
                   // jmxurl   
                   JMXServiceURL serviceURL = new JMXServiceURL(jmxURL);   
          
                   Map map = new HashMap();   
                   String[] credentials = new String[] { userName, password };   
                   map.put("jmx.remote.credentials", credentials);   
                   connector = JMXConnectorFactory.connect(serviceURL, map);   
                   MBeanServerConnection mbsc = connector.getMBeanServerConnection();   
                   return mbsc;   
          
               } catch (IOException ioe) {   
                   ioe.printStackTrace();   
                   System.err.println(ip + ":" + jmxport + " 连接建立失败");   
               }   
               return null;   
           }   
          
           /**  
            * 使用MBeanServer获取对象名为[objName]的MBean的[objAttr]属性值  
            *   
            * 静态代码: return MBeanServer.getAttribute(ObjectName name, String attribute)  
            *  
            * @param mbeanServer  
            *            - MBeanServer实例  
            * @param objName  
            *            - MBean的对象名  
            * @param objAttr  
            *            - MBean的某个属性名  
            * @return 属性值  
            */   
           private static String getAttribute(MBeanServerConnection mbeanServer,   
                   ObjectName objName, String objAttr) {   
               if (mbeanServer == null || objName == null || objAttr == null)   
                   throw new IllegalArgumentException();   
               try {   
                   return String.valueOf(mbeanServer.getAttribute(objName, objAttr));   
               } catch (Exception e) {   
                   return null;   
               }   
           }   
       }
import java.io.IOException;
import java.lang.management.ClassLoadingMXBean;
import java.lang.management.CompilationMXBean;
import java.lang.management.ManagementFactory;
import java.lang.management.OperatingSystemMXBean;
import java.lang.management.ThreadMXBean;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.Set;

import javax.management.InstanceNotFoundException;
import javax.management.IntrospectionException;
import javax.management.JMX;
import javax.management.MBeanServerConnection;
import javax.management.MalformedObjectNameException;
import javax.management.ObjectName;
import javax.management.ReflectionException;
import javax.management.remote.JMXConnector;
import javax.management.remote.JMXConnectorFactory;
import javax.management.remote.JMXServiceURL;

import org.apache.zookeeper.server.ConnectionMXBean;
import org.apache.zookeeper.server.DataTreeMXBean;
import org.apache.zookeeper.server.ZooKeeperServerMXBean;

public class ZkJMXTest {
    static JMXConnector connector;

    /**
     * @param args
     * @throws IOException
     * @throws MalformedObjectNameException
     * @throws InstanceNotFoundException
     * @throws ReflectionException
     * @throws IntrospectionException
     */
    public static void main(String[] args) throws IOException, MalformedObjectNameException,
        InstanceNotFoundException, IntrospectionException, ReflectionException {

        OperatingSystemMXBean osbean = ManagementFactory.getOperatingSystemMXBean();
        System.out.println("体系结构:" + osbean.getArch());//操作系统体系结构
        System.out.println("处理器核数:" + osbean.getAvailableProcessors());///核数
        System.out.println("名字:" + osbean.getName());//名字

        System.out.println(osbean.getVersion());//操作系统版本
        ThreadMXBean threadBean=ManagementFactory.getThreadMXBean();
        System.out.println("活动线程:" + threadBean.getThreadCount());//总线程数

        ClassLoadingMXBean classLoadingMXBean = ManagementFactory.getClassLoadingMXBean();
        CompilationMXBean compilationMXBean = ManagementFactory.getCompilationMXBean();
        System.out.println("===========");

        // 通过 MBeanServer间接地访问 MXBean 接口
        MBeanServerConnection mbsc = createMBeanServer("192.168.1.100", "9991", "controlRole", "123456");

        // 操作系统
        ObjectName os = new ObjectName("java.lang:type=OperatingSystem");
        System.out.println("体系结构:" + getAttribute(mbsc, os, "Arch"));//体系结构
        System.out.println("处理器核数:" + getAttribute(mbsc, os, "AvailableProcessors"));//核数
        System.out.println("总物理内存:" + getAttribute(mbsc, os, "TotalPhysicalMemorySize"));//总物理内存
        System.out.println("空闲物理内存:" + getAttribute(mbsc, os, "FreePhysicalMemorySize"));//空闲物理内存
        System.out.println("总交换空间:" + getAttribute(mbsc, os, "TotalSwapSpaceSize"));//总交换空间
        System.out.println("空闲交换空间:" + getAttribute(mbsc, os, "FreeSwapSpaceSize"));//空闲交换空间

        System.out.println("操作系统:" + getAttribute(mbsc, os, "Name")+ getAttribute(mbsc, os, "Version"));//操作系统
        System.out.println("提交的虚拟内存:" + getAttribute(mbsc, os, "CommittedVirtualMemorySize"));//提交的虚拟内存
        System.out.println("系统cpu使用率:" + getAttribute(mbsc, os, "SystemCpuLoad"));//系统cpu使用率
        System.out.println("进程cpu使用率:" + getAttribute(mbsc, os, "ProcessCpuLoad"));//进程cpu使用率

        System.out.println("============");//
        // 线程
        ObjectName Threading = new ObjectName("java.lang:type=Threading");
        System.out.println("活动线程:" + getAttribute(mbsc, Threading, "ThreadCount"));// 活动线程
        System.out.println("守护程序线程:" + getAttribute(mbsc, Threading, "DaemonThreadCount"));// 守护程序线程
        System.out.println("峰值:" + getAttribute(mbsc, Threading, "PeakThreadCount"));// 峰值
        System.out.println("启动的线程总数:" + getAttribute(mbsc, Threading, "TotalStartedThreadCount"));// 启动的线程总数
        ThreadMXBean threadBean2 = ManagementFactory.newPlatformMXBeanProxy
                (mbsc, ManagementFactory.THREAD_MXBEAN_NAME, ThreadMXBean.class);
        System.out.println("活动线程:" + threadBean2.getThreadCount());// 活动线程
        ThreadMXBean threadBean3 = ManagementFactory.getThreadMXBean();
        System.out.println("本地活动线程:" + threadBean3.getThreadCount());// 本地活动线程

        System.out.println("============");//
        ObjectName Compilation = new ObjectName("java.lang:type=Compilation");
        System.out.println("总编译时间 毫秒:" + getAttribute(mbsc, Compilation, "TotalCompilationTime"));// 总编译时间 毫秒

        System.out.println("============");//
        ObjectName ClassLoading = new ObjectName("java.lang:type=ClassLoading");
        System.out.println("已加载类总数:" + getAttribute(mbsc, ClassLoading, "TotalLoadedClassCount"));// 已加载类总数
        System.out.println("已加装当前类:" + getAttribute(mbsc, ClassLoading, "LoadedClassCount"));// 已加装当前类
        System.out.println("已卸载类总数:" + getAttribute(mbsc, ClassLoading, "UnloadedClassCount"));// 已卸载类总数


        System.out.println("==========================================================");//
        // http://zookeeper.apache.org/doc/r3.4.6/zookeeperJMX.html
        // org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.1,name2=Follower
        ObjectName replica = new ObjectName("org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.1");
        System.out.println("replica.1运行状态:" + getAttribute(mbsc, replica, "State"));// 运行状态

        mbsc = createMBeanServer("192.168.1.100", "9992", "controlRole", "123456");
        System.out.println("==============节点树对象===========");
        ObjectName dataTreePattern = new ObjectName("org.apache.ZooKeeperService:name0=ReplicatedServer_id?,name1=replica.?,name2=*,name3=InMemoryDataTree");
        Set dataTreeSets = mbsc.queryNames(dataTreePattern, null);
        Iterator dataTreeIterator = dataTreeSets.iterator();
        // 只有一个
        while (dataTreeIterator.hasNext()) {
            ObjectName dataTreeObjectName = dataTreeIterator.next();
            DataTreeMXBean dataTree = JMX.newMBeanProxy(mbsc, dataTreeObjectName, DataTreeMXBean.class);
            System.out.println("节点总数:" + dataTree.getNodeCount());// 节点总数
            System.out.println("Watch总数:" + dataTree.getWatchCount());// Watch总数
            System.out.println("临时节点总数:" + dataTree.countEphemerals());// Watch总数
            System.out.println("节点名及字符总数:" + dataTree.approximateDataSize());// 节点全路径和值的总字符数

            Map dataTreeMap = dataTreeObjectName.getKeyPropertyList();
            String replicaId = dataTreeMap.get("name1").replace("replica.", "");
            String role = dataTreeMap.get("name2");// Follower,Leader,Observer,Standalone
            String canonicalName = dataTreeObjectName.getCanonicalName();
            int roleEndIndex = canonicalName.indexOf(",name3");

            ObjectName roleObjectName = new ObjectName(canonicalName.substring(0, roleEndIndex));
            System.out.println("==============zk服务状态===========");
            ZooKeeperServerMXBean ZooKeeperServer = JMX.newMBeanProxy(mbsc, roleObjectName, ZooKeeperServerMXBean.class);
            System.out.println(role + " 的IP和端口:" + ZooKeeperServer.getClientPort());// IP和端口
            System.out.println(role + " 活着的连接数:" + ZooKeeperServer.getNumAliveConnections());// 连接数
            System.out.println(role + " 未完成请求数:" + ZooKeeperServer.getOutstandingRequests());// 未完成的请求数
            System.out.println(role + " 接收的包:" + ZooKeeperServer.getPacketsReceived());// 收到的包
            System.out.println(role + " 发送的包:" + ZooKeeperServer.getPacketsSent());// 发送的包
            System.out.println(role + " 平均延迟（毫秒）:" + ZooKeeperServer.getAvgRequestLatency());
            System.out.println(role + " 最大延迟（毫秒）:" + ZooKeeperServer.getMaxRequestLatency());

            System.out.println(role + " 每个客户端IP允许的最大连接数:" + ZooKeeperServer.getMaxClientCnxnsPerHost());
            System.out.println(role + " 最大Session超时（毫秒）:" + ZooKeeperServer.getMaxSessionTimeout());
            System.out.println(role + " 心跳时间（毫秒）:" + ZooKeeperServer.getTickTime());
            System.out.println(role + " 版本:" + ZooKeeperServer.getVersion());// 版本
            // 三个重置操作
//            ZooKeeperServer.resetLatency(); //重置min/avg/max latency statistics
//            ZooKeeperServer.resetMaxLatency(); //重置最大延迟统计
//            ZooKeeperServer.resetStatistics(); // 重置包和延迟所有统计


            System.out.println("==============所有客户端的连接信息===========");
            ObjectName connectionPattern = new ObjectName("org.apache.ZooKeeperService:name0=ReplicatedServer_id?,name1=replica.?,name2=*,name3=Connections,*");
            Set connectionSets = mbsc.queryNames(connectionPattern, null);
            List connectionList = new ArrayList(connectionSets.size());
            connectionList.addAll(connectionSets);
            Collections.sort(connectionList);
            for (ObjectName connectionON : connectionList) {
                System.out.println("=========================");
                ConnectionMXBean connectionBean = JMX.newMBeanProxy(mbsc, connectionON, ConnectionMXBean.class);
                System.out.println(" IP+Port:" + connectionBean.getSourceIP());//
                System.out.println(" SessionId:" + connectionBean.getSessionId());//
                System.out.println(" PacketsReceived:" + connectionBean.getPacketsReceived());// 收到的包
                System.out.println(" PacketsSent:" + connectionBean.getPacketsSent());// 发送的包
                System.out.println(" MinLatency:" + connectionBean.getMinLatency());//
                System.out.println(" AvgLatency:" + connectionBean.getAvgLatency());//
                System.out.println(" MaxLatency:" + connectionBean.getMaxLatency());//
                System.out.println(" StartedTime:" + connectionBean.getStartedTime());//
                System.out.println(" EphemeralNodes:" + connectionBean.getEphemeralNodes().length);//
                System.out.println(" EphemeralNodes:" + Arrays.asList(connectionBean.getEphemeralNodes()));//
                System.out.println(" OutstandingRequests:" + connectionBean.getOutstandingRequests());//

                //connectionBean.resetCounters();
                //connectionBean.terminateConnection();
                //connectionBean.terminateSession();
            }
        }
        // close connection
        if (connector != null) {
            connector.close();
        }
    }

    /**
     * 建立连接
     *
     * @param ip
     * @param jmxport
     * @return
     */
    public static MBeanServerConnection createMBeanServer(String ip,
            String jmxport, String userName, String password) {
        try {
            String jmxURL = "service:jmx:rmi:///jndi/rmi://" + ip + ":"
                    + jmxport + "/jmxrmi";
            // jmxurl
            JMXServiceURL serviceURL = new JMXServiceURL(jmxURL);

            Map map = new HashMap();
            String[] credentials = new String[] { userName, password };
            map.put("jmx.remote.credentials", credentials);
            connector = JMXConnectorFactory.connect(serviceURL, map);
            MBeanServerConnection mbsc = connector.getMBeanServerConnection();
            return mbsc;

        } catch (IOException ioe) {
            ioe.printStackTrace();
            System.err.println(ip + ":" + jmxport + " 连接建立失败");
        }
        return null;
    }

    /**
     * 使用MBeanServer获取对象名为[objName]的MBean的[objAttr]属性值
     *
     * 静态代码: return MBeanServer.getAttribute(ObjectName name, String attribute)
     *
     * @param mbeanServer
     *            - MBeanServer实例
     * @param objName
     *            - MBean的对象名
     * @param objAttr
     *            - MBean的某个属性名
     * @return 属性值
     */
    private static String getAttribute(MBeanServerConnection mbeanServer,
            ObjectName objName, String objAttr) {
        if (mbeanServer == null || objName == null || objAttr == null)
            throw new IllegalArgumentException();
        try {
            return String.valueOf(mbeanServer.getAttribute(objName, objAttr));
        } catch (Exception e) {
            return null;
        }
    }
}