Giraph运行报错，Error: Exceeded limits on number of counters - Counters=120 Limit=120, exiting...

HamaWhite 原创，转载请注明出处！欢迎大家加入Giraph 技术交流群： 228591158

1. 近日用Giraph跑大数据的SSSP时，遇到如下错误：

org.apache.hadoop.mapred.Counters$CountersExceededException: Error: Exceeded limits on number of counters - Counters=120 Limit=120 at org.apache.hadoop.mapred.Counters$Group.getCounterForName(Counters.java:315) at org.apache.hadoop.mapred.Counters.findCounter(Counters.java:449) at org.apache.hadoop.mapred.Task$TaskReporter.getCounter(Task.java:559) at org.apache.hadoop.mapred.Task$TaskReporter.getCounter(Task.java:506) at org.apache.hadoop.mapreduce.TaskInputOutputContext.getCounter(TaskInputOutputContext.java:88) at org.apache.giraph.counters.HadoopCountersBase.getCounter(HadoopCountersBase.java:60) at org.apache.giraph.counters.GiraphTimers.getSuperstepMs(GiraphTimers.java:117) at org.apache.giraph.master.MasterThread.run(MasterThread.java:127) FATAL graph.GraphMapper (GraphMapper.java:uncaughtException(113)) - uncaughtException: OverrideExceptionHandler on thread org.apache.giraph.master.MasterThread, msg = org.apache.hadoop.mapred.Counters$CountersExceededException: Error: Exceeded limits on number of counters - Counters=120 Limit=120, exiting... java.lang.IllegalStateException: org.apache.hadoop.mapred.Counters$CountersExceededException: Error: Exceeded limits on number of counters - Counters=120 Limit=120 at org.apache.giraph.master.MasterThread.run(MasterThread.java:181) Caused by: org.apache.hadoop.mapred.Counters$CountersExceededException: Error: Exceeded limits on number of counters - Counters=120 Limit=120 at org.apache.hadoop.mapred.Counters$Group.getCounterForName(Counters.java:315) at org.apache.hadoop.mapred.Counters.findCounter(Counters.java:449) at org.apache.hadoop.mapred.Task$TaskReporter.getCounter(Task.java:559) at org.apache.hadoop.mapred.Task$TaskReporter.getCounter(Task.java:506) at org.apache.hadoop.mapreduce.TaskInputOutputContext.getCounter(TaskInputOutputContext.java:88) at org.apache.giraph.counters.HadoopCountersBase.getCounter(HadoopCountersBase.java:60) at org.apache.giraph.counters.GiraphTimers.getSuperstepMs(GiraphTimers.java:117) at org.apache.giraph.master.MasterThread.run(MasterThread.java:127) INFO zk.ZooKeeperManager (ZooKeeperManager.java:run(655)) - run: Shutdown hook started. WARN zk.ZooKeeperManager (ZooKeeperManager.java:run(658)) - onlineZooKeeperServers: Forced a shutdown hook kill of the ZooKeeper process. INFO zookeeper.ClientCnxn (ClientCnxn.java:run(1159)) - Unable to read additional data from server sessionid 0x145750f19970000, likely server has closed socket, closing socket connection and attempting reconnect INFO zk.ZooKeeperManager (ZooKeeperManager.java:run(668)) - onlineZooKeeperServers: ZooKeeper process exited with 143 (note that 143 typically means killed).

2. 错误原因分析：该Job的Counters数量限制超过默认的120 。mapred-default.xml配置文件的默认如下：

      <property>
          <name>mapreduce.job.counters.limit</name>
          <value>120</value>
          <description>Limit on the number of counters allowed per job. </description>
     </property>

3. 修改配置文件mapred-site.xml ，添加如下内容：

      <property>
          <name>mapreduce.job.counters.limit</name>
          <value>1200</value>
          <description>Limit on the number of counters allowed per job. </description>
     </property>

4. 停止Hadoop服务 。

5. 把修改后的配置文件发送到集群中的每个节点上。

6. 启动Hadoop服务。

完毕！