Hadoop Metrics研究—2

hadoop metrics2

监控的内容: 
1. yarn 
2. jvm 
3. rpc 
4. rpcdetailed 
5. metricssystem 
6. mapred 
7. dfs 
8. ugi

已经提供的:

Source : org.apache.hadoop.metrics2.source.JvmMertics org.apache.hadoop.metrics2.source.JvmMetricsInfo

其他相关

FSOpDurations : org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSOpDurations

fairscheduler-op-durations context :

 @Metric("Duration for a continuous scheduling run")  MutableRate continuousSchedulingRun;
  @Metric("Duration to handle a node update")  MutableRate nodeUpdateCall;
  @Metric("Duration for a update thread run")  MutableRate updateThreadRun;
  @Metric("Duration for an update call")  MutableRate updateCall;
  @Metric("Duration for a preempt call") MutableRate preemptCall;

QueueMetrics : org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics

yarn context :

@Metric("# of apps submitted") MutableCounterInt appsSubmitted;
  @Metric("# of running apps") MutableGaugeInt appsRunning;
  @Metric("# of pending apps") MutableGaugeInt appsPending;
  @Metric("# of apps completed") MutableCounterInt appsCompleted;
  @Metric("# of apps killed") MutableCounterInt appsKilled;
  @Metric("# of apps failed") MutableCounterInt appsFailed;
  @Metric("Allocated memory in MB") MutableGaugeInt allocatedMB;
  @Metric("Allocated CPU in virtual cores") MutableGaugeInt allocatedVCores;
  @Metric("# of allocated containers") MutableGaugeInt allocatedContainers;
  @Metric("Aggregate # of allocated containers") MutableCounterLong aggregateContainersAllocated;
  @Metric("Aggregate # of released containers") MutableCounterLong aggregateContainersReleased;
  @Metric("Available memory in MB") MutableGaugeInt availableMB;
  @Metric("Available CPU in virtual cores") MutableGaugeInt availableVCores;
  @Metric("Pending memory allocation in MB") MutableGaugeInt pendingMB;
  @Metric("Pending CPU allocation in virtual cores") MutableGaugeInt pendingVCores;
  @Metric("# of pending containers") MutableGaugeInt pendingContainers;
  @Metric("# of reserved memory in MB") MutableGaugeInt reservedMB;
  @Metric("Reserved CPU in virtual cores") MutableGaugeInt reservedVCores;
  @Metric("# of reserved containers") MutableGaugeInt reservedContainers;
  @Metric("# of active users") MutableGaugeInt activeUsers;
  @Metric("# of active applications") MutableGaugeInt activeApplications;

FSQueueMetrics : org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSQueueMetrics

yarn context :

@Metric("Fair share of memory in MB") MutableGaugeInt fairShareMB;
  @Metric("Fair share of CPU in vcores") MutableGaugeInt fairShareVCores;
  @Metric("Steady fair share of memory in MB") MutableGaugeInt steadyFairShareMB;
  @Metric("Steady fair share of CPU in vcores") MutableGaugeInt steadyFairShareVCores;
  @Metric("Minimum share of memory in MB") MutableGaugeInt minShareMB;
  @Metric("Minimum share of CPU in vcores") MutableGaugeInt minShareVCores;
  @Metric("Maximum share of memory in MB") MutableGaugeInt maxShareMB;
  @Metric("Maximum share of CPU in vcores") MutableGaugeInt maxShareVCores;

MetricsSystemImpl : org.apache.hadoop.metrics2.impl.MetricsSystemImpl

metricssystem context :

@Metric({"Snapshot", "Snapshot stats"}) MutableStat snapshotStat;
  @Metric({"Publish", "Publishing stats"}) MutableStat publishStat;
  @Metric("Dropped updates by all sinks") MutableCounterLong droppedPubAll

Sink : org.apache.hadoop.metrics2.sink.GraphiteSink org.apache.hadoop.metrics2.sink.FileSink org.apache.hadoop.metrics2.sink.AbstractGangliaSink

metricsSystem : org.apache.hadoop.metrics2.lib.DefaultMetricsSystem

自我实现:

1. Source : org.apache.hadoop.metrics2.MetricsSource
2. Sink : org.apache.hadoop.metrics2.MetricsSink
3. MetricsSystem : org.apache.hadoop.metrics2.MetricsSystem

使用方式:

$HADOOP_HOME/etc/hadoop/hadoop-metrics2.properties中配置就可以