Hadoop study notes - monitoring metrics

There is a good blog on this.
http://www.cloudera.com/blog/2009/03/hadoop-metrics/


The HDFS and MapReduce daemons collect information about events and measurements
that are collectively known as metrics. For example, datanodes collect the following
metrics (and many more): the number of bytes written, the number of blocks
replicated, and the number of read requests from clients (both local and remote).

Metrics belong to a context, and Hadoop currently uses “dfs”, “mapred”, “rpc”, and
“jvm” contexts. Hadoop daemons usually collect metrics under several contexts. For
example, datanodes collect metrics for the “dfs”, “rpc”, and “jvm” contexts.

How Do Metrics Differ from Counters?

The main difference is their scope: metrics are collected by Hadoop daemons, whereas
counters  are collected for MapReduce tasks and aggregated
for the whole job. They have different audiences, too: broadly speaking, metrics
are for administrators, and counters are for MapReduce users.
The way they are collected and aggregated is also different. Counters are a MapReduce
feature, and the MapReduce system ensures that counter values are propagated from
the tasktrackers where they are produced, back to the jobtracker, and finally back to
the client running the MapReduce job. (Counters are propagated via RPC heartbeats;) Both the tasktrackers and the jobtracker
perform aggregation.
The collection mechanism for metrics is decoupled from the component that receives
the updates, and there are various pluggable outputs, including local files, Ganglia, and
JMX. The daemon collecting the metrics performs aggregation on them before they are
sent to the output.


Metrics are configured in the conf/hadoopmetrics.properties file

MetricsContext types
NullContext
FileContext
GangliaContext
NullContextWithUpdateThread
CompositeContext