<configuration>
<property>
<name>mapreduce.job.hdfs-serversname>
<value>${fs.defaultFS}value>
property>
<property>
<name>mapreduce.job.committer.setup.cleanup.neededname>
<value>truevalue>
<description> true, if job needs job-setup and job-cleanup.
false, otherwise
description>
property>
<property>
<name>mapreduce.task.io.sort.factorname>
<value>10value>
<description>The number of streams to merge at once while sorting
files. This determines the number of open file handles.description>
property>
<property>
<name>mapreduce.task.io.sort.mbname>
<value>100value>
<description>The total amount of buffer memory to use while sorting
files, in megabytes. By default, gives each merge stream 1MB, which
should minimize seeks.description>
property>
<property>
<name>mapreduce.map.sort.spill.percentname>
<value>0.80value>
<description>The soft limit in the serialization buffer. Once reached, a
thread will begin to spill the contents to disk in the background. Note that
collection will not block if this threshold is exceeded while a spill is
already in progress, so spills may be larger than this threshold when it is
set to less than .5description>
property>
<property>
<name>mapreduce.job.local-fs.single-disk-limit.bytesname>
<value>-1value>
<description>Enable an in task monitor thread to watch for single disk
consumption by jobs. By setting this to x nr of bytes, the task will fast
fail in case it is reached. This is a per disk configuration.description>
property>
<property>
<name>mapreduce.job.local-fs.single-disk-limit.check.interval-msname>
<value>5000value>
<description>Interval of disk limit check to run in ms.description>
property>
<property>
<name>mapreduce.job.local-fs.single-disk-limit.check.kill-limit-exceedname>
<value>truevalue>
<description>If mapreduce.job.local-fs.single-disk-limit.bytes is triggered
should the task be killed or logged. If false the intent to kill the task
is only logged in the container logs.description>
property>
<property>
<name>mapreduce.job.mapsname>
<value>2value>
<description>The default number of map tasks per job.
Ignored when mapreduce.framework.name is "local".
description>
property>
<property>
<name>mapreduce.job.reducesname>
<value>1value>
<description>The default number of reduce tasks per job. Typically set to 99%
of the cluster's reduce capacity, so that if a node fails the reduces can
still be executed in a single wave.
Ignored when mapreduce.framework.name is "local".
description>
property>
<property>
<name>mapreduce.job.running.map.limitname>
<value>0value>
<description>The maximum number of simultaneous map tasks per job.
There is no limit if this value is 0 or negative.
description>
property>
<property>
<name>mapreduce.job.running.reduce.limitname>
<value>0value>
<description>The maximum number of simultaneous reduce tasks per job.
There is no limit if this value is 0 or negative.
description>
property>
<property>
<name>mapreduce.job.max.mapname>
<value>-1value>
<description>Limit on the number of map tasks allowed per job.
There is no limit if this value is negative.
description>
property>
<property>
<name>mapreduce.job.reducer.preempt.delay.secname>
<value>0value>
<description>The threshold (in seconds) after which an unsatisfied
mapper request triggers reducer preemption when there is no anticipated
headroom. If set to 0 or a negative value, the reducer is preempted as
soon as lack of headroom is detected. Default is 0.
description>
property>
<property>
<name>mapreduce.job.reducer.unconditional-preempt.delay.secname>
<value>300value>
<description>The threshold (in seconds) after which an unsatisfied
mapper request triggers a forced reducer preemption irrespective of the
anticipated headroom. By default, it is set to 5 mins. Setting it to 0
leads to immediate reducer preemption. Setting to -1 disables this
preemption altogether.
description>
property>
<property>
<name>mapreduce.job.max.split.locationsname>
<value>15value>
<description>The max number of block locations to store for each split for
locality calculation.
description>
property>
<property>
<name>mapreduce.job.split.metainfo.maxsizename>
<value>10000000value>
<description>The maximum permissible size of the split metainfo file.
The MapReduce ApplicationMaster won't attempt to read submitted split metainfo
files bigger than this configured value.
No limits if set to -1.
description>
property>
<property>
<name>mapreduce.map.maxattemptsname>
<value>4value>
<description>Expert: The maximum number of attempts per map task.
In other words, framework will try to execute a map task these many number
of times before giving up on it.
description>
property>
<property>
<name>mapreduce.reduce.maxattemptsname>
<value>4value>
<description>Expert: The maximum number of attempts per reduce task.
In other words, framework will try to execute a reduce task these many number
of times before giving up on it.
description>
property>
<property>
<name>mapreduce.reduce.shuffle.fetch.retry.enabledname>
<value>${yarn.nodemanager.recovery.enabled}value>
<description>Set to enable fetch retry during host restart.description>
property>
<property>
<name>mapreduce.reduce.shuffle.fetch.retry.interval-msname>
<value>1000value>
<description>Time of interval that fetcher retry to fetch again when some
non-fatal failure happens because of some events like NM restart.
description>
property>
<property>
<name>mapreduce.reduce.shuffle.fetch.retry.timeout-msname>
<value>30000value>
<description>Timeout value for fetcher to retry to fetch again when some
non-fatal failure happens because of some events like NM restart.description>
property>
<property>
<name>mapreduce.reduce.shuffle.retry-delay.max.msname>
<value>60000value>
<description>The maximum number of ms the reducer will delay before retrying
to download map data.
description>
property>
<property>
<name>mapreduce.reduce.shuffle.parallelcopiesname>
<value>5value>
<description>The default number of parallel transfers run by reduce
during the copy(shuffle) phase.
description>
property>
<property>
<name>mapreduce.reduce.shuffle.connect.timeoutname>
<value>180000value>
<description>Expert: The maximum amount of time (in milli seconds) reduce
task spends in trying to connect to a remote node for getting map output.
description>
property>
<property>
<name>mapreduce.reduce.shuffle.read.timeoutname>
<value>180000value>
<description>Expert: The maximum amount of time (in milli seconds) reduce
task waits for map output data to be available for reading after obtaining
connection.
description>
property>
<property>
<name>mapreduce.shuffle.listen.queue.sizename>
<value>128value>
<description>The length of the shuffle server listen queue.description>
property>
<property>
<name>mapreduce.shuffle.connection-keep-alive.enablename>
<value>falsevalue>
<description>set to true to support keep-alive connections.description>
property>
<property>
<name>mapreduce.shuffle.connection-keep-alive.timeoutname>
<value>5value>
<description>The number of seconds a shuffle client attempts to retain
http connection. Refer "Keep-Alive: timeout=" header in
Http specification
description>
property>
<property>
<name>mapreduce.task.timeoutname>
<value>600000value>
<description>The number of milliseconds before a task will be
terminated if it neither reads an input, writes an output, nor
updates its status string. A value of 0 disables the timeout.
description>
property>
<property>
<name>mapreduce.map.memory.mbname>
<value>-1value>
<description>The amount of memory to request from the scheduler for each
map task. If this is not specified or is non-positive, it is inferred from
mapreduce.map.java.opts and mapreduce.job.heap.memory-mb.ratio.
If java-opts are also not specified, we set it to 1024.
description>
property>
<property>
<name>mapreduce.map.cpu.vcoresname>
<value>1value>
<description>The number of virtual cores to request from the scheduler for
each map task.
description>
property>
<property>
<name>mapreduce.reduce.memory.mbname>
<value>-1value>
<description>The amount of memory to request from the scheduler for each
reduce task. If this is not specified or is non-positive, it is inferred
from mapreduce.reduce.java.opts and mapreduce.job.heap.memory-mb.ratio.
If java-opts are also not specified, we set it to 1024.
description>
property>
<property>
<name>mapreduce.reduce.cpu.vcoresname>
<value>1value>
<description>The number of virtual cores to request from the scheduler for
each reduce task.
description>
property>
<property>
<name>mapred.child.java.optsname>
<value>value>
<description>Java opts for the task processes.
The following symbol, if present, will be interpolated: @taskid@ is replaced
by current TaskID. Any other occurrences of '@' will go unchanged.
For example, to enable verbose gc logging to a file named for the taskid in
/tmp and to set the heap maximum to be a gigabyte, pass a 'value' of:
-Xmx1024m -verbose:gc -Xloggc:/tmp/@[email protected]
Usage of -Djava.library.path can cause programs to no longer function if
hadoop native libraries are used. These values should instead be set as part
of LD_LIBRARY_PATH in the map / reduce JVM env using the mapreduce.map.env and
mapreduce.reduce.env config settings.
If -Xmx is not set, it is inferred from mapreduce.{map|reduce}.memory.mb and
mapreduce.job.heap.memory-mb.ratio.
description>
property>
<property>
<name>mapred.child.envname>
<value>value>
<description>User added environment variables for the task processes.
Example :
1) A=foo This will set the env variable A to foo
2) B=$B:c This is inherit nodemanager's B env variable on Unix.
3) B=%B%;c This is inherit nodemanager's B env variable on Windows.
description>
property>
<property>
<name>mapreduce.admin.user.envname>
<value>value>
<description>
Expert: Additional execution environment entries for
map and reduce task processes. This is not an additive property.
You must preserve the original value if you want your map and
reduce tasks to have access to native libraries (compression, etc).
When this value is empty, the command to set execution
envrionment will be OS dependent:
For linux, use LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native.
For windows, use PATH = %PATH%;%HADOOP_COMMON_HOME%\\bin.
description>
property>
<property>
<name>yarn.app.mapreduce.am.log.levelname>
<value>INFOvalue>
<description>The logging level for the MR ApplicationMaster. The allowed
levels are: OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE and ALL.
The setting here could be overriden if "mapreduce.job.log4j-properties-file"
is set.
description>
property>
<property>
<name>mapreduce.map.log.levelname>
<value>INFOvalue>
<description>The logging level for the map task. The allowed levels are:
OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE and ALL.
The setting here could be overridden if "mapreduce.job.log4j-properties-file"
is set.
description>
property>
<property>
<name>mapreduce.reduce.log.levelname>
<value>INFOvalue>
<description>The logging level for the reduce task. The allowed levels are:
OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE and ALL.
The setting here could be overridden if "mapreduce.job.log4j-properties-file"
is set.
description>
property>
<property>
<name>mapreduce.reduce.merge.inmem.thresholdname>
<value>1000value>
<description>The threshold, in terms of the number of files
for the in-memory merge process. When we accumulate threshold number of files
we initiate the in-memory merge and spill to disk. A value of 0 or less than
0 indicates we want to DON'T have any threshold and instead depend only on
the ramfs's memory consumption to trigger the merge.
description>
property>
<property>
<name>mapreduce.reduce.shuffle.merge.percentname>
<value>0.66value>
<description>The usage threshold at which an in-memory merge will be
initiated, expressed as a percentage of the total memory allocated to
storing in-memory map outputs, as defined by
mapreduce.reduce.shuffle.input.buffer.percent.
description>
property>
<property>
<name>mapreduce.reduce.shuffle.input.buffer.percentname>
<value>0.70value>
<description>The percentage of memory to be allocated from the maximum heap
size to storing map outputs during the shuffle.
description>
property>
<property>
<name>mapreduce.reduce.input.buffer.percentname>
<value>0.0value>
<description>The percentage of memory- relative to the maximum heap size- to
retain map outputs during the reduce. When the shuffle is concluded, any
remaining map outputs in memory must consume less than this threshold before
the reduce can begin.
description>
property>
<property>
<name>mapreduce.reduce.shuffle.memory.limit.percentname>
<value>0.25value>
<description>Expert: Maximum percentage of the in-memory limit that a
single shuffle can consume. Range of valid values is [0.0, 1.0]. If the value
is 0.0 map outputs are shuffled directly to disk.description>
property>
<property>
<name>mapreduce.shuffle.ssl.enabledname>
<value>falsevalue>
<description>
Whether to use SSL for for the Shuffle HTTP endpoints.
description>
property>
<property>
<name>mapreduce.shuffle.ssl.file.buffer.sizename>
<value>65536value>
<description>Buffer size for reading spills from file when using SSL.
description>
property>
<property>
<name>mapreduce.shuffle.max.connectionsname>
<value>0value>
<description>Max allowed connections for the shuffle. Set to 0 (zero)
to indicate no limit on the number of connections.
description>
property>
<property>
<name>mapreduce.shuffle.max.threadsname>
<value>0value>
<description>Max allowed threads for serving shuffle connections. Set to zero
to indicate the default of 2 times the number of available
processors (as reported by Runtime.availableProcessors()). Netty is used to
serve requests, so a thread is not needed for each connection.
description>
property>
<property>
<name>mapreduce.shuffle.transferTo.allowedname>
<value>value>
<description>This option can enable/disable using nio transferTo method in
the shuffle phase. NIO transferTo does not perform well on windows in the
shuffle phase. Thus, with this configuration property it is possible to
disable it, in which case custom transfer method will be used. Recommended
value is false when running Hadoop on Windows. For Linux, it is recommended
to set it to true. If nothing is set then the default value is false for
Windows, and true for Linux.
description>
property>
<property>
<name>mapreduce.shuffle.transfer.buffer.sizename>
<value>131072value>
<description>This property is used only if
mapreduce.shuffle.transferTo.allowed is set to false. In that case,
this property defines the size of the buffer used in the buffer copy code
for the shuffle phase. The size of this buffer determines the size of the IO
requests.
description>
property>
<property>
<name>mapreduce.reduce.markreset.buffer.percentname>
<value>0.0value>
<description>The percentage of memory -relative to the maximum heap size- to
be used for caching values when using the mark-reset functionality.
description>
property>
<property>
<name>mapreduce.map.speculativename>
<value>truevalue>
<description>If true, then multiple instances of some map tasks
may be executed in parallel.description>
property>
<property>
<name>mapreduce.reduce.speculativename>
<value>truevalue>
<description>If true, then multiple instances of some reduce tasks
may be executed in parallel.description>
property>
<property>
<name>mapreduce.job.speculative.speculative-cap-running-tasksname>
<value>0.1value>
<description>The max percent (0-1) of running tasks that
can be speculatively re-executed at any time.description>
property>
<property>
<name>mapreduce.job.speculative.speculative-cap-total-tasksname>
<value>0.01value>
<description>The max percent (0-1) of all tasks that
can be speculatively re-executed at any time.description>
property>
<property>
<name>mapreduce.job.speculative.minimum-allowed-tasksname>
<value>10value>
<description>The minimum allowed tasks that
can be speculatively re-executed at any time.description>
property>
<property>
<name>mapreduce.job.speculative.retry-after-no-speculatename>
<value>1000value>
<description>The waiting time(ms) to do next round of speculation
if there is no task speculated in this round.description>
property>
<property>
<name>mapreduce.job.speculative.retry-after-speculatename>
<value>15000value>
<description>The waiting time(ms) to do next round of speculation
if there are tasks speculated in this round.description>
property>
<property>
<name>mapreduce.job.map.output.collector.classname>
<value>org.apache.hadoop.mapred.MapTask$MapOutputBuffervalue>
<description>
The MapOutputCollector implementation(s) to use. This may be a comma-separated
list of class names, in which case the map task will try to initialize each
of the collectors in turn. The first to successfully initialize will be used.
description>
property>
<property>
<name>mapreduce.job.speculative.slowtaskthresholdname>
<value>1.0value>
<description>The number of standard deviations by which a task's
ave progress-rates must be lower than the average of all running tasks'
for the task to be considered too slow.
description>