拉普达男孩

Hadoop集群搭建1Master、3Slaves

master.hadoop 10.9.148.193
slave1.hadoop 10.9.148.190
slave2.hadoop 10.9.148.191
slave3.hadoop 10.9.148.192
root 123456 hadoop 123456
1. 重装系统
Centos6
2. 设置固定IP
IPv4：
手动
地址子网掩码网关
10.9.148.193 255.255.255.0 10.9.148.1
DNS：10.9.148.1
应用
3. 设置sshd启动和开机启动（centos6）
rpm –qa | grep ssh
….
service sshd start
chkconfig sshd on
4. 设置主机名
Ubuntu：10.9.148.193 master.hadoop
[root@master1 ~]# vi /etc/sysconfig/network
[root@master1 ~]# vi /etc/hostname
[root@master1 ~]# vi /etc/hosts
Centos:其它三台
[root@localhost ~]# vi /etc/hosts
127.0.0.1 localhost

::1 localhost6.localdomain6 localhost6

#hadoop
10.9.148.193 master.hadoop
10.9.148.190 slave1.hadoop
10.9.148.191 slave2.hadoop
10.9.148.192 slave3.hadoop
#hbase Master Slave IP address
10.9.148.193 master
10.9.148.190 slave1
10.9.148.191 slave2
10.9.148.192 slave3
[root@localhost ~]# vi /etc/sysconfig/network
[root@localhost ~]# reboot
5. 安装JDK
5.1Centos（为新装的系统，删除自带的java）
[root@slave1 local]# rpm -qa | grep gcj
java-1.5.0-gcj-1.5.0.0-29.1.el6.x86_64
libgcj-4.4.7-18.el6.x86_64
[root@slave1 local]# yum -y remove java java-1.5.0-gcj-1.5.0.0-29.1.el6.x86_64
5.2安装
[root@slave1 local]# tar -zxf jdk-7u45-linux-x64.tar.gz
[root@slave1 local]# ls
bin games jdk1.7.0_45 lib libexec share
etc include jdk-7u45-linux-x64.tar.gz lib64 sbin src
[root@slave1 local]# mv jdk1.7.0_45 jdk1.7
[root@slave1 local]# ls
bin games jdk1.7 lib libexec share
etc include jdk-7u45-linux-x64.tar.gz lib64 sbin src
[root@slave1 local]# rm -rf jdk-7u45-linux-x64.tar.gz
[root@slave1 local]# ls
bin etc games include jdk1.7 lib lib64 libexec sbin share src
[root@slave1 local]# vi /etc/profile
#java
export JAVA_HOME=/usr/local/jdk1.7
export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib
export PATH=$PATH:$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$CLASSPATH:
[root@slave1 local]# source /etc/profile
[root@slave1 local]# java -version
java version "1.7.0_45"
Java(TM) SE Runtime Environment (build 1.7.0_45-b18)
Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode)
[root@slave1 local]#
6. SSH无密钥验证配置（其它的节点同理，只需master和slave之间互相无密钥登录即可，这里slave和slave之间也配置了，可忽略）
[root@slave3 ~]# ssh-keygen -t rsa -P ''
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
43:a0:f3:40:59:27:76:fc:fb:55:c7:d3:8b:66:28:00 [email protected]
The key's randomart image is:
+--[ RSA 2048]----+
| .o=.o |
| ..E =. |
| + . .. ..|
| + o . .=|
| . S .. ..+|
| o.. +.. |
| ..o. |
| . |
| |
+-----------------+
[root@slave3 ~]# cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
[root@slave3 ~]# cd .ssh/
[root@slave3 .ssh]# chmod 600 authorized_keys
[root@slave3 .ssh]# vi /etc/ssh/sshd_config
修改内容：
去掉前面的“#”注释
RSAAuthentication yes #启用RSA认证
PubkeyAuthentication yes #启用公钥私钥配对认证方式
[root@slave3 .ssh]# cat ~/.ssh/id_rsa.pub | ssh [email protected] 'cat >> ~/.ssh/authorized_keys'
The authenticity of host 'slave2.hadoop (10.9.148.191)' can't be established.
RSA key fingerprint is 82:e8:77:8c:32:ed:6a:62:6c:c5:55:0d:10:4a:84:30.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'slave2.hadoop,10.9.148.191' (RSA) to the list of known hosts.
[email protected]'s password:
[root@slave3 .ssh]# cat ~/.ssh/id_rsa.pub | ssh [email protected] 'cat >> ~/.ssh/authorized_keys'
The authenticity of host 'slave1.hadoop (10.9.148.190)' can't be established.
RSA key fingerprint is 96:37:8a:3f:b1:1d:6b:c1:f7:cc:3b:c2:3e:03:09:07.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'slave1.hadoop,10.9.148.190' (RSA) to the list of known hosts.
[email protected]'s password:
[root@slave3 .ssh]# cat ~/.ssh/id_rsa.pub | ssh [email protected] 'cat >> ~/.ssh/authorized_keys'
//追加到主机Master的无密钥登录配置里
The authenticity of host 'master.hadoop (10.9.148.193)' can't be established.
RSA key fingerprint is fc:eb:53:bd:e9:e2:ec:20:fe:8a:2e:da:f9:f8:e5:b0.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'master.hadoop,10.9.148.193' (RSA) to the list of known hosts.
[email protected]'s password:
[root@slave3 .ssh]# ssh master.hadoop
Last login: Thu Aug 24 13:29:09 2017 from 10.9.148.17
[root@master ~]# ssh slave3.hadoop
Last login: Wed Aug 23 13:35:15 2017 from 10.9.148.17
[root@slave3 ~]#
7. 创建hadoop系统用户并设置密码（Master与Slave相同）（这里直接用root账户安装配置实用）
846 useradd hadoop
847 passwd hadoop
8. 安装hadoop
952 vi slaves
10.9.148.190
10.9.148.191
10.9.148.192
953 vi core-site.xml

hadoop.tmp.dir
/home/softs/hadoop/tmp
A base for other temporary directories.

fs.default.name
hdfs://master.hadoop:9000

954 vi hdfs-site.xml

dfs.namenode.secondary.http-address
master.hadoop:50090

dfs.namenode.name.dir
file:/home/softs/hadoop/tmp/dfs/name

dfs.datanode.data.dir
file:/home/softs/hadoop/tmp/dfs/data

dfs.replication
3

dfs.permissions
false

957 mv mapred-site.xml.template mapred-site.xml
958 cp mapred-site.xml mapred-site.xml.template
959 ls
960 vi mapred-site.xml

mapreduce.framework.name
yarn

mapreduce.jobhistory.address
master.hadoop:10020

mapreduce.jobhistory.webapp.address
master.hadoop:19888

961 vi yarn-site.xml

yarn.resourcemanager.hostname
master.hadoop

yarn.nodemanager.aux-services
mapreduce_shuffle

962 vi masters
master.hadoop
984 tar -zcf hadoop.master.tar.gz ./hadoop
//把master.hadoop配置好的hadoop压缩，分别复制到各slave里
985 scp ./hadoop.master.tar.gz slave1.hadoop:/home/softs/
986 scp ./hadoop.master.tar.gz slave2.hadoop:/home/softs/
987 scp ./hadoop.master.tar.gz slave3.hadoop:/home/softs/

62 tar -zxf hadoop.master.tar.gz
63 chown -R hadoop ./hadoop
<启动时，因为找不到JAVA_HOME，但是明显/etc/profile里面配置了的，所以只好手动添加java的路径到/home/softs/hadoop/etc/hadoop/hadoop-env.sh 里面、>
68 vi /home/softs/hadoop/etc/hadoop/hadoop-env.sh

<注意关闭防火墙>：否则，文件传输过程中会报错
[root@master hadoop]# hadoop fs -copyFromLocal /home/201706/* /I2
17/08/24 15:46:49 INFO hdfs.DataStreamer: Exception in createBlockOutputStream
java.net.NoRouteToHostException: 没有到主机的路由
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.hdfs.DataStreamer.createSocketForPipeline(DataStreamer.java:226)
at org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1591)
at org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1547)
at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:658)
17/08/24 15:46:49 WARN hdfs.DataStreamer: Abandoning BP-748028963-10.9.148.193-1503559871024:blk_1073741833_1009
17/08/24 15:46:49 WARN hdfs.DataStreamer: Excluding datanode DatanodeInfoWithStorage[10.9.148.190:50010,DS-e1300bd0-fb4f-488e-9c79-42b47c1ef714,DISK]
17/08/24 15:46:49 INFO hdfs.DataStreamer: Exception in createBlockOutputStream
java.net.NoRouteToHostException: 没有到主机的路由
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.hdfs.DataStreamer.createSocketForPipeline(DataStreamer.java:226)
at org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1591)
at org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1547)
at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:658)
17/08/24 15:46:49 WARN hdfs.DataStreamer: Abandoning BP-748028963-10.9.148.193-1503559871024:blk_1073741834_1010
17/08/24 15:46:49 WARN hdfs.DataStreamer: Excluding datanode DatanodeInfoWithStorage[10.9.148.191:50010,DS-969a3d41-8e2e-475d-943f-f2a02841d174,DISK]
^C17/08/24 15:46:52 INFO fs.FileSystem: Ignoring failure to deleteOnExit for path hdfs://master.hadoop:9000/I2/20170601._COPYING_
copyFromLocal: java.nio.channels.ClosedChannelException
at org.apache.hadoop.hdfs.DataStreamer$LastExceptionInStreamer.throwException4Close(DataStreamer.java:291)
at org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:147)
at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:105)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58)
at java.io.DataOutputStream.write(DataOutputStream.java:107)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:91)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:63)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:124)
at org.apache.hadoop.fs.shell.CommandWithDestination$TargetFileSystem.writeStreamToFile(CommandWithDestination.java:485)
at org.apache.hadoop.fs.shell.CommandWithDestination.copyStreamToTarget(CommandWithDestination.java:407)
at org.apache.hadoop.fs.shell.CommandWithDestination.copyFileToTarget(CommandWithDestination.java:342)
at org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:277)
at org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:262)
at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317)
at org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:289)
at org.apache.hadoop.fs.shell.CommandWithDestination.processPathArgument(CommandWithDestination.java:257)
at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271)
at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255)
at org.apache.hadoop.fs.shell.CommandWithDestination.processArguments(CommandWithDestination.java:228)
at org.apache.hadoop.fs.shell.CopyCommands$Put.processArguments(CopyCommands.java:286)
at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:119)
at org.apache.hadoop.fs.shell.Command.run(Command.java:165)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:315)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:378)

copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed

9. 格式化namenode
[root@master hadoop]# hadoop namenode –format
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.

17/08/24 15:28:29 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: user = root
STARTUP_MSG: host = master.hadoop/10.9.148.193
STARTUP_MSG: args = [–format]
STARTUP_MSG: version = 2.8.1
STARTUP_MSG: classpath = /home/softs/hadoop/etc/hadoop:/home/softs/hadoop/share/hadoop/common/lib/jsp-api-2.1.jar:/home/softs/hadoop/share/hadoop/common/lib/jersey-core-1.9.jar:/home/softs/hadoop/share/hadoop/common/lib/jersey-json-1.9.jar:/home/softs/hadoop/share/hadoop/common/lib/jettison-1.1.jar:/home/softs/hadoop/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/softs/hadoop/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/softs/hadoop/share/hadoop/common/lib/stax-api-1.0-2.jar:/home/softs/hadoop/share/hadoop/common/lib/activation-1.1.jar:/home/softs/hadoop/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/home/softs/hadoop/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/home/softs/hadoop/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/home/softs/hadoop/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/home/softs/hadoop/share/hadoop/common/lib/jersey-server-1.9.jar:/home/softs/hadoop/share/hadoop/common/lib/asm-3.2.jar:/home/softs/hadoop/share/hadoop/common/lib/log4j-1.2.17.jar:/home/softs/hadoop/share/hadoop/common/lib/jets3t-0.9.0.jar:/home/softs/hadoop/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-lang-2.6.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-configuration-1.6.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-digester-1.8.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/softs/hadoop/share/hadoop/common/lib/slf4j-api-1.7.10.jar:/home/softs/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar:/home/softs/hadoop/share/hadoop/common/lib/avro-1.7.4.jar:/home/softs/hadoop/share/hadoop/common/lib/paranamer-2.3.jar:/home/softs/hadoop/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-compress-1.4.1.jar:/home/softs/hadoop/share/hadoop/common/lib/xz-1.0.jar:/home/softs/hadoop/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/home/softs/hadoop/share/hadoop/common/lib/gson-2.2.4.jar:/home/softs/hadoop/share/hadoop/common/lib/hadoop-auth-2.8.1.jar:/home/softs/hadoop/share/hadoop/common/lib/nimbus-jose-jwt-3.9.jar:/home/softs/hadoop/share/hadoop/common/lib/jcip-annotations-1.0.jar:/home/softs/hadoop/share/hadoop/common/lib/json-smart-1.1.1.jar:/home/softs/hadoop/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/softs/hadoop/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/home/softs/hadoop/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/home/softs/hadoop/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/home/softs/hadoop/share/hadoop/common/lib/zookeeper-3.4.6.jar:/home/softs/hadoop/share/hadoop/common/lib/netty-3.6.2.Final.jar:/home/softs/hadoop/share/hadoop/common/lib/curator-framework-2.7.1.jar:/home/softs/hadoop/share/hadoop/common/lib/curator-client-2.7.1.jar:/home/softs/hadoop/share/hadoop/common/lib/jsch-0.1.51.jar:/home/softs/hadoop/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/home/softs/hadoop/share/hadoop/common/lib/htrace-core4-4.0.1-incubating.jar:/home/softs/hadoop/share/hadoop/common/lib/junit-4.11.jar:/home/softs/hadoop/share/hadoop/common/lib/hamcrest-core-1.3.jar:/home/softs/hadoop/share/hadoop/common/lib/mockito-all-1.8.5.jar:/home/softs/hadoop/share/hadoop/common/lib/hadoop-annotations-2.8.1.jar:/home/softs/hadoop/share/hadoop/common/lib/guava-11.0.2.jar:/home/softs/hadoop/share/hadoop/common/lib/jsr305-3.0.0.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-cli-1.2.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-math3-3.1.1.jar:/home/softs/hadoop/share/hadoop/common/lib/xmlenc-0.52.jar:/home/softs/hadoop/share/hadoop/common/lib/httpclient-4.5.2.jar:/home/softs/hadoop/share/hadoop/common/lib/httpcore-4.4.4.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-logging-1.1.3.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-codec-1.4.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-io-2.4.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-net-3.1.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-collections-3.2.2.jar:/home/softs/hadoop/share/hadoop/common/lib/servlet-api-2.5.jar:/home/softs/hadoop/share/hadoop/common/lib/jetty-6.1.26.jar:/home/softs/hadoop/share/hadoop/common/lib/jetty-util-6.1.26.jar:/home/softs/hadoop/share/hadoop/common/lib/jetty-sslengine-6.1.26.jar:/home/softs/hadoop/share/hadoop/common/hadoop-common-2.8.1.jar:/home/softs/hadoop/share/hadoop/common/hadoop-common-2.8.1-tests.jar:/home/softs/hadoop/share/hadoop/common/hadoop-nfs-2.8.1.jar:/home/softs/hadoop/share/hadoop/hdfs:/home/softs/hadoop/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/commons-io-2.4.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/guava-11.0.2.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/jsr305-3.0.0.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/asm-3.2.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/htrace-core4-4.0.1-incubating.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/hadoop-hdfs-client-2.8.1.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/okhttp-2.4.0.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/okio-1.4.0.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/home/softs/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.8.1.jar:/home/softs/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.8.1-tests.jar:/home/softs/hadoop/share/hadoop/hdfs/hadoop-hdfs-client-2.8.1.jar:/home/softs/hadoop/share/hadoop/hdfs/hadoop-hdfs-client-2.8.1-tests.jar:/home/softs/hadoop/share/hadoop/hdfs/hadoop-hdfs-native-client-2.8.1.jar:/home/softs/hadoop/share/hadoop/hdfs/hadoop-hdfs-native-client-2.8.1-tests.jar:/home/softs/hadoop/share/hadoop/hdfs/hadoop-hdfs-nfs-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/home/softs/hadoop/share/hadoop/yarn/lib/commons-collections-3.2.2.jar:/home/softs/hadoop/share/hadoop/yarn/lib/fst-2.24.jar:/home/softs/hadoop/share/hadoop/yarn/lib/javassist-3.18.1-GA.jar:/home/softs/hadoop/share/hadoop/yarn/lib/objenesis-2.1.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jetty-6.1.26.jar:/home/softs/hadoop/share/hadoop/yarn/lib/curator-client-2.7.1.jar:/home/softs/hadoop/share/hadoop/yarn/lib/curator-test-2.7.1.jar:/home/softs/hadoop/share/hadoop/yarn/lib/commons-math-2.2.jar:/home/softs/hadoop/share/hadoop/yarn/lib/zookeeper-3.4.6-tests.jar:/home/softs/hadoop/share/hadoop/yarn/lib/commons-lang-2.6.jar:/home/softs/hadoop/share/hadoop/yarn/lib/guava-11.0.2.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jsr305-3.0.0.jar:/home/softs/hadoop/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/home/softs/hadoop/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/home/softs/hadoop/share/hadoop/yarn/lib/commons-cli-1.2.jar:/home/softs/hadoop/share/hadoop/yarn/lib/log4j-1.2.17.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/home/softs/hadoop/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/home/softs/hadoop/share/hadoop/yarn/lib/activation-1.1.jar:/home/softs/hadoop/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/home/softs/hadoop/share/hadoop/yarn/lib/xz-1.0.jar:/home/softs/hadoop/share/hadoop/yarn/lib/servlet-api-2.5.jar:/home/softs/hadoop/share/hadoop/yarn/lib/commons-codec-1.4.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jersey-core-1.9.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jersey-client-1.9.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/home/softs/hadoop/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/home/softs/hadoop/share/hadoop/yarn/lib/guice-3.0.jar:/home/softs/hadoop/share/hadoop/yarn/lib/javax.inject-1.jar:/home/softs/hadoop/share/hadoop/yarn/lib/aopalliance-1.0.jar:/home/softs/hadoop/share/hadoop/yarn/lib/commons-io-2.4.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jersey-server-1.9.jar:/home/softs/hadoop/share/hadoop/yarn/lib/asm-3.2.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jersey-json-1.9.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jettison-1.1.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/home/softs/hadoop/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/home/softs/hadoop/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-api-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-server-common-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-server-tests-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-client-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-server-timeline-pluginstorage-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-registry-2.8.1.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/xz-1.0.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/hadoop-annotations-2.8.1.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/asm-3.2.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/guice-3.0.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/javax.inject-1.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/junit-4.11.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/home/softs/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.8.1.jar:/home/softs/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.8.1.jar:/home/softs/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.8.1.jar:/home/softs/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.8.1.jar:/home/softs/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.8.1.jar:/home/softs/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.8.1.jar:/home/softs/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.8.1.jar:/home/softs/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.1.jar:/home/softs/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.8.1-tests.jar:/home/softs/hadoop/contrib/capacity-scheduler/*.jar:/home/softs/hadoop/contrib/capacity-scheduler/*.jar
STARTUP_MSG: build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r 20fe5304904fc2f5a18053c389e43cd26f7a70fe; compiled by 'vinodkv' on 2017-06-02T06:14Z
STARTUP_MSG: java = 1.7.0_45
************************************************************/
17/08/24 15:28:29 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
17/08/24 15:28:29 INFO namenode.NameNode: createNameNode [–format]
Usage: hdfs namenode [-backup] |
[-checkpoint] |
[-format [-clusterid cid ] [-force] [-nonInteractive] ] |
[-upgrade [-clusterid cid] [-renameReserved] ] |
[-upgradeOnly [-clusterid cid] [-renameReserved] ] |
[-rollback] |
[-rollingUpgrade ] |
[-finalize] |
[-importCheckpoint] |
[-initializeSharedEdits] |
[-bootstrapStandby [-force] [-nonInteractive] [-skipSharedEditsCheck] ] |
[-recover [ -force] ] |
[-metadataVersion ]

17/08/24 15:28:30 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at master.hadoop/10.9.148.193
************************************************************/

========================================================================
[root@master hadoop]# hdfs namenode -format
17/08/24 15:31:09 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: user = root
STARTUP_MSG: host = master.hadoop/10.9.148.193
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 2.8.1
STARTUP_MSG: classpath = /home/softs/hadoop/etc/hadoop:/home/softs/hadoop/share/hadoop/common/lib/jsp-api-2.1.jar:/home/softs/hadoop/share/hadoop/common/lib/jersey-core-1.9.jar:/home/softs/hadoop/share/hadoop/common/lib/jersey-json-1.9.jar:/home/softs/hadoop/share/hadoop/common/lib/jettison-1.1.jar:/home/softs/hadoop/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/softs/hadoop/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/softs/hadoop/share/hadoop/common/lib/stax-api-1.0-2.jar:/home/softs/hadoop/share/hadoop/common/lib/activation-1.1.jar:/home/softs/hadoop/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/home/softs/hadoop/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/home/softs/hadoop/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/home/softs/hadoop/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/home/softs/hadoop/share/hadoop/common/lib/jersey-server-1.9.jar:/home/softs/hadoop/share/hadoop/common/lib/asm-3.2.jar:/home/softs/hadoop/share/hadoop/common/lib/log4j-1.2.17.jar:/home/softs/hadoop/share/hadoop/common/lib/jets3t-0.9.0.jar:/home/softs/hadoop/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-lang-2.6.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-configuration-1.6.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-digester-1.8.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/softs/hadoop/share/hadoop/common/lib/slf4j-api-1.7.10.jar:/home/softs/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar:/home/softs/hadoop/share/hadoop/common/lib/avro-1.7.4.jar:/home/softs/hadoop/share/hadoop/common/lib/paranamer-2.3.jar:/home/softs/hadoop/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-compress-1.4.1.jar:/home/softs/hadoop/share/hadoop/common/lib/xz-1.0.jar:/home/softs/hadoop/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/home/softs/hadoop/share/hadoop/common/lib/gson-2.2.4.jar:/home/softs/hadoop/share/hadoop/common/lib/hadoop-auth-2.8.1.jar:/home/softs/hadoop/share/hadoop/common/lib/nimbus-jose-jwt-3.9.jar:/home/softs/hadoop/share/hadoop/common/lib/jcip-annotations-1.0.jar:/home/softs/hadoop/share/hadoop/common/lib/json-smart-1.1.1.jar:/home/softs/hadoop/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/softs/hadoop/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/home/softs/hadoop/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/home/softs/hadoop/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/home/softs/hadoop/share/hadoop/common/lib/zookeeper-3.4.6.jar:/home/softs/hadoop/share/hadoop/common/lib/netty-3.6.2.Final.jar:/home/softs/hadoop/share/hadoop/common/lib/curator-framework-2.7.1.jar:/home/softs/hadoop/share/hadoop/common/lib/curator-client-2.7.1.jar:/home/softs/hadoop/share/hadoop/common/lib/jsch-0.1.51.jar:/home/softs/hadoop/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/home/softs/hadoop/share/hadoop/common/lib/htrace-core4-4.0.1-incubating.jar:/home/softs/hadoop/share/hadoop/common/lib/junit-4.11.jar:/home/softs/hadoop/share/hadoop/common/lib/hamcrest-core-1.3.jar:/home/softs/hadoop/share/hadoop/common/lib/mockito-all-1.8.5.jar:/home/softs/hadoop/share/hadoop/common/lib/hadoop-annotations-2.8.1.jar:/home/softs/hadoop/share/hadoop/common/lib/guava-11.0.2.jar:/home/softs/hadoop/share/hadoop/common/lib/jsr305-3.0.0.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-cli-1.2.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-math3-3.1.1.jar:/home/softs/hadoop/share/hadoop/common/lib/xmlenc-0.52.jar:/home/softs/hadoop/share/hadoop/common/lib/httpclient-4.5.2.jar:/home/softs/hadoop/share/hadoop/common/lib/httpcore-4.4.4.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-logging-1.1.3.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-codec-1.4.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-io-2.4.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-net-3.1.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-collections-3.2.2.jar:/home/softs/hadoop/share/hadoop/common/lib/servlet-api-2.5.jar:/home/softs/hadoop/share/hadoop/common/lib/jetty-6.1.26.jar:/home/softs/hadoop/share/hadoop/common/lib/jetty-util-6.1.26.jar:/home/softs/hadoop/share/hadoop/common/lib/jetty-sslengine-6.1.26.jar:/home/softs/hadoop/share/hadoop/common/hadoop-common-2.8.1.jar:/home/softs/hadoop/share/hadoop/common/hadoop-common-2.8.1-tests.jar:/home/softs/hadoop/share/hadoop/common/hadoop-nfs-2.8.1.jar:/home/softs/hadoop/share/hadoop/hdfs:/home/softs/hadoop/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/commons-io-2.4.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/guava-11.0.2.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/jsr305-3.0.0.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/asm-3.2.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/htrace-core4-4.0.1-incubating.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/hadoop-hdfs-client-2.8.1.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/okhttp-2.4.0.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/okio-1.4.0.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/home/softs/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.8.1.jar:/home/softs/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.8.1-tests.jar:/home/softs/hadoop/share/hadoop/hdfs/hadoop-hdfs-client-2.8.1.jar:/home/softs/hadoop/share/hadoop/hdfs/hadoop-hdfs-client-2.8.1-tests.jar:/home/softs/hadoop/share/hadoop/hdfs/hadoop-hdfs-native-client-2.8.1.jar:/home/softs/hadoop/share/hadoop/hdfs/hadoop-hdfs-native-client-2.8.1-tests.jar:/home/softs/hadoop/share/hadoop/hdfs/hadoop-hdfs-nfs-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/home/softs/hadoop/share/hadoop/yarn/lib/commons-collections-3.2.2.jar:/home/softs/hadoop/share/hadoop/yarn/lib/fst-2.24.jar:/home/softs/hadoop/share/hadoop/yarn/lib/javassist-3.18.1-GA.jar:/home/softs/hadoop/share/hadoop/yarn/lib/objenesis-2.1.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jetty-6.1.26.jar:/home/softs/hadoop/share/hadoop/yarn/lib/curator-client-2.7.1.jar:/home/softs/hadoop/share/hadoop/yarn/lib/curator-test-2.7.1.jar:/home/softs/hadoop/share/hadoop/yarn/lib/commons-math-2.2.jar:/home/softs/hadoop/share/hadoop/yarn/lib/zookeeper-3.4.6-tests.jar:/home/softs/hadoop/share/hadoop/yarn/lib/commons-lang-2.6.jar:/home/softs/hadoop/share/hadoop/yarn/lib/guava-11.0.2.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jsr305-3.0.0.jar:/home/softs/hadoop/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/home/softs/hadoop/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/home/softs/hadoop/share/hadoop/yarn/lib/commons-cli-1.2.jar:/home/softs/hadoop/share/hadoop/yarn/lib/log4j-1.2.17.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/home/softs/hadoop/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/home/softs/hadoop/share/hadoop/yarn/lib/activation-1.1.jar:/home/softs/hadoop/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/home/softs/hadoop/share/hadoop/yarn/lib/xz-1.0.jar:/home/softs/hadoop/share/hadoop/yarn/lib/servlet-api-2.5.jar:/home/softs/hadoop/share/hadoop/yarn/lib/commons-codec-1.4.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jersey-core-1.9.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jersey-client-1.9.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/home/softs/hadoop/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/home/softs/hadoop/share/hadoop/yarn/lib/guice-3.0.jar:/home/softs/hadoop/share/hadoop/yarn/lib/javax.inject-1.jar:/home/softs/hadoop/share/hadoop/yarn/lib/aopalliance-1.0.jar:/home/softs/hadoop/share/hadoop/yarn/lib/commons-io-2.4.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jersey-server-1.9.jar:/home/softs/hadoop/share/hadoop/yarn/lib/asm-3.2.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jersey-json-1.9.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jettison-1.1.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/home/softs/hadoop/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/home/softs/hadoop/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-api-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-server-common-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-server-tests-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-client-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-server-timeline-pluginstorage-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-registry-2.8.1.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/xz-1.0.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/hadoop-annotations-2.8.1.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/asm-3.2.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/guice-3.0.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/javax.inject-1.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/junit-4.11.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/home/softs/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.8.1.jar:/home/softs/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.8.1.jar:/home/softs/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.8.1.jar:/home/softs/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.8.1.jar:/home/softs/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.8.1.jar:/home/softs/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.8.1.jar:/home/softs/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.8.1.jar:/home/softs/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.1.jar:/home/softs/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.8.1-tests.jar:/home/softs/hadoop/contrib/capacity-scheduler/*.jar
STARTUP_MSG: build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r 20fe5304904fc2f5a18053c389e43cd26f7a70fe; compiled by 'vinodkv' on 2017-06-02T06:14Z
STARTUP_MSG: java = 1.7.0_45
************************************************************/
17/08/24 15:31:09 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
17/08/24 15:31:09 INFO namenode.NameNode: createNameNode [-format]
Formatting using clusterid: CID-4b31f9f8-c01d-45e3-b334-91bd86aafefa
17/08/24 15:31:10 INFO namenode.FSEditLog: Edit logging is async:false
17/08/24 15:31:10 INFO namenode.FSNamesystem: KeyProvider: null
17/08/24 15:31:10 INFO namenode.FSNamesystem: fsLock is fair: true
17/08/24 15:31:10 INFO namenode.FSNamesystem: Detailed lock hold time metrics enabled: false
17/08/24 15:31:10 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
17/08/24 15:31:10 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
17/08/24 15:31:10 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
17/08/24 15:31:10 INFO blockmanagement.BlockManager: The block deletion will start around 2017 八月 24 15:31:10
17/08/24 15:31:10 INFO util.GSet: Computing capacity for map BlocksMap
17/08/24 15:31:10 INFO util.GSet: VM type = 64-bit
17/08/24 15:31:10 INFO util.GSet: 2.0% max memory 889 MB = 17.8 MB
17/08/24 15:31:10 INFO util.GSet: capacity = 2^21 = 2097152 entries
17/08/24 15:31:10 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
17/08/24 15:31:10 INFO blockmanagement.BlockManager: defaultReplication = 3
17/08/24 15:31:10 INFO blockmanagement.BlockManager: maxReplication = 512
17/08/24 15:31:10 INFO blockmanagement.BlockManager: minReplication = 1
17/08/24 15:31:10 INFO blockmanagement.BlockManager: maxReplicationStreams = 2
17/08/24 15:31:10 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
17/08/24 15:31:10 INFO blockmanagement.BlockManager: encryptDataTransfer = false
17/08/24 15:31:10 INFO blockmanagement.BlockManager: maxNumBlocksToLog = 1000
17/08/24 15:31:10 INFO namenode.FSNamesystem: fsOwner = root (auth:SIMPLE)
17/08/24 15:31:10 INFO namenode.FSNamesystem: supergroup = supergroup
17/08/24 15:31:10 INFO namenode.FSNamesystem: isPermissionEnabled = false
17/08/24 15:31:10 INFO namenode.FSNamesystem: HA Enabled: false
17/08/24 15:31:10 INFO namenode.FSNamesystem: Append Enabled: true
17/08/24 15:31:10 INFO util.GSet: Computing capacity for map INodeMap
17/08/24 15:31:10 INFO util.GSet: VM type = 64-bit
17/08/24 15:31:10 INFO util.GSet: 1.0% max memory 889 MB = 8.9 MB
17/08/24 15:31:10 INFO util.GSet: capacity = 2^20 = 1048576 entries
17/08/24 15:31:10 INFO namenode.FSDirectory: ACLs enabled? false
17/08/24 15:31:10 INFO namenode.FSDirectory: XAttrs enabled? true
17/08/24 15:31:10 INFO namenode.NameNode: Caching file names occurring more than 10 times
17/08/24 15:31:10 INFO util.GSet: Computing capacity for map cachedBlocks
17/08/24 15:31:10 INFO util.GSet: VM type = 64-bit
17/08/24 15:31:10 INFO util.GSet: 0.25% max memory 889 MB = 2.2 MB
17/08/24 15:31:10 INFO util.GSet: capacity = 2^18 = 262144 entries
17/08/24 15:31:10 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
17/08/24 15:31:10 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
17/08/24 15:31:10 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
17/08/24 15:31:10 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10
17/08/24 15:31:10 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10
17/08/24 15:31:10 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
17/08/24 15:31:10 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
17/08/24 15:31:10 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
17/08/24 15:31:10 INFO util.GSet: Computing capacity for map NameNodeRetryCache
17/08/24 15:31:10 INFO util.GSet: VM type = 64-bit
17/08/24 15:31:10 INFO util.GSet: 0.029999999329447746% max memory 889 MB = 273.1 KB
17/08/24 15:31:10 INFO util.GSet: capacity = 2^15 = 32768 entries
17/08/24 15:31:11 INFO namenode.FSImage: Allocated new BlockPoolId: BP-748028963-10.9.148.193-1503559871024
17/08/24 15:31:11 INFO common.Storage: Storage directory /home/softs/hadoop/tmp/dfs/name has been successfully formatted.
17/08/24 15:31:11 INFO namenode.FSImageFormatProtobuf: Saving image file /home/softs/hadoop/tmp/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
17/08/24 15:31:11 INFO namenode.FSImageFormatProtobuf: Image file /home/softs/hadoop/tmp/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 321 bytes saved in 0 seconds.
17/08/24 15:31:11 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
17/08/24 15:31:11 INFO util.ExitUtil: Exiting with status 0
17/08/24 15:31:11 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at master.hadoop/10.9.148.193
************************************************************/
[root@master hadoop]#

10. 启动和验证：
[root@master hadoop]# start-dfs.sh
Starting namenodes on [master.hadoop]
master.hadoop: starting namenode, logging to /home/softs/hadoop/logs/hadoop-root-namenode-master.hadoop.out
10.9.148.191: starting datanode, logging to /home/softs/hadoop/logs/hadoop-root-datanode-slave2.hadoop.out
10.9.148.190: starting datanode, logging to /home/softs/hadoop/logs/hadoop-root-datanode-slave1.hadoop.out
10.9.148.192: starting datanode, logging to /home/softs/hadoop/logs/hadoop-root-datanode-slave3.hadoop.out
Starting secondary namenodes [master.hadoop]
master.hadoop: starting secondarynamenode, logging to /home/softs/hadoop/logs/hadoop-root-secondarynamenode-master.hadoop.out
[root@master hadoop]# start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /home/softs/hadoop/logs/yarn-root-resourcemanager-master.hadoop.out
10.9.148.192: starting nodemanager, logging to /home/softs/hadoop/logs/yarn-root-nodemanager-slave3.hadoop.out
10.9.148.191: starting nodemanager, logging to /home/softs/hadoop/logs/yarn-root-nodemanager-slave2.hadoop.out

10.9.148.190: starting nodemanager, logging to /home/softs/hadoop/logs/yarn-root-nodemanager-slave1.hadoop.out

在这里，我直接是start-all.sh启动namenode和yarn和dfs的，再加下面的jobhistoryserver

[root@master hadoop]# mr-jobhistory-daemon.sh start historyserver
starting historyserver, logging to /home/softs/hadoop/logs/mapred-root-historyserver-master.hadoop.out
[root@master hadoop]# jps
10342 NameNode
10665 SecondaryNameNode
10921 ResourceManager
11259 Jps
11213 JobHistoryServer
[root@master hadoop]# jps
10342 NameNode
10665 SecondaryNameNode
10921 ResourceManager
11213 JobHistoryServer
11279 Jps
[root@master hadoop]# hdfs dfsadmin -report
Configured Capacity: 458424913920 (426.94 GB)
Present Capacity: 428918013952 (399.46 GB)
DFS Remaining: 428917919744 (399.46 GB)
DFS Used: 94208 (92 KB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0
Pending deletion blocks: 0

-------------------------------------------------
Live datanodes (3):

Name: 10.9.148.190:50010 (slave1.hadoop)
Hostname: slave1.hadoop
Decommission Status : Normal
Configured Capacity: 152682475520 (142.20 GB)
DFS Used: 28672 (28 KB)
Non DFS Used: 8190533632 (7.63 GB)
DFS Remaining: 142939611136 (133.12 GB)
DFS Used%: 0.00%
DFS Remaining%: 93.62%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu Aug 24 15:55:18 CST 2017

Name: 10.9.148.191:50010 (slave2.hadoop)
Hostname: slave2.hadoop
Decommission Status : Normal
Configured Capacity: 152617463808 (142.14 GB)
DFS Used: 28672 (28 KB)
Non DFS Used: 8327135232 (7.76 GB)
DFS Remaining: 142738649088 (132.94 GB)
DFS Used%: 0.00%
DFS Remaining%: 93.53%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu Aug 24 15:55:18 CST 2017

Name: 10.9.148.192:50010 (slave3.hadoop)
Hostname: slave3.hadoop
Decommission Status : Normal
Configured Capacity: 153124974592 (142.61 GB)
DFS Used: 36864 (36 KB)
Non DFS Used: 8328491008 (7.76 GB)
DFS Remaining: 143239659520 (133.40 GB)
DFS Used%: 0.00%
DFS Remaining%: 93.54%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu Aug 24 15:55:18 CST 2017

[root@master hadoop]#

11. 使用
放入HDFS文件
[root@master hadoop]# hadoop fs -copyFromLocal /home/201706/* /I2
查看HDFS放入的文件
[root@master hadoop]# hadoop fs -ls /I2
Found 30 items
-rw-r--r-- 3 root supergroup 275137778 2017-08-24 15:57 /I2/20170601
-rw-r--r-- 3 root supergroup 287210022 2017-08-24 15:57 /I2/20170602
-rw-r--r-- 3 root supergroup 326314540 2017-08-24 15:58 /I2/20170603
-rw-r--r-- 3 root supergroup 327799830 2017-08-24 15:58 /I2/20170604
-rw-r--r-- 3 root supergroup 308526744 2017-08-24 15:59 /I2/20170605
-rw-r--r-- 3 root supergroup 349079308 2017-08-24 15:59 /I2/20170606
-rw-r--r-- 3 root supergroup 417899378 2017-08-24 16:00 /I2/20170607
-rw-r--r-- 3 root supergroup 442854968 2017-08-24 16:01 /I2/20170608
-rw-r--r-- 3 root supergroup 455375694 2017-08-24 16:01 /I2/20170609
-rw-r--r-- 3 root supergroup 486294396 2017-08-24 16:02 /I2/20170610
-rw-r--r-- 3 root supergroup 429505122 2017-08-24 16:03 /I2/20170611
-rw-r--r-- 3 root supergroup 427015202 2017-08-24 16:03 /I2/20170612
-rw-r--r-- 3 root supergroup 469780264 2017-08-24 16:04 /I2/20170613
-rw-r--r-- 3 root supergroup 510235430 2017-08-24 16:05 /I2/20170614
-rw-r--r-- 3 root supergroup 497186788 2017-08-24 16:06 /I2/20170615
-rw-r--r-- 3 root supergroup 558273348 2017-08-24 16:07 /I2/20170616
-rw-r--r-- 3 root supergroup 585348962 2017-08-24 16:07 /I2/20170617
-rw-r--r-- 3 root supergroup 573453072 2017-08-24 16:08 /I2/20170618
-rw-r--r-- 3 root supergroup 571823300 2017-08-24 16:09 /I2/20170619
-rw-r--r-- 3 root supergroup 686884860 2017-08-24 16:10 /I2/20170620
-rw-r--r-- 3 root supergroup 615866930 2017-08-24 16:11 /I2/20170621
-rw-r--r-- 3 root supergroup 738148784 2017-08-24 16:12 /I2/20170622
-rw-r--r-- 3 root supergroup 826302986 2017-08-24 16:14 /I2/20170623
-rw-r--r-- 3 root supergroup 841656010 2017-08-24 16:15 /I2/20170624
-rw-r--r-- 3 root supergroup 849475152 2017-08-24 16:16 /I2/20170625
-rw-r--r-- 3 root supergroup 860776370 2017-08-24 16:17 /I2/20170626
-rw-r--r-- 3 root supergroup 905409004 2017-08-24 16:19 /I2/20170627
-rw-r--r-- 3 root supergroup 978153598 2017-08-24 16:20 /I2/20170628
-rw-r--r-- 3 root supergroup 992471718 2017-08-24 16:22 /I2/20170629
-rw-r--r-- 3 root supergroup 1078413818 2017-08-24 16:23 /I2/20170630
[root@master hadoop]#
运行jar包
[root@master 6mouth]# ls
hdfs-mapreduce-jar-with-dependencies.jar
[root@master 6mouth]# pwd
/home/DATA/6mouth
[root@master 6mouth]# hadoop jar hdfs-mapreduce-jar-with-dependencies.jar /I2/20170601,/I2/20170602,/I2/20170603,/I2/20170604,/I2/20170605,/I2/20170606,/I2/20170607,/I2/20170608,/I2/20170609,/I2/20170610,/I2/20170611,/I2/20170612,/I2/20170613,/I2/20170614,/I2/20170615,/I2/20170616,/I2/20170617,/I2/20170618,/I2/20170619,/I2/20170620,/I2/20170621,/I2/20170622,/I2/20170623,/I2/20170624,/I2/20170625,/I2/20170626,/I2/20170627,/I2/20170628,/I2/20170629,/I2/20170630 /201706/
17/08/24 16:29:33 INFO client.RMProxy: Connecting to ResourceManager at master.hadoop/10.9.148.193:8032
17/08/24 16:29:34 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
17/08/24 16:29:38 INFO input.FileInputFormat: Total input files to process : 30
17/08/24 16:29:39 INFO mapreduce.JobSubmitter: number of splits:149
17/08/24 16:29:39 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1503561276940_0001
17/08/24 16:29:39 INFO impl.YarnClientImpl: Submitted application application_1503561276940_0001
17/08/24 16:29:39 INFO mapreduce.Job: The url to track the job: http://master.hadoop:8088/proxy/application_1503561276940_0001/
17/08/24 16:29:39 INFO mapreduce.Job: Running job: job_1503561276940_0001
17/08/24 16:29:53 INFO mapreduce.Job: Job job_1503561276940_0001 running in uber mode : false
17/08/24 16:29:53 INFO mapreduce.Job: map 0% reduce 0%
17/08/24 16:39:45 INFO mapreduce.Job: map 3% reduce 0%
17/08/24 16:39:45 INFO mapreduce.Job: Task Id : attempt_1503561276940_0001_m_000004_0, Status : FAILED
Exception from container-launch.
Container id: container_1503561276940_0001_01_000006
Exit code: 65
Stack trace: ExitCodeException exitCode=65:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:972)
at org.apache.hadoop.util.Shell.run(Shell.java:869)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1170)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:236)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:305)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:84)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

Container exited with a non-zero exit code 65

17/08/24 16:39:50 INFO mapreduce.Job: map 5% reduce 0%
17/08/24 16:39:50 INFO mapreduce.Job: Task Id : attempt_1503561276940_0001_m_000015_0, Status : FAILED
Exception from container-launch.
Container id: container_1503561276940_0001_01_000017
Exit code: 65
Stack trace: ExitCodeException exitCode=65:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:972)
at org.apache.hadoop.util.Shell.run(Shell.java:869)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1170)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:236)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:305)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:84)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

Container exited with a non-zero exit code 65

17/08/24 16:46:27 INFO mapreduce.Job: map 7% reduce 0%
17/08/24 16:46:30 INFO mapreduce.Job: map 8% reduce 0%
17/08/24 16:46:34 INFO mapreduce.Job: map 9% reduce 0%
17/08/24 16:46:40 INFO mapreduce.Job: Task Id : attempt_1503561276940_0001_m_000008_0, Status : FAILED
AttemptID:attempt_1503561276940_0001_m_000008_0 Timed out after 600 secs
Exception from container-launch.
Container id: container_1503561276940_0001_01_000010
Exit code: 65
Stack trace: ExitCodeException exitCode=65:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:972)
at org.apache.hadoop.util.Shell.run(Shell.java:869)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1170)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:236)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:305)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:84)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

Container exited with a non-zero exit code 65
Sent signal OUTPUT_THREAD_DUMP (SIGQUIT) to pid 4146 as user root for container container_1503561276940_0001_01_000010, result=failed

17/08/24 16:46:42 INFO mapreduce.Job: Task Id : attempt_1503561276940_0001_m_000012_0, Status : FAILED
AttemptID:attempt_1503561276940_0001_m_000012_0 Timed out after 600 secs
Sent signal OUTPUT_THREAD_DUMP (SIGQUIT) to pid 4147 as user root for container container_1503561276940_0001_01_000014, result=success
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

17/08/24 16:46:42 INFO mapreduce.Job: Task Id : attempt_1503561276940_0001_m_000009_0, Status : FAILED
AttemptID:attempt_1503561276940_0001_m_000009_0 Timed out after 600 secs
Sent signal OUTPUT_THREAD_DUMP (SIGQUIT) to pid 4141 as user root for container container_1503561276940_0001_01_000011, result=success
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

17/08/24 16:46:45 INFO mapreduce.Job: Task Id : attempt_1503561276940_0001_m_000017_0, Status : FAILED
AttemptID:attempt_1503561276940_0001_m_000017_0 Timed out after 600 secs
Exception from container-launch.
Container id: container_1503561276940_0001_01_000019
Exit code: 65
Stack trace: ExitCodeException exitCode=65:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:972)
at org.apache.hadoop.util.Shell.run(Shell.java:869)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1170)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:236)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:305)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:84)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

Container exited with a non-zero exit code 65
Sent signal OUTPUT_THREAD_DUMP (SIGQUIT) to pid 4023 as user root for container container_1503561276940_0001_01_000019, result=failed

17/08/24 16:46:45 INFO mapreduce.Job: Task Id : attempt_1503561276940_0001_m_000014_0, Status : FAILED
AttemptID:attempt_1503561276940_0001_m_000014_0 Timed out after 600 secs
Sent signal OUTPUT_THREAD_DUMP (SIGQUIT) to pid 4025 as user root for container container_1503561276940_0001_01_000016, result=success
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

17/08/24 16:46:46 INFO mapreduce.Job: Task Id : attempt_1503561276940_0001_m_000007_0, Status : FAILED
AttemptID:attempt_1503561276940_0001_m_000007_0 Timed out after 600 secs
Sent signal OUTPUT_THREAD_DUMP (SIGQUIT) to pid 4145 as user root for container container_1503561276940_0001_01_000009, result=failed

17/08/24 16:46:47 INFO mapreduce.Job: Task Id : attempt_1503561276940_0001_m_000006_0, Status : FAILED
AttemptID:attempt_1503561276940_0001_m_000006_0 Timed out after 600 secs
Exception from container-launch.
Container id: container_1503561276940_0001_01_000008
Exit code: 65
Stack trace: ExitCodeException exitCode=65:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:972)
at org.apache.hadoop.util.Shell.run(Shell.java:869)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1170)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:236)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:305)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:84)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

Container exited with a non-zero exit code 65
Sent signal OUTPUT_THREAD_DUMP (SIGQUIT) to pid 4149 as user root for container container_1503561276940_0001_01_000008, result=failed

17/08/24 16:46:47 INFO mapreduce.Job: Task Id : attempt_1503561276940_0001_m_000011_0, Status : FAILED
AttemptID:attempt_1503561276940_0001_m_000011_0 Timed out after 600 secs
Sent signal OUTPUT_THREAD_DUMP (SIGQUIT) to pid 4142 as user root for container container_1503561276940_0001_01_000013, result=success
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

17/08/24 16:46:47 INFO mapreduce.Job: Task Id : attempt_1503561276940_0001_m_000018_0, Status : FAILED
AttemptID:attempt_1503561276940_0001_m_000018_0 Timed out after 600 secs
Exception from container-launch.
Container id: container_1503561276940_0001_01_000020
Exit code: 65
Stack trace: ExitCodeException exitCode=65:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:972)
at org.apache.hadoop.util.Shell.run(Shell.java:869)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1170)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:236)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:305)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:84)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

Sent signal OUTPUT_THREAD_DUMP (SIGQUIT) to pid 4022 as user root for container container_1503561276940_0001_01_000020, result=success
Container exited with a non-zero exit code 65

17/08/24 16:46:48 INFO mapreduce.Job: Task Id : attempt_1503561276940_0001_m_000020_0, Status : FAILED
AttemptID:attempt_1503561276940_0001_m_000020_0 Timed out after 600 secs
Container killed by the ApplicationMaster.
Sent signal OUTPUT_THREAD_DUMP (SIGQUIT) to pid 4021 as user root for container container_1503561276940_0001_01_000022, result=success
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

17/08/24 16:46:48 INFO mapreduce.Job: Task Id : attempt_1503561276940_0001_m_000023_0, Status : FAILED
17/08/24 16:46:49 INFO mapreduce.Job: Task Id : attempt_1503561276940_0001_m_000016_0, Status : FAILED
AttemptID:attempt_1503561276940_0001_m_000016_0 Timed out after 600 secs
Sent signal OUTPUT_THREAD_DUMP (SIGQUIT) to pid 4024 as user root for container container_1503561276940_0001_01_000018, result=success
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

17/08/24 16:46:49 INFO mapreduce.Job: Task Id : attempt_1503561276940_0001_m_000010_0, Status : FAILED
AttemptID:attempt_1503561276940_0001_m_000010_0 Timed out after 600 secs
Exception from container-launch.
Container id: container_1503561276940_0001_01_000012
Exit code: 65
Stack trace: ExitCodeException exitCode=65:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:972)
at org.apache.hadoop.util.Shell.run(Shell.java:869)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1170)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:236)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:305)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:84)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

Container exited with a non-zero exit code 65

17/08/24 16:47:24 INFO mapreduce.Job: map 10% reduce 0%
17/08/24 16:47:38 INFO mapreduce.Job: map 10% reduce 3%
17/08/24 16:47:45 INFO mapreduce.Job: map 11% reduce 3%
17/08/24 16:48:30 INFO mapreduce.Job: map 11% reduce 4%
17/08/24 16:49:57 INFO mapreduce.Job: map 12% reduce 4%
17/08/24 16:51:31 INFO mapreduce.Job: map 13% reduce 4%
17/08/24 16:51:46 INFO mapreduce.Job: map 14% reduce 4%
17/08/24 16:51:49 INFO mapreduce.Job: map 16% reduce 4%
17/08/24 16:51:51 INFO mapreduce.Job: map 18% reduce 4%
17/08/24 16:51:55 INFO mapreduce.Job: map 19% reduce 4%
17/08/24 16:51:56 INFO mapreduce.Job: map 20% reduce 4%
17/08/24 16:52:38 INFO mapreduce.Job: map 21% reduce 4%
17/08/24 16:53:05 INFO mapreduce.Job: map 22% reduce 4%
17/08/24 16:55:22 INFO mapreduce.Job: map 23% reduce 4%
17/08/24 16:56:13 INFO mapreduce.Job: map 26% reduce 7%
17/08/24 16:56:21 INFO mapreduce.Job: map 25% reduce 7%
17/08/24 16:56:23 INFO mapreduce.Job: map 28% reduce 7%
17/08/24 16:56:26 INFO mapreduce.Job: map 31% reduce 7%
17/08/24 16:56:32 INFO mapreduce.Job: map 32% reduce 7%
17/08/24 16:56:39 INFO mapreduce.Job: map 33% reduce 7%
17/08/24 16:56:50 INFO mapreduce.Job: map 33% reduce 11%
17/08/24 17:01:09 INFO mapreduce.Job: map 36% reduce 11%
17/08/24 17:01:20 INFO mapreduce.Job: map 37% reduce 11%
17/08/24 17:01:46 INFO mapreduce.Job: map 38% reduce 11%
17/08/24 17:01:47 INFO mapreduce.Job: map 39% reduce 12%
17/08/24 17:01:55 INFO mapreduce.Job: map 40% reduce 12%
17/08/24 17:02:10 INFO mapreduce.Job: map 40% reduce 13%
17/08/24 17:02:24 INFO mapreduce.Job: map 41% reduce 13%
17/08/24 17:03:06 INFO mapreduce.Job: map 42% reduce 13%
17/08/24 17:05:31 INFO mapreduce.Job: map 44% reduce 13%
17/08/24 17:06:36 INFO mapreduce.Job: map 47% reduce 13%
17/08/24 17:06:52 INFO mapreduce.Job: map 48% reduce 13%
17/08/24 17:06:54 INFO mapreduce.Job: map 49% reduce 14%
17/08/24 17:07:00 INFO mapreduce.Job: map 50% reduce 14%
17/08/24 17:07:02 INFO mapreduce.Job: map 51% reduce 14%
17/08/24 17:07:04 INFO mapreduce.Job: map 52% reduce 14%
17/08/24 17:07:13 INFO mapreduce.Job: map 52% reduce 16%
17/08/24 17:07:19 INFO mapreduce.Job: map 52% reduce 17%
17/08/24 17:08:56 INFO mapreduce.Job: Task Id : attempt_1503561276940_0001_m_000092_0, Status : FAILED
17/08/24 17:08:57 INFO mapreduce.Job: Task Id : attempt_1503561276940_0001_m_000089_0, Status : FAILED
17/08/24 17:09:00 INFO mapreduce.Job: Task Id : attempt_1503561276940_0001_m_000091_0, Status : FAILED
17/08/24 17:09:00 INFO mapreduce.Job: Task Id : attempt_1503561276940_0001_m_000088_0, Status : FAILED
17/08/24 17:09:02 INFO mapreduce.Job: map 55% reduce 17%
17/08/24 17:09:47 INFO mapreduce.Job: map 56% reduce 17%
17/08/24 17:09:50 INFO mapreduce.Job: map 56% reduce 18%
17/08/24 17:10:58 INFO mapreduce.Job: map 57% reduce 18%
17/08/24 17:11:16 INFO mapreduce.Job: map 58% reduce 18%
17/08/24 17:12:13 INFO mapreduce.Job: map 59% reduce 18%
17/08/24 17:13:05 INFO mapreduce.Job: map 60% reduce 18%
17/08/24 17:13:12 INFO mapreduce.Job: map 61% reduce 18%
17/08/24 17:13:15 INFO mapreduce.Job: map 62% reduce 18%
17/08/24 17:13:17 INFO mapreduce.Job: map 63% reduce 18%
17/08/24 17:13:25 INFO mapreduce.Job: map 64% reduce 18%
17/08/24 17:13:31 INFO mapreduce.Job: map 64% reduce 20%
17/08/24 17:13:36 INFO mapreduce.Job: map 65% reduce 20%
17/08/24 17:13:45 INFO mapreduce.Job: map 66% reduce 20%
17/08/24 17:13:56 INFO mapreduce.Job: map 67% reduce 20%
17/08/24 17:14:15 INFO mapreduce.Job: map 68% reduce 20%
17/08/24 17:14:23 INFO mapreduce.Job: map 68% reduce 22%
17/08/24 17:16:38 INFO mapreduce.Job: map 69% reduce 22%
17/08/24 17:16:43 INFO mapreduce.Job: map 72% reduce 22%
17/08/24 17:16:48 INFO mapreduce.Job: map 73% reduce 22%
17/08/24 17:16:57 INFO mapreduce.Job: map 73% reduce 23%
17/08/24 17:17:03 INFO mapreduce.Job: map 73% reduce 24%
17/08/24 17:17:07 INFO mapreduce.Job: map 74% reduce 24%
17/08/24 17:17:28 INFO mapreduce.Job: map 75% reduce 24%
17/08/24 17:18:13 INFO mapreduce.Job: Task Id : attempt_1503561276940_0001_m_000122_0, Status : FAILED
17/08/24 17:18:13 INFO mapreduce.Job: Task Id : attempt_1503561276940_0001_m_000119_0, Status : FAILED
17/08/24 17:18:20 INFO mapreduce.Job: map 76% reduce 24%
17/08/24 17:18:22 INFO mapreduce.Job: map 77% reduce 24%
17/08/24 17:18:24 INFO mapreduce.Job: map 79% reduce 24%
17/08/24 17:18:26 INFO mapreduce.Job: map 79% reduce 25%
17/08/24 17:18:32 INFO mapreduce.Job: map 80% reduce 25%
17/08/24 17:18:42 INFO mapreduce.Job: map 81% reduce 25%
17/08/24 17:18:46 INFO mapreduce.Job: map 81% reduce 27%
17/08/24 17:18:54 INFO mapreduce.Job: map 82% reduce 27%
17/08/24 17:18:55 INFO mapreduce.Job: map 84% reduce 27%
17/08/24 17:19:01 INFO mapreduce.Job: map 85% reduce 27%
17/08/24 17:19:18 INFO mapreduce.Job: map 85% reduce 28%
17/08/24 17:21:21 INFO mapreduce.Job: Task Id : attempt_1503561276940_0001_m_000122_1, Status : FAILED
17/08/24 17:21:33 INFO mapreduce.Job: map 88% reduce 28%
17/08/24 17:21:36 INFO mapreduce.Job: map 89% reduce 28%
17/08/24 17:21:41 INFO mapreduce.Job: map 92% reduce 28%
17/08/24 17:21:51 INFO mapreduce.Job: map 92% reduce 29%
17/08/24 17:21:55 INFO mapreduce.Job: map 93% reduce 29%
17/08/24 17:22:01 INFO mapreduce.Job: map 93% reduce 30%
17/08/24 17:22:04 INFO mapreduce.Job: map 94% reduce 30%
17/08/24 17:22:10 INFO mapreduce.Job: map 96% reduce 30%
17/08/24 17:22:12 INFO mapreduce.Job: map 97% reduce 30%
17/08/24 17:22:15 INFO mapreduce.Job: map 97% reduce 32%
17/08/24 17:22:19 INFO mapreduce.Job: map 98% reduce 32%
17/08/24 17:22:20 INFO mapreduce.Job: map 99% reduce 32%
17/08/24 17:22:21 INFO mapreduce.Job: map 99% reduce 33%
17/08/24 17:22:28 INFO mapreduce.Job: map 100% reduce 33%
17/08/24 17:22:41 INFO mapreduce.Job: map 100% reduce 85%
17/08/24 17:22:44 INFO mapreduce.Job: map 100% reduce 100%
17/08/24 17:22:51 INFO mapreduce.Job: Job job_1503561276940_0001 completed successfully
17/08/24 17:22:53 INFO mapred.ClientServiceDelegate: Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
17/08/24 17:22:53 INFO mapred.ClientServiceDelegate: Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
17/08/24 17:22:53 INFO mapred.ClientServiceDelegate: Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
Exception in thread "main" java.io.IOException: java.io.IOException: Unknown Job job_1503561276940_0001
at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.verifyAndGetJob(HistoryClientService.java:218)
at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getCounters(HistoryClientService.java:232)
at org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getCounters(MRClientProtocolPBServiceImpl.java:159)
at org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:281)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:447)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:845)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:788)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2455)

at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:344)
at org.apache.hadoop.mapred.ClientServiceDelegate.getJobCounters(ClientServiceDelegate.java:381)
at org.apache.hadoop.mapred.YARNRunner.getJobCounters(YARNRunner.java:606)
at org.apache.hadoop.mapreduce.Job$8.run(Job.java:812)
at org.apache.hadoop.mapreduce.Job$8.run(Job.java:809)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
at org.apache.hadoop.mapreduce.Job.getCounters(Job.java:809)
at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1434)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1362)
at com.mapreduce.Driver.runJob(Driver.java:35)
at com.mapreduce.Driver.main(Driver.java:19)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:234)
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
Caused by: java.io.IOException: Unknown Job job_1503561276940_0001
at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.verifyAndGetJob(HistoryClientService.java:218)
at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getCounters(HistoryClientService.java:232)
at org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getCounters(MRClientProtocolPBServiceImpl.java:159)
at org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:281)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:447)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:845)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:788)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2455)

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:110)
at org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.unwrapAndThrowException(MRClientProtocolPBClientImpl.java:291)
at org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getCounters(MRClientProtocolPBClientImpl.java:168)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:325)
... 18 more
Caused by: org.apache.hadoop.ipc.RemoteException(java.io.IOException): Unknown Job job_1503561276940_0001
at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.verifyAndGetJob(HistoryClientService.java:218)
at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getCounters(HistoryClientService.java:232)
at org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getCounters(MRClientProtocolPBServiceImpl.java:159)
at org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:281)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:447)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:845)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:788)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2455)

at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1481)
at org.apache.hadoop.ipc.Client.call(Client.java:1427)
at org.apache.hadoop.ipc.Client.call(Client.java:1337)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy21.getCounters(Unknown Source)
at org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getCounters(MRClientProtocolPBClientImpl.java:166)
... 23 more
[root@master 6mouth]#
失败太多
运行jar失败原因：
出现以上错误是因为datanode的服务线程连接数都被占用，导致Spark等待超时。可以通过以下方法解决：

根据datanode的资源情况适当增加的服务线程数：在hdfs-default.xml配置文件里面新增或修改dfs.datanode.handler.count，默认是10
增加客户端的超时时间dfs.client.socket-timeout，默认是60000ms，注意，旧版本的Hadoop用的是dfs.socket.timeout，可参考http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/DeprecatedProperties.html

例如以下Java代码设置超时时间为300000ms=300s=5分钟：
运行jar包失败太多解决方法，修改：hdfs-site.xml
该文件与HDFS子项目密切相关，其参数对集群性能调整具有很大影响。
dfs.name.dir 指定name镜像文件存放目录，如不指定则默认为core-site中配置的tmp目录。
dfs.data.dir 真正的datanode数据保存路径，可以写多块硬盘，中间以逗号分隔。
dfs.replication hdfs数据块的复制份数，默认3，理论上份数越多跑数速度越快，但是需要的存储空间也更多。
dfs.permissions 是否需要角色权限验证，上传文件时会用到。一般设置false，设置为true有时候会遇到数据因为权限访问不了。
dfs.block.size 每个文件块的大小，默认是64M，对于大型文件可以设置为128M。
dfs.namenode.handler.count NameNode 节点上面为处理datanode节点来来气的远程调用的服务线程数量。
dfs.datanode.max.xcievers 相当于Linux下的打开文件最大数量，文档中无此参数，当出现DataXceiver报错的时候，需要调大。默认256
dfs.datanode.handler.count datanode节点上为处理datanode节点的远程调用开启的服务线程数量，默认为3。当有很多HDFS客户端时可以设置更大。

dfs.datanode.handler.count
500

dfs.client.socket-timeout
300000

修改后从新放入数据出现INFO提示：
[root@master hadoop]# hadoop fs -copyFromLocal /home/201706/* /I2
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81911ms (threshold=30000ms); ack: seqno: 1 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 82033ms (threshold=30000ms); ack: seqno: 2 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 82032ms (threshold=30000ms); ack: seqno: 3 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 82033ms (threshold=30000ms); ack: seqno: 4 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81914ms (threshold=30000ms); ack: seqno: 5 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81832ms (threshold=30000ms); ack: seqno: 6 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81832ms (threshold=30000ms); ack: seqno: 7 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81751ms (threshold=30000ms); ack: seqno: 8 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81751ms (threshold=30000ms); ack: seqno: 9 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81741ms (threshold=30000ms); ack: seqno: 10 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81741ms (threshold=30000ms); ack: seqno: 11 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81731ms (threshold=30000ms); ack: seqno: 12 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81730ms (threshold=30000ms); ack: seqno: 13 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81720ms (threshold=30000ms); ack: seqno: 14 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81719ms (threshold=30000ms); ack: seqno: 15 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81710ms (threshold=30000ms); ack: seqno: 16 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81714ms (threshold=30000ms); ack: seqno: 17 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81703ms (threshold=30000ms); ack: seqno: 18 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81702ms (threshold=30000ms); ack: seqno: 19 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81692ms (threshold=30000ms); ack: seqno: 20 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81692ms (threshold=30000ms); ack: seqno: 21 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81681ms (threshold=30000ms); ack: seqno: 22 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81681ms (threshold=30000ms); ack: seqno: 23 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81671ms (threshold=30000ms); ack: seqno: 24 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81671ms (threshold=30000ms); ack: seqno: 25 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81660ms (threshold=30000ms); ack: seqno: 26 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81660ms (threshold=30000ms); ack: seqno: 27 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81649ms (threshold=30000ms); ack: seqno: 28 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81650ms (threshold=30000ms); ack: seqno: 29 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81639ms (threshold=30000ms); ack: seqno: 30 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81639ms (threshold=30000ms); ack: seqno: 31 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81628ms (threshold=30000ms); ack: seqno: 32 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81628ms (threshold=30000ms); ack: seqno: 33 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81615ms (threshold=30000ms); ack: seqno: 34 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81615ms (threshold=30000ms); ack: seqno: 35 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81606ms (threshold=30000ms); ack: seqno: 36 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81607ms (threshold=30000ms); ack: seqno: 37 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81596ms (threshold=30000ms); ack: seqno: 38 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81596ms (threshold=30000ms); ack: seqno: 39 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81583ms (threshold=30000ms); ack: seqno: 40 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81584ms (threshold=30000ms); ack: seqno: 41 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81575ms (threshold=30000ms); ack: seqno: 42 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81575ms (threshold=30000ms); ack: seqno: 43 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81564ms (threshold=30000ms); ack: seqno: 44 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81565ms (threshold=30000ms); ack: seqno: 45 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81553ms (threshold=30000ms); ack: seqno: 46 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81553ms (threshold=30000ms); ack: seqno: 47 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81538ms (threshold=30000ms); ack: seqno: 48 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81539ms (threshold=30000ms); ack: seqno: 49 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81539ms (threshold=30000ms); ack: seqno: 50 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81531ms (threshold=30000ms); ack: seqno: 51 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81531ms (threshold=30000ms); ack: seqno: 52 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81520ms (threshold=30000ms); ack: seqno: 53 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81519ms (threshold=30000ms); ack: seqno: 54 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81510ms (threshold=30000ms); ack: seqno: 55 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81509ms (threshold=30000ms); ack: seqno: 56 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81499ms (threshold=30000ms); ack: seqno: 57 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81499ms (threshold=30000ms); ack: seqno: 58 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81488ms (threshold=30000ms); ack: seqno: 59 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81488ms (threshold=30000ms); ack: seqno: 60 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81477ms (threshold=30000ms); ack: seqno: 61 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81478ms (threshold=30000ms); ack: seqno: 62 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81462ms (threshold=30000ms); ack: seqno: 63 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81462ms (threshold=30000ms); ack: seqno: 64 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81449ms (threshold=30000ms); ack: seqno: 65 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81450ms (threshold=30000ms); ack: seqno: 66 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81439ms (threshold=30000ms); ack: seqno: 67 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81439ms (threshold=30000ms); ack: seqno: 68 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81439ms (threshold=30000ms); ack: seqno: 69 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81422ms (threshold=30000ms); ack: seqno: 70 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81423ms (threshold=30000ms); ack: seqno: 71 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81423ms (threshold=30000ms); ack: seqno: 72 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81411ms (threshold=30000ms); ack: seqno: 73 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81399ms (threshold=30000ms); ack: seqno: 74 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81400ms (threshold=30000ms); ack: seqno: 75 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81400ms (threshold=30000ms); ack: seqno: 76 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81391ms (threshold=30000ms); ack: seqno: 77 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81391ms (threshold=30000ms); ack: seqno: 78 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81380ms (threshold=30000ms); ack: seqno: 79 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81380ms (threshold=30000ms); ack: seqno: 80 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 80425ms (threshold=30000ms); ack: seqno: 81 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
[root@master hadoop]#
修改后再次运行jar包代码：
出错一大堆。
解决（有点投机的方法）：重新删掉数据、tmp、log，重新格式化，然后相当于重新启动集群。重新放入数据，运行jar包代码。

你可能感兴趣的:(大数据)

nosql数据库技术与应用知识点皆过客，揽星河 NoSQL nosql 数据库大数据数据分析数据结构非关系型数据库
Nosql知识回顾大数据处理流程数据采集(flume、爬虫、传感器)数据存储(本门课程NoSQL所处的阶段)Hdfs、MongoDB、HBase等数据清洗(入仓)Hive等数据处理、分析(Spark、Flink等)数据可视化数据挖掘、机器学习应用(Python、SparkMLlib等)大数据时代存储的挑战(三高)高并发(同一时间很多人访问)高扩展(要求随时根据需求扩展存储)高效率(要求读写速度快)
ES聚合分析原理与代码实例讲解光剑书架上的书大厂Offer收割机面试题简历程序员读书硅基计算碳基计算认知计算生物计算深度学习神经网络大数据 AIGC AGI LLM Java Python 架构设计 Agent 程序员实现财富自由
ES聚合分析原理与代码实例讲解1.背景介绍1.1问题的由来在大规模数据分析场景中，特别是在使用Elasticsearch（ES）进行数据存储和检索时，聚合分析成为了一个至关重要的功能。聚合分析允许用户对数据集进行细分和分组，以便深入探索数据的结构和模式。这在诸如实时监控、日志分析、业务洞察等领域具有广泛的应用。1.2研究现状目前，ES聚合分析已经成为现代大数据平台的核心组件之一。它支持多种类型的聚
WebMagic：强大的Java爬虫框架解析与实战 Aaron_945 Java java 爬虫开发语言
文章目录引言官网链接WebMagic原理概述基础使用1.添加依赖2.编写PageProcessor高级使用1.自定义Pipeline2.分布式抓取优点结论引言在大数据时代，网络爬虫作为数据收集的重要工具，扮演着不可或缺的角色。Java作为一门广泛使用的编程语言，在爬虫开发领域也有其独特的优势。WebMagic是一个开源的Java爬虫框架，它提供了简单灵活的API，支持多线程、分布式抓取，以及丰富的
免费的GPT可在线直接使用（一键收藏） kkai人工智能 gpt
1、LuminAI（https://kk.zlrxjh.top）LuminAI标志着一款融合了星辰大数据模型与文脉深度模型的先进知识增强型语言处理系统，旨在自然语言处理（NLP）的技术开发领域发光发热。此系统展现了卓越的语义把握与内容生成能力，轻松驾驭多样化的自然语言处理任务。VisionAI在NLP界的应用领域广泛，能够胜任从机器翻译、文本概要撰写、情绪分析到问答等众多任务。通过对大量文本数据的
如何利用大数据与AI技术革新相亲交友体验 h17711347205 回归算法安全系统架构交友小程序
在数字化时代，大数据和人工智能（AI）技术正逐渐革新相亲交友体验，为寻找爱情的过程带来前所未有的变革（编辑h17711347205）。通过精准分析和智能匹配，这些技术能够极大地提高相亲交友系统的效率和用户体验。大数据的力量大数据技术能够收集和分析用户的行为模式、偏好和互动数据，为相亲交友系统提供丰富的信息资源。通过分析用户的搜索历史、浏览记录和点击行为，系统能够深入了解用户的兴趣和需求，从而提供更
未来软件市场是怎么样的？做开发的生存空间如何？ cesske 软件需求
目录前言一、未来软件市场的发展趋势二、软件开发人员的生存空间前言未来软件市场是怎么样的？做开发的生存空间如何？一、未来软件市场的发展趋势技术趋势：人工智能与机器学习：随着技术的不断成熟，人工智能将在更多领域得到应用，如智能客服、自动驾驶、智能制造等，这将极大地推动软件市场的增长。云计算与大数据：云计算服务将继续普及，大数据技术的应用也将更加广泛。企业将更加依赖云计算和大数据来优化运营、提升效率，并
Hadoop架构 henan程序媛 hadoop 大数据分布式
一、案列分析1.1案例概述现在已经进入了大数据(BigData)时代，数以万计用户的互联网服务时时刻刻都在产生大量的交互，要处理的数据量实在是太大了，以传统的数据库技术等其他手段根本无法应对数据处理的实时性、有效性的需求。HDFS顺应时代出现，在解决大数据存储和计算方面有很多的优势。1.2案列前置知识点1.什么是大数据大数据是指无法在一定时间范围内用常规软件工具进行捕捉、管理和处理的大量数据集合，
[转载] NoSQL简介 weixin_30325793 大数据数据库运维
摘自“百度百科”。NoSQL，泛指非关系型的数据库。随着互联网web2.0网站的兴起，传统的关系数据库在应付web2.0网站，特别是超大规模和高并发的SNS类型的web2.0纯动态网站已经显得力不从心，暴露了很多难以克服的问题，而非关系型的数据库则由于其本身的特点得到了非常迅速的发展。NoSQL数据库的产生就是为了解决大规模数据集合多重数据种类带来的挑战，尤其是大数据应用难题。虽然NoSQL流行语
Kafka详细解析与应用分析芊言芊语 kafka 分布式
Kafka是一个开源的分布式事件流平台（EventStreamingPlatform），由LinkedIn公司最初采用Scala语言开发，并基于ZooKeeper协调管理。如今，Kafka已经被Apache基金会纳入其项目体系，广泛应用于大数据实时处理领域。Kafka凭借其高吞吐量、持久化、分布式和可靠性的特点，成为构建实时流数据管道和流处理应用程序的重要工具。Kafka架构Kafka的架构主要由
分享一个基于python的电子书数据采集与可视化分析 hadoop电子书数据分析与推荐系统 spark大数据毕设项目（源码、调试、LW、开题、PPT) 计算机源码社 Python项目大数据大数据 python hadoop 计算机毕业设计选题计算机毕业设计源码数据分析 spark毕设
作者：计算机源码社个人简介：本人八年开发经验，擅长Java、Python、PHP、.NET、Node.js、Android、微信小程序、爬虫、大数据、机器学习等，大家有这一块的问题可以一起交流！学习资料、程序开发、技术解答、文档报告如需要源码，可以扫取文章下方二维码联系咨询Java项目微信小程序项目Android项目Python项目PHP项目ASP.NET项目Node.js项目选题推荐项目实战|p
疫情，疫情东山草
2020年，疫情爆发，至今已近三年，反反复复，此起彼伏。不但没被消灭，还自我发展，从德尔塔到奥密克戎，与时俱进的变异着。去年11月，疫情之下，大数据800米范围内，都成为时空伴随者。“你的码儿有没有变颜色”“你绿码还是黄码”成为那段时间的流行语，当然少不了的还有全员核酸。段子手整出来一首歌：我走过你走过的路,这算不算相逢？我吹过你吹过的风，这算不算相拥？800米内我们不曾擦肩而过，你却要我14天相
在服务器计算节点中使用 jupyter Lab ranshan567 程序人生
JupyterLab是一个基于网页的交互式开发环境,用于科学计算、数据分析和机器学.jupyterlab是jupyternotebook的下一代产品,集成了更多功能,使用起来更方便.在进行数据分析及可视化时，个人电脑不能满足大数据的分析需求，就需要用到高性能计算机集群资源，然而计算机集群的计算节点往往没有联网功能，所以在计算机集群中使用jupyterLab需要进行一些配置。具体的步骤如下：
大数据真实面试题---SQL The博宇大数据面试题——SQL 大数据 mysql sql 数据库 big data
视频号数据分析组外包招聘笔试题时间限时45分钟完成。题目根据3张表表结构，写出具体求解的SQL代码（搞笑品类定义：视频分类或者视频创建者分类为“搞笑”）1、表创建语句：createtablet_user_video_action_d(dsint,user_idstring,video_idstring,action_typeint,`timestamp`bigint)rowformatdelimi
Flume：大规模日志收集与数据传输的利器傲雪凌霜，松柏长青后端大数据 flume 大数据
Flume：大规模日志收集与数据传输的利器在大数据时代，随着各类应用的不断增长，产生了海量的日志和数据。这些数据不仅对业务的健康监控至关重要，还可以通过深入分析，帮助企业做出更好的决策。那么，如何高效地收集、传输和存储这些海量数据，成为了一项重要的挑战。今天我们将深入探讨ApacheFlume，它是如何帮助我们应对这些挑战的。一、Flume概述ApacheFlume是一个分布式、可靠、可扩展的日志
云服务业界动态简报-20180128 Captain7
一、青云青云QingCloud推出深度学习平台DeepLearningonQingCloud，包含了主流的深度学习框架及数据科学工具包，通过QingCloudAppCenter一键部署交付，可以让算法工程师和数据科学家快速构建深度学习开发环境，将更多的精力放在模型和算法调优。二、腾讯云1.腾讯云正式发布腾讯专有云TCE(TencentCloudEnterprise)矩阵，涵盖企业版、大数据版、AI
大数据毕业设计hadoop+spark+hive知识图谱租房数据分析可视化大屏租房推荐系统 58同城租房爬虫房源推荐系统房价预测系统计算机毕业设计机器学习深度学习人工智能 2401_84572577 程序员大数据 hadoop 人工智能
做了那么多年开发，自学了很多门编程语言，我很明白学习资源对于学一门新语言的重要性，这些年也收藏了不少的Python干货，对我来说这些东西确实已经用不到了，但对于准备自学Python的人来说，或许它就是一个宝藏，可以给你省去很多的时间和精力。别在网上瞎学了，我最近也做了一些资源的更新，只要你是我的粉丝，这期福利你都可拿走。我先来介绍一下这些东西怎么用，文末抱走。（1）Python所有方向的学习路线（
架构评审的自动化与人工智能: 如何提高效率光剑书架上的书架构自动化人工智能运维
1.背景介绍架构评审是软件开发过程中的一个关键环节，它旨在确保软件架构的质量、可维护性和可扩展性。传统的架构评审通常是由人工进行，需要大量的时间和精力。随着大数据技术和人工智能的发展，自动化和人工智能技术已经开始应用于架构评审，从而提高评审的效率和准确性。在本文中，我们将讨论如何通过自动化和人工智能技术来提高架构评审的效率。我们将从以下几个方面进行讨论：背景介绍核心概念与联系核心算法原理和具体操作
【数字化供应链】数字化供应链架构、全景管理、全流程贯通方案数字化建设方案数字化转型数据治理主数据数据仓库供应链数字仓储智慧物流智慧仓储物流园区架构微服务数据挖掘大数据人工智能
原文《数字化供应链架构、全景管理、全流程贯通方案》PPT格式。主要从供应链管理全景、智慧供应链建设总体目标、供应链总体业务流程、供应链总体功能架构、供应链总体技术架构、供应链全流程贯通、供应链全领域管理、供应链数据数据分析、供应链决策中台等进行建设。本文仅对主要内容进行介绍。来源网络公开渠道，旨在交流学习，如有侵权联系速删，更多参考公众号：优享智库基于先进IT技术、大数据能力、物联网应用、区块链平
80 鑫_259b
科普一个谈恋爱的方法。在以前，谈恋爱千难万难，就难在对对方不知底细，不知道对方希望自己是一个怎样的人，要耗费大量的时间去试探、再磨合，往往会因为一些小事一些细节，满盘皆输。在一个信息化的时代，在一个大数据近乎变成了流行语的时代，我们要跟上时代的步伐，通过大数据，去寻找异性最希望自己展现出来的形象是什么，才可以在爱情的道路上少走弯路。那这个大数据怎么操作呢？上街发问卷？问别人的择偶标准？一来会被打死
解锁企业潜能，Vatee万腾平台引领智能新纪元自媒体经济说其他
在数字化转型的浪潮中，企业正站在一个前所未有的十字路口，面对着前所未有的机遇与挑战。解锁企业内在潜能，实现跨越式发展，已成为众多企业的共同追求。而Vatee万腾平台，作为智能科技的先锋，正以其强大的智能赋能能力，引领企业步入一个全新的智能纪元。Vatee万腾平台，是一个集成了人工智能、大数据、云计算等前沿技术的综合性智能服务平台。它不仅仅是一个技术工具，更是企业转型升级的加速器，能够深入企业运营的
释放“AI+”新质生产力，深算院如何“把大数据变小”？ YashanDB YashanDB 国产数据库数据库数据库大数据
近期，南都·湾财社推出《新质·中国造》栏目，深入千行百业，遍访湾区企业，解锁湾区新质生产力，共探高质量发展之道。本期对话深圳计算科学研究院YashanDB首席技术官陈志标，探讨国产数据库如何实现创新突围，抢抓数字经济时代的新机遇。以下是专访内容：如何应对AI时代所面临的算力挑战？南都·湾财社：数据、算力和算法是发展人工智能的三要素，深算院做了怎样的前瞻性布局？陈志标：今年，政府工作报告中首次提及开
数字化智能工厂数字化供应链架构、全景管理、全流程贯通方案数字化建设方案智能制造数字工厂制造业数字化转型工业互联网架构
随着信息技术的飞速发展，数字化转型已成为制造企业提升竞争力的关键途径。数字化智能工厂通过集成先进的物联网(IoT)、大数据、云计算、人工智能(AI)等技术，实现了生产过程的智能化、供应链管理的精准化及决策的科学化。本方案旨在构建一套完善的数字化供应链架构，实现全景管理、全流程贯通、智慧化升级，以数据为驱动，强化技术支撑与安全管理体系，推动企业向智能制造迈进。一、数字化供应链架构1.**集成化平台构
日记——我的歌单静若小猴
又到一年一度大数据汇总的时候了，听歌已经成为很多人生活里的一种乐趣。春夏秋冬，我们都有自己喜欢的歌，歌词歌曲唱出沃尔玛你的心声。还记得大学时候最喜欢听的《春天里》，我有一天单曲回放了30遍，总觉得听着仿佛看到自己声音。还有的歌，初听不知曲中意，再听已经是曲终人，听着歌流泪，听着歌入睡……还记得那些年少的故事吗，总觉得自己才是故事外的人，却不是自己已经入歌。一段时间会喜欢一个人的音乐，一段时间会沉静
Linux dmesg命令：显示开机信息 fafadsj666 linux 数据库数据挖掘机器学习大数据
通过学习《Linux启动管理》一章可以知道，在系统启动过程中，内核还会进行一次系统检测（第一次是BIOS进行加测），但是检测的过程不是没有显示在屏幕上，就是会快速的在屏幕上一闪而过那么，如果开机时来不及查看相关信息，我们是否可以在开机后查看呢？答案是肯定的，使用dmesg命令就可以。无论是系统启动过程中，还是系统运行过程中，只要是内核产生的信息，都会被存储在系统缓冲区中，已经为大家精心准备了大数据
大数据新视界 --大数据大厂之揭秘大数据时代 Excel 魔法：大厂数据分析师进阶秘籍青云交大数据新视界 Excel 数据分析函数公式数据透视表图表功能规划求解数据分析工具库大数据新视界数据库
亲爱的朋友们，热烈欢迎你们来到青云交的博客！能与你们在此邂逅，我满心欢喜，深感无比荣幸。在这个瞬息万变的时代，我们每个人都在苦苦追寻一处能让心灵安然栖息的港湾。而我的博客，正是这样一个温暖美好的所在。在这里，你们不仅能够收获既富有趣味又极为实用的内容知识，还可以毫无拘束地畅所欲言，尽情分享自己独特的见解。我真诚地期待着你们的到来，愿我们能在这片小小的天地里共同成长，共同进步。本博客的精华专栏：Ja
大数据新视界 --大数据大厂之数据挖掘入门：用 R 语言开启数据宝藏的探索之旅青云交大数据新视界数据库大数据数据挖掘 R 语言算法案例未来趋势应用场景学习建议大数据新视界
亲爱的朋友们，热烈欢迎你们来到青云交的博客！能与你们在此邂逅，我满心欢喜，深感无比荣幸。在这个瞬息万变的时代，我们每个人都在苦苦追寻一处能让心灵安然栖息的港湾。而我的博客，正是这样一个温暖美好的所在。在这里，你们不仅能够收获既富有趣味又极为实用的内容知识，还可以毫无拘束地畅所欲言，尽情分享自己独特的见解。我真诚地期待着你们的到来，愿我们能在这片小小的天地里共同成长，共同进步。本博客的精华专栏：Ja
高职人工智能训练师边缘计算实训室解决方案武汉唯众智创人工智能训练师边缘计算实训室人工智能训练师实训室边缘计算实训室
一、引言随着物联网（IoT）、大数据、人工智能（AI）等技术的飞速发展，计算需求日益复杂和多样化。传统的云计算模式虽在一定程度上满足了这些需求，但在处理海量数据、保障实时性与安全性、提升计算效率等方面仍面临诸多挑战。在此背景下，边缘计算作为一种新兴的计算模式应运而生，通过将计算能力推向数据生成或用户所在的网络边缘，显著降低了数据传输的延迟，提升了处理效率，并增强了数据安全性。针对高等职业院校的人工
python基于django/flask的NBA球员大数据分析与可视化python+java+node.js QQ_511008285 python django flask java spring boot 数据分析
前端开发框架:vue.js数据库mysql版本不限后端语言框架支持：1java(SSM/springboot)-idea/eclipse2.Nodejs+Vue.js-vscode3.python(flask/django)--pycharm/vscode4.php(thinkphp/laravel)-hbuilderx数据库工具：Navicat/SQLyog等都可以本文针对NBA球员的大数据进行
Java基于spring boot的国产电影数据分析与可视化python+java+node.js QQ_511008285 java spring boot 数据分析 python django vue.js flask
前端开发框架:vue.js数据库mysql版本不限后端语言框架支持：1java(SSM/springboot)-idea/eclipse2.Nodejs+Vue.js-vscode3.python(flask/django)--pycharm/vscode4.php(thinkphp/laravel)-hbuilderx数据库工具：Navicat/SQLyog等都可以该系统使用进行大数据处理和
数字化（电子化）招标采购平台系统核心功能详细介绍 xinyuan_123456 oracle
数智化招标采购平台覆盖全业务类型、全采购流程、全采购方式，是郑州信源公司运用“互联网+”、大数据、人工智能、区块链、物联网等新兴技术，结合供应链管理理念，以招标采购为核心，提供交易、管理、数据、服务、监管为一体的高标准采购管理平台，赋能政企用户实现采购业务全流程的电子化、数字化、智慧化。根据产品功能及应用领域，产品包括：企业数智化招采供应链平台、金融数智化招采平台、政府数智化采购平台、公共资源数智
github中多个平台共存 jackyrong github
在个人电脑上，如何分别链接比如oschina,github等库呢，一般教程之列的，默认 ssh链接一个托管的而已，下面讲解如何放两个文件 1）设置用户名和邮件地址 $ git config --global user.name "xx" $ git config --global user.email "[email protected]"
ip地址与整数的相互转换(javascript) alxw4616 JavaScript
//IP转成整型 function ip2int(ip){ var num = 0; ip = ip.split("."); num = Number(ip[0]) * 256 * 256 * 256 + Number(ip[1]) * 256 * 256 + Number(ip[2]) * 256 + Number(ip[3]); n
读书笔记-jquey+数据库+css chengxuyuancsdn html jquery oracle
1、grouping ,group by rollup, GROUP BY GROUPING SETS区别 2、$("#totalTable tbody>tr td:nth-child(" + i + ")").css({"width":tdWidth, "margin":"0px", &q
javaSE javaEE javaME == API下载 Array_06 java
oracle下载各种API文档： http://www.oracle.com/technetwork/java/embedded/javame/embed-me/documentation/javame-embedded-apis-2181154.html JavaSE文档： http://docs.oracle.com/javase/8/docs/api/ JavaEE文档： ht
shiro入门学习 cugfy java Web 框架
声明本文只适合初学者，本人也是刚接触而已，经过一段时间的研究小有收获，特来分享下希望和大家互相交流学习。首先配置我们的web.xml代码如下，固定格式，记死就成 <filter> <filter-name>shiroFilter</filter-name> &nbs
Array添加删除方法 357029540 js
刚才做项目前台删除数组的固定下标值时，删除得不是很完整，所以在网上查了下，发现一个不错的方法，也提供给需要的同学。 //给数组添加删除 Array.prototype.del = function(n){
navigation bar 更改颜色张亚雄 IO
今天郁闷了一下午，就因为objective-c默认语言是英文，我写的中文全是一些乱七八糟的样子，到不是乱码，但是，前两个自字是粗体，后两个字正常体，这可郁闷死我了，问了问大牛，人家告诉我说更改一下字体就好啦，比如改成黑体，哇塞，茅塞顿开。翻书看，发现，书上有介绍怎么更改表格中文字字体的，代码如下
unicode转换成中文 adminjun unicode 编码转换
在Java程序中总会出现\u6b22\u8fce\u63d0\u4ea4\u5fae\u535a\u641c\u7d22\u4f7f\u7528\u53cd\u9988\uff0c\u8bf7\u76f4\u63a5这个的字符，这是unicode编码，使用时有时候不会自动转换成中文就需要自己转换了使用下面的方法转换一下即可。 /** * unicode 转换成中文
一站式 Java Web 框架 firefly aijuans Java Web
Firefly是一个高性能一站式Web框架。涵盖了web开发的主要技术栈。包含Template engine、IOC、MVC framework、HTTP Server、Common tools、Log、Json parser等模块。 firefly-2.0_07修复了模版压缩对javascript单行注释的影响，并新增了自定义错误页面功能。更新日志：增加自定义系统错误页面功能
设计模式——单例模式 ayaoxinchao 设计模式
定义 Java中单例模式定义：“一个类有且仅有一个实例，并且自行实例化向整个系统提供。” 分析从定义中可以看出单例的要点有三个：一是某个类只能有一个实例；二是必须自行创建这个实例；三是必须自行向系统提供这个实例。 &nb
Javascript 多浏览器兼容性问题及解决方案 BigBird2012 JavaScript
不论是网站应用还是学习js,大家很注重ie与firefox等浏览器的兼容性问题，毕竟这两中浏览器是占了绝大多数。一、document.formName.item(”itemName”) 问题问题说明：IE下，可以使用 document.formName.item(”itemName”) 或 document.formName.elements ["elementName&quo
JUnit-4.11使用报java.lang.NoClassDefFoundError: org/hamcrest/SelfDescribing错误 bijian1013 junit4.11 单元测试
下载了最新的JUnit版本，是4.11，结果尝试使用发现总是报java.lang.NoClassDefFoundError: org/hamcrest/SelfDescribing这样的错误，上网查了一下，一般的解决方案是，换一个低一点的版本就好了。还有人说，是缺少hamcrest的包。去官网看了一下，如下发现：
[Zookeeper学习笔记之二]Zookeeper部署脚本 bit1129 zookeeper
Zookeeper伪分布式安装脚本(此脚本在一台机器上创建Zookeeper三个进程，即创建具有三个节点的Zookeeper集群。这个脚本和zookeeper的tar包放在同一个目录下，脚本中指定的名字是zookeeper的3.4.6版本，需要根据实际情况修改)： #!/bin/bash #!!!Change the name!!! #The zookeepe
【Spark八十】Spark RDD API二 bit1129 spark
coGroup package spark.examples.rddapi import org.apache.spark.{SparkConf, SparkContext} import org.apache.spark.SparkContext._ object CoGroupTest_05 { def main(args: Array[String]) { v
Linux中编译apache服务器modules文件夹缺少模块(.so)的问题 ronin47 modules
在modules目录中只有httpd.exp，那些so文件呢？我尝试在fedora core 3中安装apache 2. 当我解压了apache 2.0.54后使用configure工具并且加入了 --enable-so 或者 --enable-modules=so (两个我都试过了) 去make并且make install了。我希望在/apache2/modules/目录里有各种模块，
Java基础-克隆 BrokenDreams java基础
Java中怎么拷贝一个对象呢？可以通过调用这个对象类型的构造器构造一个新对象，然后将要拷贝对象的属性设置到新对象里面。Java中也有另一种不通过构造器来拷贝对象的方式，这种方式称为克隆。 Java提供了java.lang.
读《研磨设计模式》-代码笔记-适配器模式-Adapter bylijinnan java 设计模式
声明：本文只为方便我个人查阅和理解，详细的分析以及源代码请移步原作者的博客http://chjavach.iteye.com/ package design.pattern; /* * 适配器模式解决的主要问题是，现有的方法接口与客户要求的方法接口不一致 * 可以这样想，我们要写这样一个类（Adapter）: * 1.这个类要符合客户的要求 ---> 那显然要
HDR图像PS教程集锦&心得 cherishLC PS
HDR是指高动态范围的图像，主要原理为提高图像的局部对比度。软件有photomatix和nik hdr efex。一、教程叶明在知乎上的回答： http://www.zhihu.com/question/27418267/answer/37317792 大意是修完后直方图最好是等值直方图，方法是HDR软件调一遍，再结合不透明度和蒙版细调。二、心得 1、去除阴影部分的
maven-3.3.3 mvn archetype 列表 crabdave ArcheType
maven-3.3.3 mvn archetype 列表可以参考最新的：http://repo1.maven.org/maven2/archetype-catalog.xml [INFO] Scanning for projects... [INFO]
linux shell 中文件编码查看及转换方法 daizj shell 中文乱码 vim 文件编码
一、查看文件编码。在打开文件的时候输入:set fileencoding 即可显示文件编码格式。二、文件编码转换 1、在Vim中直接进行转换文件编码,比如将一个文件转换成utf-8格式 &
MySQL--binlog日志恢复数据 dcj3sjt126com binlog
恢复数据的重要命令如下 mysql> flush logs; 默认的日志是mysql-bin.000001，现在刷新了重新开启一个就多了一个mysql-bin.000002
数据库中数据表数据迁移方法 dcj3sjt126com sql
刚开始想想好像挺麻烦的，后来找到一种方法了，就SQL中的 INSERT 语句，不过内容是现从另外的表中查出来的，其实就是 MySQL中INSERT INTO SELECT的使用下面看看如何使用语法：MySQL中INSERT INTO SELECT的使用 1. 语法介绍有三张表a、b、c，现在需要从表b
Java反转字符串 dyy_gusi java 反转字符串
前几天看见一篇文章，说使用Java能用几种方式反转一个字符串。首先要明白什么叫反转字符串，就是将一个字符串到过来啦，比如"倒过来念的是小狗"反转过来就是”狗小是的念来过倒“。接下来就把自己能想到的所有方式记录下来了。 1、第一个念头就是直接使用String类的反转方法，对不起，这样是不行的，因为Stri
UI设计中我们为什么需要设计动效 gcq511120594 UI linux
随着国际大品牌苹果和谷歌的引领，最近越来越多的国内公司开始关注动效设计了，越来越多的团队已经意识到动效在产品用户体验中的重要性了，更多的UI设计师们也开始投身动效设计领域。但是说到底，我们到底为什么需要动效设计？或者说我们到底需要什么样的动效？做动效设计也有段时间了，于是尝试用一些案例，从产品本身出发来说说我所思考的动效设计。一、加强体验舒适度嗯，就是让用户更加爽更加爽的用
JBOSS服务部署端口冲突问题 HogwartsRow java 应用服务器 jboss server EJB3
服务端口冲突问题的解决方法，一般修改如下三个文件中的部分端口就可以了。 1、jboss5/server/default/conf/bindingservice.beans/META-INF/bindings-jboss-beans.xml 2、./server/default/deploy/jbossweb.sar/server.xml 3、.
第三章 Redis/SSDB+Twemproxy安装与使用 jinnianshilongnian ssdb reids twemproxy
目前对于互联网公司不使用Redis的很少，Redis不仅仅可以作为key-value缓存，而且提供了丰富的数据结果如set、list、map等，可以实现很多复杂的功能；但是Redis本身主要用作内存缓存，不适合做持久化存储，因此目前有如SSDB、ARDB等，还有如京东的JIMDB，它们都支持Redis协议，可以支持Redis客户端直接访问；而这些持久化存储大多数使用了如LevelDB、RocksD
ZooKeeper原理及使用 liyonghui160com
ZooKeeper是Hadoop Ecosystem中非常重要的组件，它的主要功能是为分布式系统提供一致性协调(Coordination)服务，与之对应的Google的类似服务叫Chubby。今天这篇文章分为三个部分来介绍ZooKeeper，第一部分介绍ZooKeeper的基本原理，第二部分介绍ZooKeeper
程序员解决问题的60个策略 pda158 框架工作单元测试
根本的指导方针 1. 首先写代码的时候最好不要有缺陷。最好的修复方法就是让 bug 胎死腹中。良好的单元测试强制数据库约束使用输入验证框架避免未实现的“else”条件在应用到主程序之前知道如何在孤立的情况下使用日志 2. print 语句。往往额外输出个一两行将有助于隔离问题。 3. 切换至详细的日志记录。详细的日
Create the Google Play Account sillycat Google
Create the Google Play Account Having a Google account, pay 25$, then you get your google developer account. References: http://developer.android.com/distribute/googleplay/start.html https://p
JSP三大指令 vikingwei jsp
JSP三大指令一个jsp页面中，可以有0~N个指令的定义！ 1. page --> 最复杂：<%@page language="java" info="xxx"...%> * pageEncoding和contentType： > pageEncoding：它