CentOS 7部署Hadoop（伪分布式）
目录
测试环境
安装CentOS 7
配置Java环境
配置单机版Hadoop
配置SSH免密登录
修改Hadoop配置文件
关闭防火墙
HDFS初始化
启动Hadoop
关闭Hadoop
测试环境
Linux系统版本：CentOS 7 64位
Hadoop版本：hadoop-2.7.3
Java版本：jdk-8u181-linux-x64
安装CentOS 7
VMware虚拟机安装Linux系统
配置Java环境
Linux系统下安装Java环境
配置单机版Hadoop
CentOS 7部署Hadoop（单机版）
配置SSH免密登录
用hadoop用户登录进行下面的操作：
1、生成公钥私钥对，输入下面命令一直回车就可以了
ssh-keygen -t rsa执行情况如下：
[hadoop@master100 ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:nx33fWWuccTRMo+zMzr/v4bMenWYFoPuFmOucqdwp90 hadoop@master100
The key's randomart image is:
+---[RSA 2048]----+
| |
| .|
| .o..|
| . o*.|
| S .. +=*|
| . o=o+O+|
| . ++=+B B|
| .o.+*B B.|
| o+*=oEo=|
+----[SHA256]-----+
2、实现本地免密登录,将id_rsa.pub中的内容拷贝到authorized_keys
ssh-copy-id localhost
[hadoop@master100 ~]$ ssh-copy-id localhost
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/hadoop/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
hadoop@localhost's password: //这里输入hadoop用户的密码
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'localhost'"
and check to make sure that only the key(s) you wanted were added.
~/.ssh目录下会生成一个新的文件:authorized_keys，如下：
[hadoop@master100 ~]$ cd ~/.ssh
[hadoop@master100 .ssh]$ ll
总用量 16
-rw-------. 1 hadoop hadoop 398 10月 3 19:47 authorized_keys
-rw-------. 1 hadoop hadoop 1679 10月 3 19:46 id_rsa
-rw-r--r--. 1 hadoop hadoop 398 10月 3 19:46 id_rsa.pub
-rw-r--r--. 1 hadoop hadoop 171 10月 3 19:42 known_hosts
3、完成上述步骤后就可以本地SSH免密登录了,运行下面代码出现一行登录时间就代表本地SSH免密登录成功
ssh localhost下面是本地SSH免密登录成功的标志:
[hadoop@master100 ~]$ ssh localhost
Last login: Wed Oct 3 19:45:11 2018 from 192.168.33.2
想了解更多，可以参考下文或者是自己百度相关知识
Linux系统配置SSH免密登录(多主机互通)
修改Hadoop配置文件
1、进入/usr/local/hadoop/etc/hadoop/目录，可以看到如下配置文件，如果需要修改的文件不存在，那就创建一个，vi命令可以直接创建
[hadoop@master100 ~]$ cd /usr/local/hadoop/etc/hadoop/
[hadoop@master100 hadoop]$ ll
总用量 152
-rw-r--r--. 1 hadoop hadoop 4436 8月 18 2016 capacity-scheduler.xml
-rw-r--r--. 1 hadoop hadoop 1335 8月 18 2016 configuration.xsl
-rw-r--r--. 1 hadoop hadoop 318 8月 18 2016 container-executor.cfg
-rw-r--r--. 1 hadoop hadoop 774 8月 18 2016 core-site.xml
-rw-r--r--. 1 hadoop hadoop 3589 8月 18 2016 hadoop-env.cmd
-rw-r--r--. 1 hadoop hadoop 4224 8月 18 2016 hadoop-env.sh
-rw-r--r--. 1 hadoop hadoop 2598 8月 18 2016 hadoop-metrics2.properties
-rw-r--r--. 1 hadoop hadoop 2490 8月 18 2016 hadoop-metrics.properties
-rw-r--r--. 1 hadoop hadoop 9683 8月 18 2016 hadoop-policy.xml
-rw-r--r--. 1 hadoop hadoop 775 8月 18 2016 hdfs-site.xml
-rw-r--r--. 1 hadoop hadoop 1449 8月 18 2016 httpfs-env.sh
-rw-r--r--. 1 hadoop hadoop 1657 8月 18 2016 httpfs-log4j.properties
-rw-r--r--. 1 hadoop hadoop 21 8月 18 2016 httpfs-signature.secret
-rw-r--r--. 1 hadoop hadoop 620 8月 18 2016 httpfs-site.xml
-rw-r--r--. 1 hadoop hadoop 3518 8月 18 2016 kms-acls.xml
-rw-r--r--. 1 hadoop hadoop 1527 8月 18 2016 kms-env.sh
-rw-r--r--. 1 hadoop hadoop 1631 8月 18 2016 kms-log4j.properties
-rw-r--r--. 1 hadoop hadoop 5511 8月 18 2016 kms-site.xml
-rw-r--r--. 1 hadoop hadoop 11237 8月 18 2016 log4j.properties
-rw-r--r--. 1 hadoop hadoop 931 8月 18 2016 mapred-env.cmd
-rw-r--r--. 1 hadoop hadoop 1383 8月 18 2016 mapred-env.sh
-rw-r--r--. 1 hadoop hadoop 4113 8月 18 2016 mapred-queues.xml.template
-rw-r--r--. 1 hadoop hadoop 758 8月 18 2016 mapred-site.xml.template
-rw-r--r--. 1 hadoop hadoop 10 8月 18 2016 slaves
-rw-r--r--. 1 hadoop hadoop 2316 8月 18 2016 ssl-client.xml.example
-rw-r--r--. 1 hadoop hadoop 2268 8月 18 2016 ssl-server.xml.example
-rw-r--r--. 1 hadoop hadoop 2191 8月 18 2016 yarn-env.cmd
-rw-r--r--. 1 hadoop hadoop 4567 8月 18 2016 yarn-env.sh
-rw-r--r--. 1 hadoop hadoop 690 8月 18 2016 yarn-site.xml
2、修改core-site.xml文件
[hadoop@master100 hadoop]$ vi core-site.xml配置文件如下：
hadoop.tmp.dir file:/usr/local/hadoop/tmp 指定hadoop运行时产生文件的存储路径 fs.defaultFS hdfs://localhost:9000 hdfs namenode的通信地址,通信端口
3、修改hdfs-site.xml
vi hdfs-site.xml配置文件如下：
dfs.replication 1 指定HDFS存储数据的副本数目，默认情况下是3份 dfs.namenode.name.dir file:/usr/local/hadoop/hadoopdata/namenode namenode存放数据的目录 dfs.datanode.data.dir file:/usr/local/hadoop/hadoopdata/datanode datanode存放block块的目录 dfs.permissions.enabled false 关闭权限验证
4、修改mapred-site.xml
/usr/local/hadoop/etc/hadoop文件夹中并没有mapred-site.xml文件，但提供了模板mapred-site.xml.template，将其复制一份重命名为mapred-site.xml 即可
[hadoop@master100 hadoop]$ cp mapred-site.xml.template mapred-site.xml
[hadoop@master100 hadoop]$ vi mapred-site.xml
配置文件如下：
mapreduce.framework.name yarn 指定mapreduce运行在yarn上
5、修改yarn-site.xml
[hadoop@master100 hadoop]$ vi yarn-site.xmlyarn.nodemanager.aux-services mapreduce_shuffle mapreduce执行shuffle时获取数据的方式
6、为了防止运行时报错，修改一下hadoop-env.sh文件
[hadoop@master100 hadoop]$ vi hadoop-env.sh找到export JAVA_HOME=${JAVA_HOME}，在前面加个#注释掉，将JAVA_HOME用路径代替，如下：
#export JAVA_HOME=${JAVA_HOME} export JAVA_HOME=/usr/java/jdk1.8.0_181-amd64
上述配置文件我已经上传至：https://github.com/PengShuaixin/hadoop-2.7.3_centos7
可以直接下载下来，通过上传到Linux直接覆盖原来文件的方式进行配置
关闭防火墙
CentOS 7 使用的是firewalld作为防火墙，与CentOS 6 有所不同
查看防火墙状态：
systemctl status firewalld如下状态说明正在运行：
[hadoop@master100 hadoop]$ systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
Active: active (running) since 三 2018-10-03 19:35:22 CST; 1h 49min ago
Docs: man:firewalld(1)
Main PID: 724 (firewalld)
CGroup: /system.slice/firewalld.service
└─724 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid
关闭防火墙：
systemctl stop firewalld关闭之后状态如下：
[hadoop@master100 hadoop]$ systemctl stop firewalld
==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===
Authentication is required to manage system services or units.
Authenticating as: root
Password: #这里需要输入root账户密码
==== AUTHENTICATION COMPLETE ===
[hadoop@master100 hadoop]$ systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
Active: inactive (dead) since 三 2018-10-03 21:27:34 CST; 40s ago
Docs: man:firewalld(1)
Process: 724 ExecStart=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS (code=exited, status=0/SUCCESS)
Main PID: 724 (code=exited, status=0/SUCCESS)
关闭防火墙开机自动启动：
systemctl disable firewalld执行过程如下：
[hadoop@master100 hadoop]$ systemctl disable firewalld
==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-unit-files ===
Authentication is required to manage system service or unit files.
Authenticating as: root
Password: #输入root用户密码
==== AUTHENTICATION COMPLETE ===
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
==== AUTHENTICATING FOR org.freedesktop.systemd1.reload-daemon ===
Authentication is required to reload the systemd state.
Authenticating as: root
Password: #输入root用户密码
==== AUTHENTICATION COMPLETE ===
#查看开机启动状态，确认关闭成功
[hadoop@master100 hadoop]$ systemctl is-enabled firewalld.service
disabled
更多firewalld防火墙操作请参考：
CentOS7使用firewalld打开关闭防火墙与端口
HDFS初始化
Hadoop配置完后，格式化namenode
[hadoop@master100 hadoop]$ hdfs namenode -format出现如下信息说明格式化成功：
[hadoop@master100 hadoop]$ hdfs namenode -format 18/10/03 21:05:52 INFO namenode.NameNode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG: host = master100/192.168.33.100 STARTUP_MSG: args = [-format] STARTUP_MSG: version = 2.7.3 STARTUP_MSG: classpath = /usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hadoop/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/usr/local/hadoop/share/hadoop/common/lib/stax-api-1.0-2.jar:/usr/local/hadoop/share/hadoop/common/lib/activation-1.1.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/usr/local/hadoop/share/hadoop/common/lib/jersey-server-1.9.jar:/usr/local/hadoop/share/hadoop/common/lib/asm-3.2.jar:/usr/local/hadoop/share/hadoop/common/lib/log4j-1.2.17.jar:/usr/local/hadoop/share/hadoop/common/lib/jets3t-0.9.0.jar:/usr/local/hadoop/share/hadoop/common/lib/httpclient-4.2.5.jar:/usr/local/hadoop/share/hadoop/common/lib/httpcore-4.2.5.jar:/usr/local/hadoop/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-lang-2.6.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-configuration-1.6.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-digester-1.8.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop/share/hadoop/common/lib/slf4j-api-1.7.10.jar:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar:/usr/local/hadoop/share/hadoop/common/lib/avro-1.7.4.jar:/usr/local/hadoop/share/hadoop/common/lib/paranamer-2.3.jar:/usr/local/hadoop/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-compress-1.4.1.jar:/usr/local/hadoop/share/hadoop/common/lib/xz-1.0.jar:/usr/local/hadoop/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/share/hadoop/common/lib/gson-2.2.4.jar:/usr/local/hadoop/share/hadoop/common/lib/hadoop-auth-2.7.3.jar:/usr/local/hadoop/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/local/hadoop/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/usr/local/hadoop/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/usr/local/hadoop/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/usr/local/hadoop/share/hadoop/common/lib/zookeeper-3.4.6.jar:/usr/local/hadoop/share/hadoop/common/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/share/hadoop/common/lib/curator-framework-2.7.1.jar:/usr/local/hadoop/share/hadoop/common/lib/curator-client-2.7.1.jar:/usr/local/hadoop/share/hadoop/common/lib/jsch-0.1.42.jar:/usr/local/hadoop/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/usr/local/hadoop/share/hadoop/common/lib/htrace-core-3.1.0-incubating.jar:/usr/local/hadoop/share/hadoop/common/lib/junit-4.11.jar:/usr/local/hadoop/share/hadoop/common/lib/hamcrest-core-1.3.jar:/usr/local/hadoop/share/hadoop/common/lib/mockito-all-1.8.5.jar:/usr/local/hadoop/share/hadoop/common/lib/hadoop-annotations-2.7.3.jar:/usr/local/hadoop/share/hadoop/common/lib/guava-11.0.2.jar:/usr/local/hadoop/share/hadoop/common/lib/jsr305-3.0.0.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-cli-1.2.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-math3-3.1.1.jar:/usr/local/hadoop/share/hadoop/common/lib/xmlenc-0.52.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-httpclient-3.1.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-logging-1.1.3.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-codec-1.4.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-io-2.4.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-net-3.1.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-collections-3.2.2.jar:/usr/local/hadoop/share/hadoop/common/lib/servlet-api-2.5.jar:/usr/local/hadoop/share/hadoop/common/lib/jetty-6.1.26.jar:/usr/local/hadoop/share/hadoop/common/lib/jetty-util-6.1.26.jar:/usr/local/hadoop/share/hadoop/common/lib/jsp-api-2.1.jar:/usr/local/hadoop/share/hadoop/common/lib/jersey-core-1.9.jar:/usr/local/hadoop/share/hadoop/common/lib/jersey-json-1.9.jar:/usr/local/hadoop/share/hadoop/common/lib/jettison-1.1.jar:/usr/local/hadoop/share/hadoop/common/hadoop-common-2.7.3.jar:/usr/local/hadoop/share/hadoop/common/hadoop-common-2.7.3-tests.jar:/usr/local/hadoop/share/hadoop/common/hadoop-nfs-2.7.3.jar:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/guava-11.0.2.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jsr305-3.0.0.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-io-2.4.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/asm-3.2.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/htrace-core-3.1.0-incubating.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/usr/local/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.7.3.jar:/usr/local/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.7.3-tests.jar:/usr/local/hadoop/share/hadoop/hdfs/hadoop-hdfs-nfs-2.7.3.jar:/usr/local/hadoop/share/hadoop/yarn/lib/zookeeper-3.4.6-tests.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-lang-2.6.jar:/usr/local/hadoop/share/hadoop/yarn/lib/guava-11.0.2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jsr305-3.0.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/usr/local/hadoop/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-cli-1.2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/log4j-1.2.17.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/activation-1.1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/xz-1.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/servlet-api-2.5.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-codec-1.4.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-core-1.9.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-client-1.9.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/usr/local/hadoop/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/guice-3.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/javax.inject-1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/aopalliance-1.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-io-2.4.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-server-1.9.jar:/usr/local/hadoop/share/hadoop/yarn/lib/asm-3.2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-json-1.9.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jettison-1.1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/usr/local/hadoop/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/usr/local/hadoop/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-collections-3.2.2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jetty-6.1.26.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-api-2.7.3.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.7.3.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-common-2.7.3.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.7.3.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.7.3.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.7.3.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.7.3.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-tests-2.7.3.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-client-2.7.3.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.7.3.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.3.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.7.3.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-registry-2.7.3.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/xz-1.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/hadoop-annotations-2.7.3.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/asm-3.2.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/guice-3.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/javax.inject-1.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/junit-4.11.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.7.3.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.7.3.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.7.3.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.7.3.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.7.3.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.3.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.3.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.3-tests.jar:/usr/local/hadoop/contrib/capacity-scheduler/*.jar STARTUP_MSG: build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r baa91f7c6bc9cb92be5982de4719c1c8af91ccff; compiled by 'root' on 2016-08-18T01:41Z STARTUP_MSG: java = 1.8.0_181 ************************************************************/ 18/10/03 21:05:52 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT] 18/10/03 21:05:52 INFO namenode.NameNode: createNameNode [-format] Formatting using clusterid: CID-54d602ae-8c60-49f4-bcb9-8a687ba97501 18/10/03 21:05:52 INFO namenode.FSNamesystem: No KeyProvider found. 18/10/03 21:05:52 INFO namenode.FSNamesystem: fsLock is fair:true 18/10/03 21:05:52 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000 18/10/03 21:05:52 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true 18/10/03 21:05:52 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000 18/10/03 21:05:52 INFO blockmanagement.BlockManager: The block deletion will start around 2018 十月 03 21:05:52 18/10/03 21:05:52 INFO util.GSet: Computing capacity for map BlocksMap 18/10/03 21:05:52 INFO util.GSet: VM type = 64-bit 18/10/03 21:05:52 INFO util.GSet: 2.0% max memory 966.7 MB = 19.3 MB 18/10/03 21:05:52 INFO util.GSet: capacity = 2^21 = 2097152 entries 18/10/03 21:05:52 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false 18/10/03 21:05:52 INFO blockmanagement.BlockManager: defaultReplication = 1 18/10/03 21:05:52 INFO blockmanagement.BlockManager: maxReplication = 512 18/10/03 21:05:52 INFO blockmanagement.BlockManager: minReplication = 1 18/10/03 21:05:52 INFO blockmanagement.BlockManager: maxReplicationStreams = 2 18/10/03 21:05:52 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000 18/10/03 21:05:52 INFO blockmanagement.BlockManager: encryptDataTransfer = false 18/10/03 21:05:52 INFO blockmanagement.BlockManager: maxNumBlocksToLog = 1000 18/10/03 21:05:52 INFO namenode.FSNamesystem: fsOwner = hadoop (auth:SIMPLE) 18/10/03 21:05:52 INFO namenode.FSNamesystem: supergroup = supergroup 18/10/03 21:05:52 INFO namenode.FSNamesystem: isPermissionEnabled = true 18/10/03 21:05:52 INFO namenode.FSNamesystem: HA Enabled: false 18/10/03 21:05:52 INFO namenode.FSNamesystem: Append Enabled: true 18/10/03 21:05:52 INFO util.GSet: Computing capacity for map INodeMap 18/10/03 21:05:52 INFO util.GSet: VM type = 64-bit 18/10/03 21:05:52 INFO util.GSet: 1.0% max memory 966.7 MB = 9.7 MB 18/10/03 21:05:52 INFO util.GSet: capacity = 2^20 = 1048576 entries 18/10/03 21:05:52 INFO namenode.FSDirectory: ACLs enabled? false 18/10/03 21:05:52 INFO namenode.FSDirectory: XAttrs enabled? true 18/10/03 21:05:52 INFO namenode.FSDirectory: Maximum size of an xattr: 16384 18/10/03 21:05:52 INFO namenode.NameNode: Caching file names occuring more than 10 times 18/10/03 21:05:53 INFO util.GSet: Computing capacity for map cachedBlocks 18/10/03 21:05:53 INFO util.GSet: VM type = 64-bit 18/10/03 21:05:53 INFO util.GSet: 0.25% max memory 966.7 MB = 2.4 MB 18/10/03 21:05:53 INFO util.GSet: capacity = 2^18 = 262144 entries 18/10/03 21:05:53 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033 18/10/03 21:05:53 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0 18/10/03 21:05:53 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000 18/10/03 21:05:53 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10 18/10/03 21:05:53 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10 18/10/03 21:05:53 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25 18/10/03 21:05:53 INFO namenode.FSNamesystem: Retry cache on namenode is enabled 18/10/03 21:05:53 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis 18/10/03 21:05:53 INFO util.GSet: Computing capacity for map NameNodeRetryCache 18/10/03 21:05:53 INFO util.GSet: VM type = 64-bit 18/10/03 21:05:53 INFO util.GSet: 0.029999999329447746% max memory 966.7 MB = 297.0 KB 18/10/03 21:05:53 INFO util.GSet: capacity = 2^15 = 32768 entries 18/10/03 21:05:53 INFO namenode.FSImage: Allocated new BlockPoolId: BP-938082284-192.168.33.100-1538571953311 18/10/03 21:05:53 INFO common.Storage: Storage directory /usr/local/hadoop/hadoopdata/namenode has been successfully formatted. 18/10/03 21:05:53 INFO namenode.FSImageFormatProtobuf: Saving image file /usr/local/hadoop/hadoopdata/namenode/current/fsimage.ckpt_0000000000000000000 using no compression 18/10/03 21:05:53 INFO namenode.FSImageFormatProtobuf: Image file /usr/local/hadoop/hadoopdata/namenode/current/fsimage.ckpt_0000000000000000000 of size 353 bytes saved in 0 seconds. 18/10/03 21:05:53 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0 18/10/03 21:05:53 INFO util.ExitUtil: Exiting with status 0 18/10/03 21:05:53 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at master100/192.168.33.100 ************************************************************/
启动Hadoop
执行命令：
start-all.sh
#或者是使用如下两条命令启动hadoop,第一条命令实际上操作的就是下面两条命令
start-dfs.sh
start-yarn.sh出现如下信息：
[hadoop@master100 hadoop]$ start-all.sh #执行上述命令会出现如下信息 This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh Starting namenodes on [localhost] localhost: starting namenode, logging to /usr/local/hadoop/logs/hadoop-hadoop-namenode-master100.out localhost: starting datanode, logging to /usr/local/hadoop/logs/hadoop-hadoop-datanode-master100.out Starting secondary namenodes [0.0.0.0] The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established. ECDSA key fingerprint is SHA256:smnjN2nAXE9l/kx1fQ2r3KBQt14TMjgaVlh+65clSrA. ECDSA key fingerprint is MD5:75:f5:c8:d0:06:c2:c8:0d:25:8b:b9:d5:47:8c:3a:f5. #到这里时注意输入yes后回车 Are you sure you want to continue connecting (yes/no)? yes 0.0.0.0: Warning: Permanently added '0.0.0.0' (ECDSA) to the list of known hosts. 0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-hadoop-secondarynamenode-master100.out starting yarn daemons starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-hadoop-resourcemanager-master100.out localhost: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-hadoop-nodemanager-master100.out
查看启动的进程：
jps出现如下进程说明Hadoop安装并启动成功
[hadoop@master100 hadoop]$ jps
6566 NameNode
7399 Jps
6664 DataNode
7016 ResourceManager
7114 NodeManager
6862 SecondaryNameNode
可以进入如下网页查看Namenode与ResourceManager
192.168.33.100:50070
192.168.33.100:8088
关闭Hadoop
[hadoop@master100 sbin]$ stop-all.sh
This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh
Stopping namenodes on [localhost]
localhost: stopping namenode
localhost: stopping datanode
Stopping secondary namenodes [0.0.0.0]
0.0.0.0: stopping secondarynamenode
stopping yarn daemons
stopping resourcemanager
localhost: stopping nodemanager
no proxyserver to stop
相关推荐
在Windows中安装Hadoop（非虚拟机安装）
CentOS 7部署Hadoop（单机版）
CentOS 7部署Hadoop集群（完全分布式）
CentOS 7部署Hadoop集群（HA高可用集群）