hadoop伪分布式安装HDFS(CDH5.16.2版本)

安装步骤

1. root用户安装JDK
2. 创建目录：mkdir /usr/java,上传jdk-8u45-linux-x64.gz包到该目录
3. 解压 tar -zxvf jdk-8u45-linux-x64.gz
   
4. 修改用户和用户组：chown -R root:root jdk1.8.0_45
5. 建立软连接：ln -s jdk1.8.0_45 jdk
6. 修改环境变量：vi /etc/profile

export JAVA_HOME=/usr/java/jdk
export PATH=$JAVA_HOME/bin:$PATH

7. 创建用户：useradd ruoze,切换目录su - ruoze
8. 创建目录：mkdir app software sourcecode log tmp data lib

drwxrwxr-x 2 ruoze ruoze 6 Nov 27 21:32 app         解压的文件夹,软连接
drwxrwxr-x 2 ruoze ruoze 6 Nov 27 21:32 data        数据
drwxrwxr-x 2 ruoze ruoze 6 Nov 27 21:32 lib         第三方的jar
drwxrwxr-x 2 ruoze ruoze 6 Nov 27 21:32 log         日志文件夹
drwxrwxr-x 2 ruoze ruoze 6 Nov 27 21:32 software    压缩包
drwxrwxr-x 2 ruoze ruoze 6 Nov 27 21:32 sourcecode  源代码编译
drwxrwxr-x 2 ruoze ruoze 6 Nov 27 21:32 tmp         临时文件夹 ？？？/tmp

9. 上传文件到software目录,
10. 解压文件到app目录：tar -zxvf hadoop-2.6.0-cdh5.16.2.tar.gz -C /home/ruoze/app/
11. 进入app目录做软连接：ln -s hadoop-2.6.0-cdh5.16.2 hadoop
12. 配置hadoop的JAVA_HOME显性配置:vi /home/ruoze/app/hadoop/etc/hadoop/hadoop-env.sh

# The java implementation to use.
export JAVA_HOME=/usr/java/jdk

13. 配置文件

vi /home/ruoze/app/hadoop/etc/hadoop/core-site.xml
<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://ruozedata001:9000</value>
    </property>
</configuration>
vi /home/ruoze/app/hadoop/etc/hadoop/hdfs-site.xml
<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
    <!--配置snn-->
    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>ruozedata001:50090</value>
    </property>
    <property>
        <name>dfs.namenode.secondary.https-address</name>
        <value>ruozedata001:50091</value>
    </property>
</configuration>

14. ssh无密码信任关系

[ruoze@ruozedata001 ~]$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/ruoze/.ssh/id_rsa): 
Created directory '/home/ruoze/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/ruoze/.ssh/id_rsa.
Your public key has been saved in /home/ruoze/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:pBLxyxiDd/ou7f5LQjTsc81VTaqPbVuoboCqCZMcq+k ruoze@ruozedata001
The key's randomart image is:
+---[RSA 2048]----+
|    .         .o.|
|   . +       . ..|
|  . = * .   . .  |
|   . @ = o . .   |
|  . + B S.o .    |
| . + + o. .  + . |
|  *  .o..  .. = .|
| o o.ooo    .o o |
|+E  o=+.o. oo .  |
+----[SHA256]-----+
[ruoze@ruozedata001 ~]$ ls -a
.  ..  app  .bash_history  .bash_logout  .bash_profile  .bashrc  data  lib  log  software  sourcecode  .ssh  tmp
[ruoze@ruozedata001 ~]$ cd .ssh
[ruoze@ruozedata001 .ssh]$ ls
id_rsa  id_rsa.pub
[ruoze@ruozedata001 .ssh]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
[ruoze@ruozedata001 .ssh]$ ll
total 12
-rw-rw-r-- 1 ruoze ruoze  400 Dec  1 20:44 authorized_keys
-rw------- 1 ruoze ruoze 1675 Dec  1 20:43 id_rsa
-rw-r--r-- 1 ruoze ruoze  400 Dec  1 20:43 id_rsa.pub
[ruoze@ruozedata001 .ssh]$ chmod 600 authorized_keys 
[ruoze@ruozedata001 .ssh]$ ll
total 12
-rw------- 1 ruoze ruoze  400 Dec  1 20:44 authorized_keys
-rw------- 1 ruoze ruoze 1675 Dec  1 20:43 id_rsa
-rw-r--r-- 1 ruoze ruoze  400 Dec  1 20:43 id_rsa.pub
[ruoze@ruozedata001 .ssh]$ ssh ruozedata001 date
The authenticity of host 'ruozedata001 (192.168.0.3)' can't be established.
ECDSA key fingerprint is SHA256:OLqoaMxlGFbCq4sC9pYgF+FdbcXHbEbtSrnMiGGFbVw.
ECDSA key fingerprint is MD5:d3:5b:4a:ef:8e:00:41:a0:5e:80:ef:75:76:8a:a3:49.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'ruozedata001,192.168.0.3' (ECDSA) to the list of known hosts.
Sun Dec  1 20:53:59 CST 2019
[ruoze@ruozedata001 .ssh]$ ssh ruozedata001 date
Sun Dec  1 20:54:23 CST 2019
[ruoze@ruozedata001 .ssh]$ ls
authorized_keys  id_rsa  id_rsa.pub  known_hosts
[ruoze@ruozedata001 .ssh]$ cat known_hosts 
ruozedata001,192.168.0.3 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG9N5IGRTqwqGGHZcyNJ2i7lG54isK19GMq+Zw3VDIr64dS2sqoZ79n+8Ibz8ZJsU1aNiaJJTzYUvuxZv5W4iHQ=

当ssh登录到某台机器的时候，会把相应的ip和主机名追加到known_hosts中

15.配置环境变量vi .bashrc

# .bashrc

# Source global definitions
if [ -f /etc/bashrc ]; then
        . /etc/bashrc
fi

# Uncomment the following line if you don't like systemctl's auto-paging feature:
# export SYSTEMD_PAGER=

# User specific aliases and functions
export HADOOP_HOME=/home/ruoze/app/hadoop
export PATH=${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin:$PATH
export HADOOP_CONF_DIR=/home/ruoze/app/hadoop/etc/hadoop

1. source .bashrc
2. which hadoop
3. 格式化:hdfs namenode -format
4. 第一次启动：start-dfs.sh

start-dfs.sh
19/12/01 21:06:46 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [ruozedata001]
ruozedata001: starting namenode, logging to /home/ruoze/app/hadoop-2.6.0-cdh5.16.2/logs/hadoop-ruoze-namenode-ruozedata001.out
The authenticity of host 'localhost (::1)' can't be established.
ECDSA key fingerprint is SHA256:OLqoaMxlGFbCq4sC9pYgF+FdbcXHbEbtSrnMiGGFbVw.
ECDSA key fingerprint is MD5:d3:5b:4a:ef:8e:00:41:a0:5e:80:ef:75:76:8a:a3:49.
Are you sure you want to continue connecting (yes/no)? yes
localhost: Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
localhost: starting datanode, logging to /home/ruoze/app/hadoop-2.6.0-cdh5.16.2/logs/hadoop-ruoze-datanode-ruozedata001.out
Starting secondary namenodes [0.0.0.0]
The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.
ECDSA key fingerprint is SHA256:OLqoaMxlGFbCq4sC9pYgF+FdbcXHbEbtSrnMiGGFbVw.
ECDSA key fingerprint is MD5:d3:5b:4a:ef:8e:00:41:a0:5e:80:ef:75:76:8a:a3:49.
Are you sure you want to continue connecting (yes/no)? yes
0.0.0.0: Warning: Permanently added '0.0.0.0' (ECDSA) to the list of known hosts.
0.0.0.0: starting secondarynamenode, logging to /home/ruoze/app/hadoop-2.6.0-cdh5.16.2/logs/hadoop-ruoze-secondarynamenode-ruozedata001.out
19/12/01 21:07:05 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[ruoze@ruozedata001 ~]$ jps
14129 NameNode
14613 Jps
14253 DataNode
14414 SecondaryNameNode

16. namenode:ruozedata001,fs.defaultFS控制的;datanode:slaves控制的，secondarynamenode:dfs.namenode.secondary.http-address,dfs.namenode.secondary.https-address控制的
    
17. namenode:主节点，名称节点（老大），读写请求先经过它；datanode:从节点，数据节点（小弟），存储数据和检索数据；secondarynamenode:第二名称节点（老二）