Hadoop--hdfs单机安装文档

1.下载安装包

wget http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.7.0.tar.gz或者rz 上传本地包到linux系统

root@Hadoop001 ~]# ll
-rw-r--r--  1 root root 311585484 Jul  1  2019 hadoop-2.6.0-cdh5.7.0.tar.gz

2.创建专门维护的用户:hadoop

[root@Hadoop001 ~]# useradd hadoop
[root@Hadoop001 ~]# id hadoop
uid=515(hadoop) gid=515(hadoop) groups=515(hadoop)
[root@Hadoop001 ~]# su - hadoop
[hadoop@Hadoop001 ~]$ pwd
/home/hadoop

3.jdk部署(这里就不演示了)

[hadoop@Hadoop001 ~]$ which java
/usr/java/jdk1.8.0_45/bin/java

4.规划目录

[hadoop@Hadoop001 ~]$ mkdir app data logs software
[hadoop@Hadoop001 ~]$ ll
total 16
drwxrwxr-x 2 hadoop hadoop 4096 Jun 30 14:52 app
drwxrwxr-x 2 hadoop hadoop 4096 Jun 30 14:52 data
drwxrwxr-x 2 hadoop hadoop 4096 Jun 30 14:52 logs
drwxrwxr-x 2 hadoop hadoop 4096 Jun 30 14:52 software

5.解压安装包

[root@Hadoop001 ~]# mv hadoop-2.6.0-cdh5.7.0.tar.gz /home/hadoop/software/
[hadoop@Hadoop001 software]$ ll
total 304284
-rw-r--r-- 1 root root 311585484 Jul  1  2019 hadoop-2.6.0-cdh5.7.0.tar.gz
[root@Hadoop001 software]# chown hadoop:hadoop hadoop-2.6.0-cdh5.7.0.tar.gz
[hadoop@Hadoop001 software]$ tar -xzvf hadoop-2.6.0-cdh5.7.0.tar.gz 
[hadoop@Hadoop001 app]$ ln -s /home/hadoop/software/hadoop-2.6.0-cdh5.7.0 /home/hadoop/app/hadoop

6.配置hadoop

[hadoop@Hadoop001 hadoop]$ pwd
/home/hadoop/app/hadoop/etc/hadoop
[hadoop@Hadoop001 hadoop]$ vi hadoop-env.sh
export JAVA_HOME=/usr/java/jdk1.8.0_45
export HADOOP_CONF_DIR=/home/hadoop/app/hadoop/etc/hadoop
[hadoop@Hadoop001 hadoop]$ vi core-site.xml 

    
        fs.defaultFS
        hdfs://localhost:9000
    

[hadoop@Hadoop001 hadoop]$ vi hdfs-site.xml 

    
        dfs.replication
        1
    

7.配置无密码登入本机

[hadoop@Hadoop001 ~]$ ssh-keygen 
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa): 
Created directory '/home/hadoop/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
24:75:46:0a:ca:00:01:5c:a3:0a:73:06:89:b5:96:f8 hadoop@Hadoop001
The key's randomart image is:
+--[ RSA 2048]----+
|O=+o  . ..+      |
|o+.=.. o +       |
|+.* o . o        |
|o*     o         |
|. E     S        |
|                 |
|                 |
|                 |
|                 |
+-----------------+
[hadoop@Hadoop001 .ssh]$ cat id_rsa.pub >> authorized_keys
[hadoop@Hadoop001 .ssh]$ ll
total 12
-rw-rw-r-- 1 hadoop hadoop  398 Jul  1 09:10 authorized_keys
-rw------- 1 hadoop hadoop 1675 Jul  1 09:09 id_rsa
-rw-r--r-- 1 hadoop hadoop  398 Jul  1 09:09 id_rsa.pub
必须要修改authorized_keys文件权限
[root@Hadoop001 .ssh]# chmod 600 authorized_keys
加date可以直接返回结果，而不做机器切换
[hadoop@Hadoop001 .ssh]$ ssh localhost date
The authenticity of host 'localhost (::1)' can't be established.
RSA key fingerprint is db:f0:34:e6:e1:90:a5:84:25:18:99:2e:00:cb:e5:88.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
Mon Jul  1 09:13:05 CST 2019

8.启动hdfs

第一次先要格式化namenode
[hadoop@Hadoop001 hadoop]$ bin/hdfs namenode -format
[hadoop@Hadoop001 hadoop]$ sbin/start-dfs.sh 

9.检查

在window系统访问ip:50070地址。

2019-07-02_145159.png

10.验证

[hadoop@Hadoop001 hadoop]$ bin/hdfs dfs -ls /
19/07/02 09:01:14 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@Hadoop001 hadoop]$ bin/hdfs dfs -mkdir /user
19/07/02 09:02:19 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@Hadoop001 hadoop]$ bin/hdfs dfs -mkdir /user/hadoop
19/07/02 09:04:30 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@Hadoop001 hadoop]$ bin/hdfs dfs -put etc/hadoop input
19/07/02 09:04:38 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar grep input output 'dfs[a-z.]+'
[hadoop@Hadoop001 hadoop]$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar grep input output 'dfs[a-z.]+'
[hadoop@Hadoop001 hadoop]$ bin/hdfs dfs -cat /user/hadoop/output/*
19/07/02 09:07:34 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
6       dfs.audit.logger
4       dfs.class
3       dfs.server.namenode.
2       dfs.period
2       dfs.audit.log.maxfilesize
2       dfs.audit.log.maxbackupindex
1       dfsmetrics.log
1       dfsadmin
1       dfs.servers
1       dfs.replication
1       dfs.file

11.停止hdfs

[hadoop@Hadoop001 hadoop]$ sbin/stop-dfs.sh