单节点部署Hadoop教程

单节点部署Hadoop教程

搭建HDFS

增加主机名

我这里仅仅增加了master主机名

[root@10 /xinghl/hadoop/bin]$ cat /etc/hosts
127.0.0.1   localhost 10.0.67.101
::1         localhost 10.0.67.101
10.0.67.101 master    10.0.67.101

如果配置远程集群，则需要配置SSH

我这里就是单节点测试玩玩，所以这步骤先省略了。

解压缩hadoop到/usr目录下

创建几个需要的目录

mkdir /dfs
mkdir /dfs/name
mkdir /dfs/data
mkdir /tmp

修改配置文件，在$HADOOP_HOME/etc/hadoop下

修改hadoop-env.sh

export JAVA_HOME=/usr/java

修改slaves

我这里就是

localhost

修改core-site.xml

       
                fs.defaultFS
                hdfs://master:8020
       
       
                io.file.buffer.size
                131072
        
       
               hadoop.tmp.dir
               file:/usr/hadoop/tmp
               Abase for other temporary   directories.
       
        
               hadoop.proxyuser.u0.hosts
               *
       
       
               hadoop.proxyuser.u0.groups
               *
       

修改hdfs-site.xml

       
                dfs.namenode.secondary.http-address
               master:9001
       
     
             dfs.namenode.name.dir
             file:/usr/hadoop/dfs/name
       
      
              dfs.datanode.data.dir
              file:/usr/hadoop/dfs/data
       
       
               dfs.replication
               3
        
        
                 dfs.webhdfs.enabled
                  true
         

启动hadoop

hadoop namenode -format #在$HADOOP_HOME/bin目录下
start-all.sh #在$HADOOP_HOME/sbin目录下

检查运行状态

1 使用jps命令查看
2 登录http://10.0.67.101:8088/cluster

配置SSH，支持无密码登录

[root@localhost sbin]# cd ~/.ssh/
[root@localhost .ssh]# ll
total 0
[root@localhost .ssh]# ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
75:ea:4f:ec:df:55:06:3c:af:91:12:50:69:a5:86:27 [email protected]
The key's randomart image is:
+--[ RSA 2048]----+
|          ...o.  |
|           ooo   |
|          E.* +  |
|         . * . = |
|        S . . o +|
|         . . . +.|
|          . o . .|
|           +   ..|
|            o.. .|
+-----------------+
[root@localhost .ssh]# 
[root@localhost .ssh]# ll
total 8
-rw-------. 1 root root 1675 Aug 16 10:55 id_rsa
-rw-r--r--. 1 root root  408 Aug 16 10:55 id_rsa.pub
[root@localhost .ssh]# cat id_rsa.pub >> ~/.ssh/authorized_keys
[root@localhost .ssh]# ll
total 12
-rw-r--r--. 1 root root  408 Aug 16 10:55 authorized_keys
-rw-------. 1 root root 1675 Aug 16 10:55 id_rsa
-rw-r--r--. 1 root root  408 Aug 16 10:55 id_rsa.pub
[root@localhost .ssh]# ssh localhost
The authenticity of host 'localhost (::1)' can't be established.
RSA key fingerprint is 9e:7b:cb:c5:06:e9:81:e5:db:57:9c:f9:79:a2:c4:ce.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
[root@localhost ~]# exit
logout
Connection to localhost closed.
[root@localhost .ssh]# ssh localhost
Last login: Tue Aug 16 10:55:30 2016 from localhost
[root@localhost ~]# exit
logout
Connection to localhost closed.
[root@localhost .ssh]# 

posted @ 2016-07-28 09:49 xingoo 阅读( ...) 评论( ...) 编辑 收藏