hadoop部署

1.  安装jdk

2.  上传并解压hadoop

3.  配置免密登录

如果是集群模式,localhost替换为其他服务器的ip地址或主机名,所有的服务器之间均需要配置免密登陆

$ ssh-keygen

$ ssh-copy-id root@localhost

4.  配置${hadoop_home}/etc/hadoop/core-site.xml

hadoop.tmp.dir

file:/usr/local/soft/hadoop/tmp

Abase for othertemporary directories.

fs.defaultFS

hdfs://localhost:9000

fs.checkpoint.period

3600

The number of secondsbetween two periodic checkpoints.

fs.checkpoint.size        

67108864

5.  配置${hadoop_home}/etc/hadoop/hdfs-site.xml

dfs.replication

1

dfs.namenode.name.dir

file:/usr/local/soft/hadoop/tmp/dfs/name

dfs.datanode.data.dir

file:/usr/local/soft/hadoop/tmp/dfs/data

dfs.namenode.secondary.http-address

data2.ccsp.yota:50090

6.  配置${hadoop_home}/etc/hadoop/mapred-site.xml

       

               

               mapreduce.input.fileinputformat.input.dir.recursive

               true

       

       

                

               mapreduce.framework.name

               yarn

       

       

               

               mapred.job.tracker

                bigdata1.ccsp.jv:9001

       

       

               yarn.log.server.url

               http://172.16.7.30:19888/jobhistory/logs/

       

7.  配置${hadoop_home}/etc/hadoop/yarn-site.xml

 

       

               

               yarn.log-aggregation-enable

               true

               

       

       

               

               yarn.resourcemanager.hostname

               bigdata1.ccsp.jv

       

       

               

               yarn.nodemanager.aux-services

               mapreduce_shuffle

       

8.  配置${hadoop_home}/etc/hadoop/slaves

datanode的地址列表,每行一个

vim slaves

9.  格式化namenode

$ ./bin/hdfs namenode -format

10. 启动hadoop

最后一行是启动日志记录

$ ./sbin/start-dfs.sh

$ ./sbin/start-yarn.sh

$ ./sbin/ mr-jobhistory-daemon.sh starthistoryserver