hadoop+spark配置笔记

vi  .bashrc

添加

export SCALA_HOME=/opt/scala/scala-2.11.7
export JAVA_HOME=/usr/lib/jvm/java
export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/jre/lib
export HADOOP_HOME=/opt/hadoop-2.6.0
export HADOOP_CONFIG_HOME=$HADOOP_HOME/etc/hadoop
export SPARK_HOME=/opt/spark-1.6.0-bin-hadoop2.6
export SBT_HOME=/opt/scala/scala-2.11.7/sbt
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:/opt/scala/scala-2.11.7/bin:$JAVA_HOME/jre/bin:$JAVA_HOME/bin:$SPARK_HOME/bin:$SBT_HOME/bin

#autorun
/usr/sbin/sshd

安装ssh     启动/etc/init.d/ssh start

下载jdk  scala hadoop spark spark并安装配置具体见

http://tashan10.com/yong-dockerda-jian-hadoopwei-fen-bu-shi-ji-qun/

http://blog.csdn.net/stark_summer/article/details/42458081

配置hadoop

core-site.xm

            hadoop.tmp.dir
            file:/opt/hadoop-2.6.0/tmp
            A base for other temporary directories.
   


   
            fs.default.name
            hdfs://master01:9000
            true


   

hadoop-env.sh 

export JAVA_HOME=/usr/lib/jvm/java

hdfs-site.xml 

        dfs.replication
        2
        true
   
 
        dfs.namenode.secondary.http-address
        master01:50090
   
   
        dfs.namenode.name.dir
        file:/opt/hadoop-2.6.0/namenode
        true
   
   
        dfs.datanode.data.dir
        file:/opt/hadoop-2.6.0/datanode
        true
   

 mapred-site.xml

    mapreduce.framework.name
    yarn

yarn-site.xml 

    yarn.resourcemanager.hostname
    master01


    yarn.nodemanager.aux-services
    mapreduce_shuffle

配置 yarn-env.sh （修改JAVA_HOME）

# some Java parameters

export JAVA_HOME=/usr/lib/jvm/java

slaves

slave01

slave02

slave03

然后把它弄成镜像发布

在开启

sudo docker commit -m "hadoop+weiyunxing" master01 shanyx/ubuntu:spark10

sudo docker run -it  -v /home/syx/ins:/mnt -p 50071:50070 -p 50076:50075 -p 4041:4040 -p 8081:8080 -p 8089:8088 -p 19889:19888 -p 8043:8042 -h master01 --name master01 shanyx/ubuntu:spark10 /bin/bash

格式化在hadoop

./bin/hdfs namenode -format

.sbin/start-all.sh

jps