docker安装 spark1.4 hadoop2.6

docker安装 spark1.4.md

adoop 2.6.0 and Apache Spark v1.4.0 on Centos

###一、 docker简介

docker新手入门30分钟

http://help.daocloud.io/tutorials/index.html

###二、docker安装

参考http://get.daocloud.io/

在linux上 安装docker

在windows、mac上安装boot2docker

boot2docker ssh

查看镜像   docker images

查看的容器 docker ps -a 

docker stop Name/ID 

docker start Name/ID  

 

###三、spark安装

 

1. pull镜像

docker pull sequenceiq/spark:1.4.0

下载比较慢，可以参考我的网盘

1. 运行docker

```

交互式运行

docker run --name spark01 -it -p 8088:8088 -p 8042:8042 -h sandbox sequenceiq/spark:1.4.0 bash

或后台运行

 

docker run --name spark01 -d -h sandbox sequenceiq/spark:1.4.0

```

spark01 ~spark06启动六个node

###四、 hadoop测试

boot2docker ip 查看ip 加入host文件 hostname 为 sandbox

Spark web UI at http:// sandbox :4040

http://192.168.59.103:8088/cluster

```

cd $HADOOP_PREFIX

# 运行 mapreduce

bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar grep input output 'dfs[a-z.]+'

 

# 查看输出

bin/hdfs dfs -cat output/*

```

###五、 spark测试

交互模式下

1 YARN-client模式

 

```

# run the spark shell

spark-shell \

--master yarn-client \

--driver-memory 1g \

--executor-memory 1g \

--executor-cores 1

 

# execute the the following command which should return 1000

scala> sc.parallelize(1 to 1000).count()

scala>exit

```

2 YARN集群模式

 

```

park-submit \

--class org.apache.spark.examples.SparkPi \

--files $SPARK_HOME/conf/metrics.properties \

--master yarn-cluster \

--driver-memory 1g \

--executor-memory 1g \

--executor-cores 1 \

$SPARK_HOME/lib/spark-examples-1.4.0-hadoop2.6.0.jar

```

3 pi测试

```

运行测试样例 输出 "Pi is roughly 3.1418" 

spark-submit \

--class org.apache.spark.examples.SparkPi \

--master yarn-client \

--driver-memory 1g \

--executor-memory 1g \

--executor-cores 1 \

$SPARK_HOME/lib/spark-examples-1.4.0-hadoop2.6.0.jar

```

来自为知笔记(Wiz)