spark1.6+hadoop2.6+kafka2.10-0.8.2.1+zookeeper3.3.6安装及sparkStreaming代码编写和调试

安装环境
安装之前确保设备至少有4GB内存,推荐8GB
centos7.2
docker(这个安装请参考我的另一篇博客https://blog.csdn.net/qq_16563637/article/details/81699251)

目标安装软件	目标安装版本	实际安装版本
hadoop	2.6	2.6
spark	1.6	1.6
kafka	2.10-0.8.2.1	2.10-0.8.2.1
zookeeper	3.3.3	3.3.6

说明 因为kafka要配置HOST_NAME,HOST_NAME必须是宿主机IP地址,否则消费者程序会根据容器IP地址寻找kafka,所以kafka不能使用docker安装
本人亲自测试能够正常使用
-------------------------------------------开始安装spark和hadoop------------------------------------------------------
下载spark(说明这个说明:spark安装时会自动安装hadoop,不用再单独安装hadoop)

docker pull registry.docker-cn.com/sequenceiq/spark:1.6.0

docker运行

docker run -it -p 4040:4040 -p 7077:7077 -p 8088:8088 -p 8081:8081 -p 8080:8080 -p 8042:8042 -p 8030:8030 -p 8031:8031 -p 8040:8040 -p 9000:9000 -p 49707:49707 -p 50010:50010 -p 50070:50070 -p 50075:50075 -p 50020:50020 -p 50090:50090 --name spark --rm sequenceiq/spark:1.6.0 /bin/bash

设置spark
进入容器(docker run运行后会直接进入容器,该步骤可以省略)
docker exec -it 容器名 /bin/bash

docker exec -it 05d499dd260f /bin/bash
cd /usr/local/spark-1.6.0-bin-hadoop2.6
cd conf
cp spark-env.sh.template spark-env.sh
vi spark-env.sh

在最底部添加

export JAVA_HOME=/usr/java/jdk1.7.0_51
export SPARK_MASTER_PORT=7077

保存

cp slaves.template slaves
vi slaves

去掉localhost
添加192.168.1.103
保存

cd ../sbin
./stop-all.sh
./start-master.sh
./start-slave.sh 192.168.1.103:7077 --webui-port 8081 

为了正常使用配置环境变量,进入容器中设置

docker exec -it 05d499dd260f /bin/bash
vi /etc/profile
export SPARK_HOME=/usr/local/spark-1.6.0-bin-hadoop2.6
export HADOOP_HOME="/usr/local/hadoop-2.6.0"
export PATH=$PATH:$SPARK_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

保存:wq
使配置文件生效

source /etc/profile

测试spark是否安装OK
执行下面命令(该算法是利用蒙特·卡罗算法求PI)

/usr/local/spark-1.6.0-bin-hadoop2.6/bin/spark-submit \
--class org.apache.spark.examples.SparkPi \
--master spark://192.168.1.103:7077 \
--executor-memory 1G \
--total-executor-cores 2 \
/usr/local/spark-1.6.0-bin-hadoop2.6/lib/spark-examples-1.6.0-hadoop2.6.0.jar \
100

如果没有报错说明spark安装正常
打开spark-shell

spark-shell

出现 scala> 说明安装正常
开始检查hadoop是否正常
ctrl+C退出spark-shell
查看命令行是否能用

hadoop version

如果输出版本信息继续输入

cd /usr/local/hadoop-2.6.0
bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar grep input output 'dfs[a-z.]+'

如果输出mapreduce程序执行说明安装正常
查看spark 管控台
http://192.168.1.103:8080/
查看hadoop 管控台
http://192.168.1.103:50070
结束
------------------------------------------------zookeeper的docker安装------------------------------------------------------
说明:kafka_2.10-0.8.2.1推荐的zookeeper版本为3.3.3,然而zookeeper官方镜像没有3.3.3,选用3.3.6安装

docker pull zookeeper:3.3.6

启动zookeeper容器

docker run -d --name zookeeper -p 2181:2181 -t zookeeper:3.3.6

zookeeper安装完成
------------------------------------------------kafka本地安装--------------------------------------------------------------
先安装jdk(可参考我的另一篇博客https://blog.csdn.net/qq_16563637/article/details/81738113)
安装成功后下载kafka_2.10-0.8.2.1.tgz
kafka下载地址:https://archive.apache.org/dist/kafka/0.8.2.1/kafka_2.10-0.8.2.1.tgz
将文件上传至服务器解压缩

tar zxf kafka_2.10-0.8.2.1.tgz

修改配置文件

cd /root/kafka_2.10-0.8.2.1/config
vi server.properties

修改下面几项内容

host.name=192.168.1.103
log.dirs=/root/kafka_2.10-0.8.2.1/logs
zookeeper.connect=192.168.1.103:2181

保存

cd ..

启动kafka(前台启动)
建议先前台启动观看日志没有报错,ctrl+c退出,再后台启动

bin/kafka-server-start.sh  config/server.properties

启动kafka(后台启动)

bin/kafka-server-start.sh  config/server.properties > /dev/null 2>&1 &

创建topic(此处partitions数量不能大于broker,replication-factor 为副本数量)

bin/kafka-topics.sh --create --zookeeper 192.168.1.103:2181 --replication-factor 1 --partitions 1 --topic test

列出所有topic

bin/kafka-topics.sh --list --zookeeper 192.168.1.103:2181

向topic中写入数据

bin/kafka-console-producer.sh --broker-list 192.168.1.103:9092 --topic test

消费数据

bin/kafka-console-consumer.sh --zookeeper 192.168.1.103:2181 --topic test --from-beginning

查看指定topic的详情

bin/kafka-topics.sh --describe --zookeeper 192.168.1.103:2181 --topic test

停止kafka

kill -s TERM $(jps -l | grep 'kafka\.Kafka' | awk '{print $1}')

----------------------------------------------spark测试代码---------------------------------------------------------
说明:下面这段代码采用maven项目,请直接在本地启动运行,并且在Program arguments中写入下面内容

192.168.1.103:2181 g1 test 2

package cn.itcast.spark.day5

import org.apache.spark.storage.StorageLevel
import org.apache.spark.{HashPartitioner, SparkConf}
import org.apache.spark.streaming.kafka.KafkaUtils
import org.apache.spark.streaming.{Seconds, StreamingContext}

/**
  * Created by root on 2016/5/21.
  */
//(多个zookeeper用,隔开)
//zookeper groupid topics numThreads
//入参:192.168.1.103:2181 g1 test 2
object KafkaWordCount {

  val updateFunc = (iter: Iterator[(String, Seq[Int], Option[Int])]) => {
    //iter.flatMap(it=>Some(it._2.sum + it._3.getOrElse(0)).map(x=>(it._1,x)))
    iter.flatMap { case (x, y, z) => Some(y.sum + z.getOrElse(0)).map(i => (x, i)) }
  }


  def main(args: Array[String]) {
    LoggerLevels.setStreamingLogLevels()
    val Array(zkQuorum, group, topics, numThreads) = args
    val sparkConf = new SparkConf().setAppName("KafkaWordCount").setMaster("local[2]")
    val ssc = new StreamingContext(sparkConf, Seconds(5))
    ssc.checkpoint("c://ck2")
    //"alog-2016-04-16,alog-2016-04-17,alog-2016-04-18"
    //"Array((alog-2016-04-16, 2), (alog-2016-04-17, 2), (alog-2016-04-18, 2))"
    val topicMap = topics.split(",").map((_, numThreads.toInt)).toMap
    val data = KafkaUtils.createStream(ssc, zkQuorum, group, topicMap, StorageLevel.MEMORY_AND_DISK_SER)
    val words = data.map(_._2).flatMap(_.split(" "))
    val wordCounts = words.map((_, 1)).updateStateByKey(updateFunc, new HashPartitioner(ssc.sparkContext.defaultParallelism), true)
    wordCounts.print()
    ssc.start()
    ssc.awaitTermination()
  }
}

pom.xml如下

    4.0.0

    cn.itcast.spark
    hello-spark
    1.0

    
        1.7
        1.7
        UTF-8
        2.10.6
        1.6.1
        2.6.4
    

    
        
            org.scala-lang
            scala-library
            ${scala.version}
        

        
            org.apache.spark
            spark-core_2.10
            ${spark.version}
        

        
            org.apache.hadoop
            hadoop-client
            ${hadoop.version}
        

        
            org.apache.spark
            spark-sql_2.10
            ${spark.version}
        

        
            org.apache.spark
            spark-streaming_2.10
            ${spark.version}
        

        
            org.apache.spark
            spark-streaming-kafka_2.10
            1.6.1
        

        
            org.apache.spark
            spark-streaming-flume_2.10
            ${spark.version}
        

        
            mysql
            mysql-connector-java
            5.1.38
        

        
            redis.clients
            jedis
            2.8.1
        

    

    
        src/main/scala
        src/test/scala
        
            
                net.alchim31.maven
                scala-maven-plugin
                3.2.2
                
                    
                        
                            compile
                            testCompile
                        
                        
                            
                                -make:transitive
                                -dependencyfile
                                ${project.build.directory}/.scala_dependencies
                            
                        
                    
                
            

            
                org.apache.maven.plugins
                maven-shade-plugin
                2.4.3
                
                    
                        package
                        
                            shade
                        
                        
                            
                                
                                    *:*
                                    
                                        META-INF/*.SF
                                        META-INF/*.DSA
                                        META-INF/*.RSA