kafka maven 依赖_pyspark + kafka 环境搭建

环境: win7 + Anaconda3 + pyspark 2.3.1 + kafka 1.0.1

一,win7安装kafka(单机)

1.下载压缩包

Index of /dist/kafka/1.0.1​archive.apache.org

2.解压并进入 KAFKA_HOME/bin/windows

3,通过CMD启动zookeeper 和 kafka

启动zookeeper:

zookeeper-server-start.bat 你的kafka根目录configzookeeper.properties (启动成功后不要关掉cmd对话框)

启动kafka:

kafka-server-start.bat 你的kafka根目录configserver.properties(启动成功后不要关掉cmd对话框)

5.测试

创建topic test:

你的kafka根目录binwindowskafka-topics.bat --zookeeper localhost:2181 --create --topic test --partitions 1 --replication-factor 1

控制台生产数据:

你的kafka根目录binwindowskafka-console-producer.bat --broker-list localhost:9092 --topic test

控制台消费数据:

你的kafka根目录binwindowskafka-console-consumer.bat --zookeeper localhost:2181 --topic test --from-beginning

kafka搭建成功!!!

二,安装pyspark

通过 anaconda安装

1.打开cmd并切换Python空间

activate tensorflow (你的python空间)

2.查看pyspark安装渠道:

anaconda search -t conda pyspark

3.查看安装命令

anaconda show conda-forge/pyspark

4.安装pyspark

等号后面接你要安装的版本号

conda install --channel Package repository for conda-forge pyspark=2.3.1

5.测试 cmd -> python -> import pyspark

6.下载spark依赖

spark-2.3.1-bin-hadoop2.7.rar

莫名原因官网无法下载，下面是网盘链接

链接：https://pan.baidu.com/s/1Jtt8M_p3gYIBbwZNHZsfMw

提取码：mx4c

--

下载之后解压，pycharm运行时需要指定 spark-2.3.1-bin-hadoop2.7 路径

7.测试

from 

如果报错 : FileNotFoundError: [WinError 2] 系统找不到指定的文件。

检查是否spark依赖添加成功

检查pyspark 和spark依赖的版本是否对应。

三.整和pyspark 和 kafka

1.下载spark kafka整合包

spark-streaming-kafka-0-8-assembly, Version = 2.3.1

spark-streaming-kafka-0-8，Version = 2.3.1

在以下地址分别 search spark-streaming-kafka-0-8-assembly 和spark-streaming-kafka-0-8，下载jar包，切换版本参照下方图片。

Maven Central Repository Search​search.maven.org

下载好两个jar包放到 spark-2.3.1-bin-hadoop2.7(二-6的目录下面)/jars目录下

如果没有这两个jar包会出现以下错误。

Spark Streaming's Kafka libraries not found in class path. Try one of the following.
1. Include the Kafka library and its dependencies with in the
spark-submit command as
$ bin/spark-submit --packages org.apache.spark:spark-streaming-kafka-0-8:2.3.1 ...
2. Download the JAR of the artifact from Maven Central Maven Central Repository Search,
Group Id = org.apache.spark, Artifact Id = spark-streaming-kafka-0-8-assembly, Version = 2.3.1.
Then, include the jar in the spark-submit command as
$ bin/spark-submit --jars ...

四.使用pyspark消费kafka

from 

ERROR 1.

2020-10-22 14:36:46 ERROR Executor:91 - Exception in task 0.0 in stage 1.0 (TID 1)
org.apache.spark.api.python.PythonException: Traceback (most recent call last):
File " E:Anaconda3installtensorflowenvstensorflowlibsite-packagespysparkrdd.py", line 1371, in takeUpToNumLeft
yield next(iterator)
StopIteration
The above exception was the direct cause of the following exception:

分析 :

使用python3.7版本迭代会出现一些问题

解决:

编辑 E:Anaconda3installtensorflowenvstensorflowlibsite-packagespysparkrdd.py 1392行

将

yield next(iterator)

修改为

try:
yield next(iterator)
except StopIteration:
return

问题解决!!!