spark+phoenix读取hbase
正常来说这个内容应该网上可参考的文章很多，但是我还是捣鼓了好久，现在记录下来，给自己个备忘录。
phoenix是操作hbase的皮肤，他可以轻松的使用sql语句来操作hbase，比直接用hbase的原语操作要友好的多。spark直接操作hbase也是通过hbase的原语操作，操作起来比较繁琐，下面就是将spark和phoenix相结合的方法步骤。
我用的是scala语言，首先pom.xml中添加依赖
org.apache.phoenix
phoenix-spark
5.0.0-HBase-2.0
provided
org.apache.phoenix
phoenix-core
5.0.0-HBase-2.0
org.apache.hbase
hbase-client
2.4.12
org.apache.hbase
hbase-server
2.4.12
org.apache.hbase
hbase-common
2.4.12
这里添加的版本信息要和你要访问的hbase相一致！
接下来，到phoenix官网下载jar包，Overview | Apache Phoenix
然后解压缩，将里面的phoenix-server-hbase-2.4-5.1.3.jar（你的版本可能和我下载的不一致，这个根据hadoop上安装的hbase的版本来定）拷贝到hbase/lib/目录下，然后重启hbase。
然后将解压的phoenix-client-hbase-2.4-5.1.3.jar包拷贝到你的工程resources目录下，然后将hadoop中的配置文件也都放到resources/conf/这个目录下，接下来开始写代码。
import org.apache.spark.SparkContext
import org.apache.spark.sql.{SQLContext, SparkSession}
import org.apache.phoenix.spark.datasource.v2.PhoenixDataSource
val spark = SparkSession
.builder()
.appName("phoenix-test")
.master("local")
.getOrCreate()
// Load data from TABLE1
val df = spark.sqlContext
.read
.format("phoenix")
.options(Map("table" -> "TABLE1", PhoenixDataSource.ZOOKEEPER_URL -> "phoenix-server:2181"))
.load
df.filter(df("COL1") === "test_row_1" && df("ID") === 1L)
.select(df("ID"))
.show这是phoenix官网提供的代码，我执行没成功，显示org.apache.phoenix.spark.datasource.v2.PhoenixDataSource这个找不到，我不知道是我依赖包没引对还是其他原因，我的代码在上面的基础上做了一些改动。
import org.apache.spark.SparkConf
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.execution.datasources.jdbc._
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.Path
import org.apache.log4j.Logger
object SparkPhoenixHbase {
@transient lazy val log = Logger.getLogger(this.getClass)
def main(args: Array[String]): Unit = {
readFromHBaseWithPhoenix()
}
def readFromHBaseWithPhoenix(): Unit = {
val hadoopConf = new Configuration()
hadoopConf.addResource(new Path("conf/core-site.xml"))
hadoopConf.addResource(new Path("conf/hdfs-site.xml"))
hadoopConf.addResource(new Path("conf/mapred-site.xml"))
hadoopConf.addResource(new Path("conf/yarn-site.xml"))
hadoopConf.addResource(new Path("conf/hbase-site.xml"))
val conf = new SparkConf()
.setAppName("phoenix-spark-hdase")
.setMaster("local[*]")
conf.set("spark.driver.extraClassPath","/resources/phoenix-client-hbase-2.4-5.1.3.jar")
conf.set("spark.executor.extraClassPath","/resources/phoenix-client-hbase-2.4-5.1.3.jar")
val it = hadoopConf.iterator()
while (it.hasNext){
val entry = it.next()
conf.set(entry.getKey, entry.getValue)
}
val spark = SparkSession
.builder()
.master("local")
.appName("phoenix-hbase")
.config(conf)
.getOrCreate()
val phoenixConfig = Map(
"url" -> "jdbc:phoenix:10.12.4.51:2181", //这里是你hadoop上安装的zookeeper的地址
"driver" -> "org.apache.phoenix.jdbc.PhoenixDriver"
)
val df = spark.read
.format("jdbc")
.options(phoenixConfig)
.option("dbtable","student")
.load()
df.show()
spark.close()
}
}
最好要在工程里配置上日志打印，不然执行过程中的错误信息是看不到的。
最后执行成功的结果如下所示
2024-01-18 08:53:52,487 INFO [org.apache.spark.executor.Executor] : Finished task 0.0 in stage 0.0 (TID 0). 1509 bytes result sent to driver
2024-01-18 08:53:52,493 INFO [org.apache.spark.scheduler.TaskSetManager] : Finished task 0.0 in stage 0.0 (TID 0) in 580 ms on DESKTOP-FT30H9D (executor driver) (1/1)
2024-01-18 08:53:52,494 INFO [org.apache.spark.scheduler.TaskSchedulerImpl] : Removed TaskSet 0.0, whose tasks have all completed, from pool
2024-01-18 08:53:52,500 INFO [org.apache.spark.scheduler.DAGScheduler] : ResultStage 0 (show at SparkPhoenixHbase.scala:70) finished in 0.774 s
2024-01-18 08:53:52,502 INFO [org.apache.spark.scheduler.DAGScheduler] : Job 0 is finished. Cancelling potential speculative or zombie tasks for this job
2024-01-18 08:53:52,502 INFO [org.apache.spark.scheduler.TaskSchedulerImpl] : Killing all running tasks in stage 0: Stage finished
2024-01-18 08:53:52,504 INFO [org.apache.spark.scheduler.DAGScheduler] : Job 0 finished: show at SparkPhoenixHbase.scala:70, took 0.808840 s
2024-01-18 08:53:52,538 INFO [org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator] : Code generated in 14.3886 ms
+----+--------+---+-------+
| ID| NAME|AGE| ADDR|
+----+--------+---+-------+
|1001|zhangsan| 10|tianjin|
+----+--------+---+-------+
// 能看到这个就说明成功了，我的hbase student表里就这么一行信息
2024-01-18 08:53:52,555 INFO [org.sparkproject.jetty.server.AbstractConnector] : Stopped Spark@4108fa66{HTTP/1.1, (http/1.1)}{0.0.0.0:4040}
2024-01-18 08:53:52,556 INFO [org.apache.spark.ui.SparkUI] : Stopped Spark web UI at http://DESKTOP-FT30H9D:4040
2024-01-18 08:53:52,566 INFO [org.apache.spark.MapOutputTrackerMasterEndpoint] : MapOutputTrackerMasterEndpoint stopped!
2024-01-18 08:53:52,581 INFO [org.apache.spark.storage.memory.MemoryStore] : MemoryStore cleared
2024-01-18 08:53:52,581 INFO [org.apache.spark.storage.BlockManager] : BlockManager stopped
2024-01-18 08:53:52,587 INFO [org.apache.spark.storage.BlockManagerMaster] : BlockManagerMaster stopped
2024-01-18 08:53:52,589 INFO [org.apache.spark.scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint] : OutputCommitCoordinator stopped!
2024-01-18 08:53:52,595 INFO [org.apache.spark.SparkContext] : Successfully stopped SparkContext
2024-01-18 08:53:59,207 INFO [org.apache.spark.util.ShutdownHookManager] : Shutdown hook called
2024-01-18 08:53:59,207 INFO [org.apache.spark.util.ShutdownHookManager] : Deleting directory C:\Users\shell\AppData\Local\Temp\spark-344ef832-7438-47dd-9126-725e6c2d8af4