SQL On Streaming

实时计算的一个方向

实时计算未来会成为一个趋势，基本上所有的离线计算任务都能通过实时计算来完成，对于实时计算来算，除了性能，延迟性和吞吐量这些硬指标要求以外，我觉得易用性上面应该是未来的一个发展方向，毕竟现在的实时计算入storm，flink，sparkstreaming等都是通过API来进行的，这些使用起来都不太方便，后续更大的一个侧重方向应该是SQL ON STREAMING，对storm了解不是很多，但是有些公司已经针对storm进行了sql封装,下面只想谈下两个比较流行的开源流计算引擎对SQL的封装粒度。

Flink

SQL on Streaming Tables

code examples

val env = StreamExecutionEnvironment.getExecutionEnvironment
val tEnv = TableEnvironment.getTableEnvironment(env)// read a DataStream from an external source
val ds: DataStream[(Long, String, Integer)] = env.addSource(...)// register the DataStream under the name "Orders"
tableEnv.registerDataStream("Orders", ds, 'user, 'product, 'amount)// run a SQL query on the Table and retrieve the result as a new Table
val result = tableEnv.sql( "SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'")

限制

  1.2版本 只支持SELECT, FROM, WHERE, and UNION，不支持聚合，join操作，感觉离真正的使用还是有一段距离要走。

spark 2.0，structure streaming

code examples

import org.apache.spark.sql.functions._
import org.apache.spark.sql.SparkSession
val input = spark.readStream.text("file:///home/hadoop/data/")
val words = input.as[String].flatMap(_.split(" "))
val wordCounts = words.groupBy("value").count()
val query = wordCounts.writeStream.outputMode("complete").format("console").start
query.awaitTermination

限制

output mode只实现了两种，且有限制

• Append mode (default)
  This is the default mode, where only the new rows added to the result table since the last trigger will be outputted to the sink. This is only applicable to queries that do not have any aggregations (e.g. queries with only select, where, map, flatMap, filter,join, etc.).

• Complete mode
  The whole result table will be outputted to the sink.This is only applicable to queries that have aggregations

• 不支持update模式

从两种的限制来看，structure streaming更新的更加快些。