SparkStreaming可视化之Wisp
-----------------------------------目录---------------------------------------
1、Wisp基础
2、Highchart函数
3、SparkStream和Wisp结合实验
-------------------------------------------------------------------------------
1、Wisp基础
Wisp的gitHub地址：
https://github.com/quantifind/wispWisp的sbt路径依赖
"com.quantifind" %% "wisp" % "0.0.4"Wisp的maven依赖
com.quantifind
wisp_2.10
0.0.4
注意:Wisp的开发已经在一年多前就停止了开发了，但是因为能满足一些必要的功能，而且可以把Spark处理后的数据 进行Web端的显示，所以不失为一种数据展示的一种手段。
1、wisp的初次见面
第一步：注入依赖，我的sbt的build.sbt如下：
name := "hello"
organization := "blog.csdn.net\legotime"
version := "0.0.1-SNAPSHOT"
scalaVersion := "2.10.4"
libraryDependencies ++= Seq("com.quantifind" %% "wisp" % "0.0.4")运行如下：
import com.quantifind.charts.Highcharts._
line((1 to 10), (1 to 10))第三步：web端就可以直接显示刚刚我们画的图
不仅仅sbt，而且可以直接在IDEA上注入maven依赖之后，可以在IDEA上操作，数据展示依然在web端。
可以说 Wisp在数据展示也不比Zeppelin差
更多初步操作可以访问：
https://github.com/quantifind/wisp/wiki/Blog
case class Highchart(val series : scala.collection.Traversable[com.quantifind.charts.highcharts.Series],
val title : scala.Option[com.quantifind.charts.highcharts.Title] = { /* compiled code */ },val chart : scala.Option[com.quantifind.charts.highcharts.Chart] = { /* compiled code */ },
val colors : scala.Option[scala.Array[com.quantifind.charts.highcharts.Color.Type]] = { /* compiled code */ },
val credits : scala.Option[com.quantifind.charts.highcharts.Credits] = { /* compiled code */ },
val exporting : scala.Option[com.quantifind.charts.highcharts.Exporting] = { /* compiled code */ },
val legend : scala.Option[com.quantifind.charts.highcharts.Legend] = { /* compiled code */ },
val plotOptions : scala.Option[com.quantifind.charts.highcharts.PlotOptions] = { /* compiled code */ },
val subtitle : scala.Option[com.quantifind.charts.highcharts.Title] = { /* compiled code */ },
val setTurboThreshold : scala.Boolean = { /* compiled code */ },
val tooltip : scala.Option[com.quantifind.charts.highcharts.ToolTip] = { /* compiled code */ },
val xAxis : scala.Option[scala.Array[com.quantifind.charts.highcharts.Axis]] = { /* compiled code */ },
val yAxis : scala.Option[scala.Array[com.quantifind.charts.highcharts.Axis]] = { /* compiled code */ }) extends scala.AnyRef with scala.Product with scala.Serializable {
第一部分：series
case class Series(val data : scala.collection.Traversable[com.quantifind.charts.highcharts.Data[_, _]],
val index : scala.Option[scala.Int] = { /* compiled code */ },
val legendIndex : scala.Option[scala.Int] = { /* compiled code */ },
val name : scala.Option[scala.Predef.String] = { /* compiled code */ },
val chart : scala.Option[com.quantifind.charts.highcharts.SeriesType.Type] = { /* compiled code */ },
val visible : scala.Option[scala.Boolean] = { /* compiled code */ },
val color : scala.Option[com.quantifind.charts.highcharts.Color.Type] = { /* compiled code */ },
val xAxis : scala.Option[scala.Predef.String] = { /* compiled code */ },
val yAxis : scala.Option[scala.Predef.String] = { /* compiled code */ },
val __name : scala.Predef.String = { /* compiled code */ }) extends com.quantifind.charts.highcharts.HighchartKey with scala.Product with scala.Serializable {
9个参数：
data:她是一个具有traversable属性的数据集合，内部形式是Data[x,y],其中 x是在坐标的位置，y的数值大小
index:目前没有看出有什么作用
legendIndex:目前没有看出有什么作用
name:这张表的名字和功能（底部和数据内都可见）
chart：使用那个表（(area, areaspline, bar, column, line, pie, scatter（上面一个一个的点）, spline)）
note:pie内其他标记名字不一样
visible：在界面可以是否可以观看到
color："#"+Integer.toHexString(color.getRGB()).substring(2)发现就几种颜色
xAxis:
yAxis:不知道怎么设置
__name：设置没图像：Ensures the object can be cast to Map[String, Any] so we can perform json serialization(_name = "Series",才有图像)
第二部：title
case class Title(
text: String = "", // Override default "Chart title"
align: Option[Alignment.Type] = None,
floating: Option[Boolean] = None,
style: Option[CSSObject] = None,
useHTML: Option[Boolean] = None,
verticalAlign: Option[VerticalAlignment.Type] = None,
x: Option[Int] = None,
y: Option[Int] = None,
var __name: String = "title"
) extends HighchartKey(__name) {
def toServiceFormat =
Map("text" -> text) ++
Map(
"align" -> align,
"floating" -> floating,
"useHTML" -> useHTML,
"verticalAlign" -> verticalAlign,
"x" -> x,
"y" -> y
).flatMap(HighchartKey.flatten) ++
HighchartKey.someStyleToServiceFormat(style)
}
参数分析：
text：title = "hello"：设置图像的title为 "hello" 不设置 是没有
align: 是判断在（左，中，右）("left", "center", "right") 不设置是中
floating：true是在图内部的上面，false 是在图的外部上面
style: Option[CSSObject] = None,
useHTML：没效果
verticalAlign：("top", "middle", "bottom") title竖直位置 默认是 top
x
y: x，y是自己设定位置
var __name: String = "title"（现在默认是title,恐怕以后还有其他功能）
第三部：Chart
zoomType: Option[Zoom.Type] = Some(Zoom.xy) 默认情况下是 Zoom.xy,也就是上下都放大
第四部分：colors
object Color {
type Type = String
implicit def javaColorToHex(color: java.awt.Color): Type = "#"+Integer.toHexString(color.getRGB()).substring(2)
implicit def javaColorToHexOption(color: java.awt.Color): Option[Type] = Some("#"+Integer.toHexString(color.getRGB()).substring(2))
}
通过设置java里面的颜色来进行画线，但是必须保证前面是没有设置颜色的，如果设置，那么就是取前面设置的颜色
Color.javaColorToHex(java.awt.Color.PINK) //在這里设置颜色的不好之处在于pie全部颜色为一个颜色
第五部分：credits
case class Credits(
enabled: Option[Boolean] = None,
href: String = "", // Override default Highcharts
position: Option[Position] = None,
style: Option[CSSObject] = None,
text: String = "" // Override default Highcharts
) extends HighchartKey("credits")
Credits(enabled = true,href = "good",position = Position(x = 100,y = 100),style = CSSObject(color = Color.javaColorToHex(java.awt.Color.BLUE),fontWeight = "lighter"),text = "what the ")
第六部分：exporting（保存图像配置信息）
case class Exporting(
//buttons
//chartOptions
filename: String = "chart", //保存时候的文件名字
scale: Option[Int] = None,
sourceHeight: Option[Int] = None,
sourceWidth: Option[Int] = None,
_type: Option[String] = None,
url: Option[String] = None,
width: Option[Int] = None
) extends HighchartKey("exporting") {/**/}
第七部分：legend
// No more than 22 members in a case class TODO
case class Legend(
align: Option[Alignment.Type] = None,
backgroundColor: Option[Color.Type] = None,
borderColor: Option[Color.Type] = None,
// borderRadius: Int = 5,
// borderWidth: Int = 2,
enabled: Option[Boolean] = Some(false), // override default
floating: Option[Boolean] = None,
itemDistance: Option[Int] = None,
//itemHiddenStyle
//itemHoverStyle
itemMarginBottom: Option[Int] = None,
itemMarginTop: Option[Int] = None,
//itemStyle
itemWidth: Option[Int] = None,
labelFormat: Option[String] = None, // TODO - format string helpers
//labelFormatter
layout: Option[Layout.Type] = None,
margin: Option[Int] = None,
maxHeight: Option[Int] = None,
//navigation
padding: Option[Int] = None,
reversed: Option[Boolean] = None,
rtl: Option[Boolean] = None, // right-to-left
//shadow
//style
// symbolPadding: Int = 5,
// symbolWidth: Int = 30,
title: Option[String] = None, // todo - css title
// useHTML: Boolean = false,
verticalAlign: Option[VerticalAlignment.Type] = None,
width: Option[Int] = None,
x: Option[Int] = None,
y: Option[Int] = None
) extends HighchartKey("legend") {/**/}
第八部分：plotOptions
// These are overlapping options, but are well over 22 options.
case class PlotOptions(
area: Option[PlotOptionKey] = None,
arearange: Option[PlotOptionKey] = None,
areaspline: Option[PlotOptionKey] = None,
areasplinerange: Option[PlotOptionKey] = None,
bar: Option[PlotOptionKey] = None,
boxplot: Option[PlotOptionKey] = None,
bubble: Option[PlotOptionKey] = None,
column: Option[PlotOptionKey] = None,
columnrange: Option[PlotOptionKey] = None,
errorbar: Option[PlotOptionKey] = None,
funnel: Option[PlotOptionKey] = None,
gauge: Option[PlotOptionKey] = None,
heatmap: Option[PlotOptionKey] = None,
line: Option[PlotOptionKey] = None,
pie: Option[PlotOptionKey] = None,
pyramid: Option[PlotOptionKey] = None,
scatter: Option[PlotOptionKey] = None,
series: Option[PlotOptionKey] = None,
solidgauge: Option[PlotOptionKey] = None,
spline: Option[PlotOptionKey] = None,
waterfall: Option[PlotOptionKey] = None,
var __name: String = PlotOptions.name
) extends HighchartKey(__name) {/**/}
第九个参数 ：subtitle
case class Title(
text: String = "", // Override default "Chart title"
align: Option[Alignment.Type] = None,
floating: Option[Boolean] = None,
style: Option[CSSObject] = None,
useHTML: Option[Boolean] = None,
verticalAlign: Option[VerticalAlignment.Type] = None,
x: Option[Int] = None,
y: Option[Int] = None,
var __name: String = "title"
)
第十个参数：setTurboThreshold: Boolean = true,
第十一个参数：tooltip: Option[ToolTip] = None,
case class ToolTip(
animation: Option[Boolean] = None,
backgroundColor: Option[Color.Type] = None,
borderColor: Option[Color.Type] = None,
borderRadius: Option[Int] = None,
borderWidth: Option[Int] = None,
// crosshairs
dateTimeLabelFormats: Option[DateTimeFormats] = None, // this has different defaults than the Axis
enabled: Option[Boolean] = None,
followPointer: Option[Boolean] = None,
followTouchMove: Option[Boolean] = None,
footerFormat: Option[String] = None,
//formatter
//headerFormat
hideDelay: Option[Int] = None,
//pointFormat
//positioner
shadow: Option[Boolean] = None,
shared: Option[Boolean] = None,
snap: Option[Int] = None,
//style
useHTML: Option[Boolean] = None,
valueDecimals: Option[Int] = None,
valuePrefix: Option[String] = None,
valueSuffix: Option[String] = None,
xDateFormat: Option[String] = None
) extends HighchartKey("ToolTip") {/**/}
第十二个参数：xAxis
case class Axis(
title: Option[AxisTitle] = Some(AxisTitle()),
allowDecimals: Option[Boolean] = None,
alternateGridColor: Option[Color.Type] = None,
categories: Option[Array[String]] = None,
dateTimeLabelFormats: Option[DateTimeFormats] = None,
endOnTick: Option[Boolean] = None,
//events
// gridLineColor: Color.Type = "#C0C0C0",
// gridLineDashStyle: String = "Solid", // TODO Bundle
// gridLineWidth: Int = 2,
var id: Option[String] = None,
labels: Option[AxisLabel] = None,
lineColor: Option[Color.Type] = None,
lineWidth: Option[Int] = None,
//linkedTo
max: Option[Int] = None,
// maxPadding: Double = 0.01,
min: Option[Int] = None,
// minPadding: Double = 0.01,
minRange: Option[Int] = None,
minTickInterval: Option[Int] = None,
//minor
offset: Option[Int] = None,
opposite: Option[Boolean] = None, // opposite side of chart, ie left / right for y-axis
//plotBands
//plotLines // TODO Kevin wants these
reversed: Option[Boolean] = None,
// showEmpty: Boolean = false,
showFirstLabel: Option[Boolean] = None,
showLastLabel: Option[Boolean] = None,
//startOfWeek
startOnTick: Option[Boolean] = None,
// tickColor: Color.Type = "#C0D0E0",
// TICK STUFF TODO
axisType: Option[AxisType.Type] = None,
var __name: String = "yAxis"
) extends HighchartKey(__name) {/**/}
第十三个参数：yAxis：同上
3、SparkStream和Wisp结合实验
import org.apache.log4j.{Level, Logger}
import org.apache.spark.streaming.{Seconds, StreamingContext}
import org.apache.spark.{SparkConf, SparkContext}
import com.quantifind.charts.Highcharts._
import com.quantifind.charts.highcharts._
import com.quantifind.charts.highcharts.Highchart._
object windowOnStreaming {
def main(args: Array[String]) {
/**
* this is test of SparkStreaming and wisp
*/
Logger.getLogger("org.apache.spark").setLevel(Level.ERROR)
Logger.getLogger("org.eclipse.jetty.Server").setLevel(Level.OFF)
val conf = new SparkConf().setAppName("the Window operation of SparK Streaming").setMaster("local[2]")
val sc = new SparkContext(conf)
val ssc = new StreamingContext(sc,Seconds(10))
//set the Checkpoint directory
ssc.checkpoint("/Res")
//get the socket Streaming data
val socketStreaming = ssc.socketTextStream("master",9999)
val data = socketStreaming.map(x =>(x,1)).reduceByKey(_+_)
data.foreachRDD( rdd =>{
val topWords=rdd.top(20)(Ordering.by[(String,Int),Int](_._2))
val (words,counts)=topWords.unzip
var hc=Highchart(Series(counts.zipWithIndex.map({case(x,y)=>(y,x)}), chart = SeriesType.column, name = "Word Counts", color = "#377"))
/**
* 设置参数 可以按如下模板
* var hc=Highchart(Series(counts.zipWithIndex.map({case(x,y)=>(y,x)}), chart = SeriesType.column,name = "Word Counts", __name = "series"),
Title(text = "hello",align = "center",floating = false,useHTML = true,verticalAlign = "top",x = 1,y = 1,__name = "title"),
Chart(zoomType = Zoom.xy),
Color.javaColorToHex(java.awt.Color.magenta),
Credits(enabled = true,href = "good",position = Position(x = 100,y = 100),style = CSSObject(color = Color.javaColorToHex(java.awt.Color.BLUE),fontWeight = "lighter"),text = "what the "),
Exporting(filename = "hi"),
Legend(backgroundColor = Color.javaColorToHex(java.awt.Color.darkGray)),
PlotOptions(area = PlotOptionKey(borderWidth = 12,groupPadding = 10,pointPadding = 14,stacking = Stacking.normal)),
Title(text = "I am lego"),
true,
ToolTip(animation = true,backgroundColor = Color.javaColorToHex(java.awt.Color.cyan)),
Axis( title = AxisTitle(text = "i am X "),lineColor = Color.javaColorToHex(java.awt.Color.BLACK)),
Axis( title = AxisTitle(text = "i am Y "),lineColor = Color.javaColorToHex(java.awt.Color.gray))
)
*/
hc = hc.copy(xAxis = hc.xAxis.map {
axisArray => axisArray.map { _.copy(axisType = Option("category"), categories = Option(words.toArray)) }})
plot(hc)
}
)
ssc.start()
ssc.awaitTermination()
}
}结果如下：
每个batchInterval 一个图
