代码工具 流数据分析 华为

诺亚方舟实验室的资深研究员Albert Bifet等最近在GitHub上发布了Spark Streaming上的流数据分析工具包，这个叫做StreamDM的开源软件现在包括5个算法，今后还会有更多算法被加入。网页链接 请大家关注。Albert也是Storm上的流数据分析工具包Samoa的主要开发者。

Big Data Stream Learning
Big Data stream learning is more challenging than batch or offline learning, since the data may not keep the same distribution over the lifetime of the stream. Moreover, each example coming in a stream can only be processed once, or they need to be summarized with a small memory footprint, and the learning algorithms must be very efficient.
Spark Streaming
Spark Streaming is an extension of the core Spark API that enables stream processing from a variety of sources. Spark is a extensible and programmable framework for massive distributed processing of datasets, called Resilient Distributed Datasets (RDD). Spark Streaming receives input data streams and divides the data into batches, which are then processed by the Spark engine to generate the results.
Spark Streaming data is organized into a sequence of DStreams, represented internally as a sequence of RDDs.
Included Methods
In this first pre-release of StreamDM, we have implemented:
SGD Learner and Perceptron
Naive Bayes
CluStream
Hoeffding Decision Trees
Stream KM++

In the next releases we plan to add:
Random Forests
Frequent Itemset Miner: IncMine

Going Further
For a quick introduction to running StreamDM, refer to the Getting Started document. The StreamDMProgramming Guide presents a detailed view of StreamDM. The full API documentation can be consulted here.