Flink Steam Demo

使用mvn创建项目

mvn archetype:generate \
    -DarchetypeGroupId=org.apache.flink \
    -DarchetypeArtifactId=flink-quickstart-java \
    -DarchetypeVersion=1.9.0 \
    -DgroupId=wiki-edits \
    -DartifactId=wiki-edits \
    -Dversion=0.1 \
    -Dpackage=wikiedits \
    -DinteractiveMode=false \
    -DarchetypeCatalog=internal

增加最后一个选项防止卡死在创建阶段.

rm wiki-edits/src/main/java/wikiedits/*.java

修改依赖

    
        org.apache.flink
        flink-java
        ${flink.version}
    
    
        org.apache.flink
        flink-streaming-java_2.11
        ${flink.version}
    
    
        org.apache.flink
        flink-clients_2.11
        ${flink.version}
    
    
        org.apache.flink
        flink-connector-wikiedits_2.11
        ${flink.version}
    

编写代码
创建WikipediaAnalysis.java

package wikiedits;

import org.apache.flink.api.common.functions.AggregateFunction;
import org.apache.flink.api.java.functions.KeySelector;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.datastream.KeyedStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.windowing.time.Time;
import org.apache.flink.streaming.connectors.wikiedits.WikipediaEditEvent;
import org.apache.flink.streaming.connectors.wikiedits.WikipediaEditsSource;

public class WikipediaAnalysis {

  public static void main(String[] args) throws Exception {

    StreamExecutionEnvironment see = StreamExecutionEnvironment.getExecutionEnvironment();

    DataStream edits = see.addSource(new WikipediaEditsSource());

    KeyedStream keyedEdits = edits
      .keyBy(new KeySelector() {
        @Override
        public String getKey(WikipediaEditEvent event) {
          return event.getUser();
        }
      });

    DataStream> result = keyedEdits
      .timeWindow(Time.seconds(5))
      .aggregate(new AggregateFunction, Tuple2>() {
        @Override
        public Tuple2 createAccumulator() {
          return new Tuple2<>("", 0L);
        }

        @Override
        public Tuple2 add(WikipediaEditEvent value, Tuple2 accumulator) {
          accumulator.f0 = value.getUser();
          accumulator.f1 += value.getByteDiff();
          return accumulator;
        }

        @Override
        public Tuple2 getResult(Tuple2 accumulator) {
          return accumulator;
        }

        @Override
        public Tuple2 merge(Tuple2 a, Tuple2 b) {
          return new Tuple2<>(a.f0, a.f1 + b.f1);
        }
      });

    result.print();

    see.execute();
  }
}

编译执行

mvn clean package
mvn exec:java -Dexec.mainClass=wikiedits.WikipediaAnalysis

需要下载好久编译需要的包.

别管如何跑起来了.
作用是是从流数据源中获取编辑的文字并统计.

7> (4.198.82.219,-28)
7> (Greyhoundlover0,812)
5> (Mauritaniaahu,72)
6> (Ser Amantio di Nicolao,42)
4> (.76.229.79,0)
4> (Waschlamadam,4)
8> (.123.174.203,7)
3> (Apnagm,-124)
2> (6.181.80.33,-10)
8> (Citation bot,-16)
3> (Lowercase sigmabot III,0)
5> (InternetArchiveBot,39)
5> (Khonda8,11)

代码看起来好抽象.
感觉python或scala写起来会简单很多.
ML的库居然在1.9木有....啥意思.