Flink流处理常见API

1. map

DataStream mapStram = dataStream.map(new MapFunction() {

    public Integer map(String value) throws Exception {

        return value.length();

    }

});

---

2. flatMap

DataStream flatMapStream = dataStream.flatMap(new FlatMapFunction() {

    public void flatMap(String value, Collector out) throws Exception { 

        String[] fields = value.split(",");

        for (String field: fields) {

            out.collect(field);

        }

    }

});

---

3. filter

DataStream filterStream = dataStream.filter(new FilterFunction() {

    public boolean filter(String value) throws Exception {

        return value == 1;

    }

});

---

4. keyBy

DataStream -> KeyedStream. 逻辑上将相同key拆分到同一分区

---

5. Rolling Aggregation

对KeyedStream的每一个支流做滚动聚合。sum(), min(), max(), minBy(), maxBy()

---

6. reduce

KeyedStream -> DataStream. 合并当前的元素和上次聚合的结果，产生一个新的值，返回的流中包含每一次聚合的结果

// 分组

KeyedStream keyedStream= dataStream.keyBy("id");

// reduce 聚合，取最小的温度值，并输出当前的时间戳

DataStream reduceStream = keyedStream.reduce(new ReduceFunction() { 

    @Override

    public SensorReading reduce(SensorReading value1,SensorReading value2) throws Exception {

        return new SensorReading(value1.getId(), value2.getTimestamp(), Math.min(value1.getTemperature(), value2.getTemperature()));

    }

});

---

7. Split和Select

DataStream -> split -> SplitStream -> select -> DataStream.

SplitStream splitStream = dataStream.split(new OutputSelector() {

    @Override

    public Iterable select(SensorReading value) {

        return (value.getTemperature()>30) ? Collections.singletonList("high") : Collections.singletonList("low");

    }

});

DataStream highTempStream = splitStream.select("high");

DataStream lowTempStream = splitStream.select("low");

DataStream allTempStream = splitStream.select("high","low");

---

8. Connect

DataStream,DataStream -> ConnectedStreams. 可以连接两个不同类型的数据流，两个流被connect之后，只是被放到了同一个流里，仍然保持各自的数据类型

---

9. CoMap和CoFlatMap

ConnectedStreams -> DataStream. 

// 合流 connect

DataStream> warningStream = highTempStream.map(new MapFunction>() {

    @Override

    public Tuple2 map(SensorReading value) throws Exception {

        return new Tuple2<>(value.getId(), value.getTemperature());

    }

});

ConnectedStreams, SensorReading> connectedStreams = warningStream.connect(lowTempStream);

DataStream