Flink Operators 实战进阶篇

返回

DataStream Transformations进阶

Union

定义

Transformation	Description
DataStream* → DataStream	Union of two or more data streams creating a new stream containing all the elements from all the streams.Creates a new DataStream by merging DataStream outputs of the same type with each other. The DataStreams merged using this operator will be transformed simultaneously.

说明

两个或以上的流合并成一个，这两个流的数据类型必须一样

样例

代码

public class UnionDemo {

    public static void main(String[] args) throws Exception{
        final StreamExecutionEnvironment env = StreamExecutionEnvironment.createLocalEnvironment();
        env.setParallelism(1);

        DataStream> orangeStream = env.addSource(new DataSource("orangeStream"));
        DataStream> greenStream = env.addSource(new DataSource("greenStream"));

        orangeStream.union(greenStream).print("union");
        env.execute("Union Demo");
    }

    private static class DataSource extends RichParallelSourceFunction> {
        private volatile boolean running = true;
        private volatile String name;

        public DataSource(String name) {
            this.name = name;
        }

        @Override
        public void run(SourceContext> ctx) throws Exception {
            Random random = new Random();
            int bound = 100;
            final long numElements = 6;
            int i = 0;

            while (running && i < numElements) {
                Thread.sleep(1500);
                Tuple2 data = new Tuple2<>("foo", random.nextInt(bound));
                ctx.collect(data);
                System.out.println(Thread.currentThread().getId() + "-" + this.name + "-sand data:" + data);
                i++;
            }
        }

        @Override
        public void cancel() {
            running = false;
        }
    }
}

输出结果

59-greenStream-sand data:(foo,6)
58-orangeStream-sand data:(foo,33)
union> (foo,33)
union> (foo,6)
59-greenStream-sand data:(foo,99)
58-orangeStream-sand data:(foo,99)
union> (foo,99)
union> (foo,99)
58-orangeStream-sand data:(foo,23)
59-greenStream-sand data:(foo,36)
union> (foo,23)
union> (foo,36)
59-greenStream-sand data:(foo,78)
58-orangeStream-sand data:(foo,7)
union> (foo,7)
union> (foo,78)
59-greenStream-sand data:(foo,7)
58-orangeStream-sand data:(foo,58)
union> (foo,58)
union> (foo,7)
59-greenStream-sand data:(foo,66)
58-orangeStream-sand data:(foo,80)
union> (foo,80)
union> (foo,66)

说明

DataSource类随机发送数据作为数据源，将两个流做union输出

Connect&CoFlatMap

定义

Transformation	Description
DataStream,DataStream → ConnectedStreams	"Connects" two data streams retaining their types. Connect allowing for shared state between the two streams.
ConnectedStreams → DataStream	Similar to flatMap on a connected data stream

说明

Connect可以连接两个不同数据类型的流，这个是和union最主要的区别，其次union支持2个以上流的合并，而Connect只支持2个流；可以借助CoFlatMap将不同类型的流进行类型统一等操作。

样例

代码

public class ConnectDemo {
    public static void main(String[] args) throws Exception{
        final StreamExecutionEnvironment env = StreamExecutionEnvironment.createLocalEnvironment();
        env.setParallelism(1);

        DataStream> orangeStream = env.addSource(new DataSource1());
        DataStream> greenStream = env.addSource(new DataSource2());

        orangeStream.connect(greenStream).flatMap(new CoFlatMapFunction, Tuple3, Object>() {
            @Override
            public void flatMap1(Tuple2 value, Collector