flink 配置流

场景

开发流数据时，经常会用一些配置来和数据流进行合并操作进行业务逻辑的实现。spark streaming可以使用广播变量的形式来实现，flink可以通过配置流和数据流连接来实现。

接口

主要使用CoFlatMapFunction这个接口。
源码：

package org.apache.flink.streaming.api.functions.co;

import org.apache.flink.annotation.Public;
import org.apache.flink.api.common.functions.Function;
import org.apache.flink.util.Collector;

import java.io.Serializable;

/**
 * A CoFlatMapFunction implements a flat-map transformation over two
 * connected streams.
 *
 * 
The same instance of the transformation function is used to transform
 * both of the connected streams. That way, the stream transformations can
 * share state.
 *
 * 
An example for the use of connected streams would be to apply rules that change over time
 * onto elements of a stream. One of the connected streams has the rules, the other stream the
 * elements to apply the rules to. The operation on the connected stream maintains the
 * current set of rules in the state. It may receive either a rule update (from the first stream)
 * and update the state, or a data element (from the second stream) and apply the rules in the
 * state to the element. The result of applying the rules would be emitted.
 *
 * @param  Type of the first input.
 * @param  Type of the second input.
 * @param  Output type.
 */
@Public
public interface CoFlatMapFunction extends Function, Serializable {

    /**
     * This method is called for each element in the first of the connected streams.
     *
     * @param value The stream element
     * @param out The collector to emit resulting elements to
     * @throws Exception The function may throw exceptions which cause the streaming program
     *                   to fail and go into recovery.
     */
    void flatMap1(IN1 value, Collector out) throws Exception;

    /**
     * This method is called for each element in the second of the connected streams.
     *
     * @param value The stream element
     * @param out The collector to emit resulting elements to
     * @throws Exception The function may throw exceptions which cause the streaming program
     *                   to fail and go into recovery.
     */
    void flatMap2(IN2 value, Collector out) throws Exception;
}

就像注释中写的 flatMap1中实现具体逻辑。flatMap2中读取配置流数据。

实现

先来一个普通的数据流source，再来一个配置流，这里数据流source是kafka的source，配置流是自定义的source，并且把配置流广播出去。

 /* 初始化配置流 */
    val httpSource = new HttpSource(ruleBatchTime,ruleAddress,name)
    val ruleStream = env.addSource(httpSource).name("ruleSource").broadcast
    
    val dataStream = env.addSource(kafkaConsumer08).name("dataSource")

然后用数据流的去关联配置流,在用flatMap实现逻辑。

val ruleData = dataStream.connect(ruleStream).flatMap(new RuleData(logSource,virtualRedis,ruleObject))

flatMap中的逻辑代码实现要继承自上面提到的CoFlatMapFunction接口。

//data,rule,output
class RuleData
  extends RichCoFlatMapFunction[String,String,String] {
  lazy val logger = LoggerFactory.getLogger(this.getClass.getName)
  var conf: String = null

  override def flatMap1(value: String, out: Collector[String]): Unit = {
    //使用conf来实现业务逻辑，value是数据流里面的数据
  }

  override def flatMap2(value: String, out: Collector[String]): Unit = {
  conf = value //更新配置变量
  }

}