11-flink如何正确分流
分流方式
- filter分流
- split分流 (只能一次分流，分流后的流不能继续分流)
- side output分流 (推荐使用)
场景
输入数据：
{"key":"001","type":"1","data":"data1"}
{"key":"001","type":"11","data":"data11"}
{"key":"001","type":"12","data":"data12"}
{"key":"002","type":"2","data":"data2"}
{"key":"002","type":"21","data":"data21"}
{"key":"002","type":"22","data":"data22"}
Operator中处理：转为jsonObject处理,并在jsonObject对象中添加流标志；
pom中需要引入fastjson (fastjson使用教程);
com.alibaba
fastjson
1.2.60
输出结果为：
11> {"data":"data1","stream":"stream1","type":"1","key":"001"}
12> {"data":"data11","stream":"stream1","type":"11","key":"001"}
12> {"data":"data11","stream":"stream11","type":"11","key":"001"}
1> {"data":"data12","stream":"stream1","type":"12","key":"001"}
1> {"data":"data12","stream":"stream12","type":"12","key":"001"}
2> {"data":"data2","stream":"stream2","type":"2","key":"002"}
3> {"data":"data21","stream":"stream2","type":"21","key":"002"}
3> {"data":"data21","stream":"stream21","type":"21","key":"002"}
4> {"data":"data22","stream":"stream2","type":"22","key":"002"}
4> {"data":"data22","stream":"stream22","type":"22","key":"002"}
发送数据客户端
nc程序发送(nc程序实现)
Fliter分流
代码实现：
public class FliterStreamTest {
public static final String TYPE = "type";
public static void main(String[] args) throws Exception {
//获取执行环境
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
final ParameterTool params = ParameterTool.fromArgs(args);
String hostName = params.get("hostname", "localhost");
int port = params.getInt("port", 9000);
DataStream sourceStream = env.socketTextStream(hostName, port, "\n");
SingleOutputStreamOperator jsonObjectStream = sourceStream.map(s -> JSONObject.parseObject(s));
DataStream stream1 = jsonObjectStream
.filter(e -> e.get(TYPE).equals("1") || e.get(TYPE).equals("11") || e.get(TYPE).equals("12"));
DataStream stream11 = stream1.filter(e -> e.get(TYPE).equals("11"));
DataStream stream12 = stream1.filter(e -> e.get(TYPE).equals("12"));
DataStream stream2 = jsonObjectStream
.filter(e -> e.get(TYPE).equals("2") || e.get(TYPE).equals("21") || e.get(TYPE).equals("22"));
DataStream stream21 = stream2.filter(e -> e.get(TYPE).equals("21"));
DataStream stream22 = stream2.filter(e -> e.get(TYPE).equals("22"));
//数据去向
stream1.map(e -> {e.put("stream", "stream1");return e;}).print();
stream11.map(e -> {e.put("stream", "stream11");return e;}).print();
stream12.map(e -> {e.put("stream", "stream12");return e;}).print();
stream2.map(e -> {e.put("stream", "stream2");return e;}).print();
stream21.map(e -> {e.put("stream", "stream21");return e;}).print();
stream22.map(e -> {e.put("stream", "stream22");return e;}).print();
env.execute("SocketStreamTest");
}
}
输出结果(与正确结果一直)：
11> {"data":"data1","stream":"stream1","type":"1","key":"001"}
12> {"data":"data11","stream":"stream1","type":"11","key":"001"}
12> {"data":"data11","stream":"stream11","type":"11","key":"001"}
1> {"data":"data12","stream":"stream1","type":"12","key":"001"}
1> {"data":"data12","stream":"stream12","type":"12","key":"001"}
2> {"data":"data2","stream":"stream2","type":"2","key":"002"}
3> {"data":"data21","stream":"stream2","type":"21","key":"002"}
3> {"data":"data21","stream":"stream21","type":"21","key":"002"}
4> {"data":"data22","stream":"stream2","type":"22","key":"002"}
4> {"data":"data22","stream":"stream22","type":"22","key":"002"}
Split分流
代码实现：
public class SplitStreamTest {
public static final String TYPE = "type";
public static void main(String[] args) throws Exception {
//获取执行环境
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
final ParameterTool params = ParameterTool.fromArgs(args);
String hostName = params.get("hostname", "localhost");
int port = params.getInt("port", 9000);
DataStream sourceStream = env.socketTextStream(hostName, port, "\n");
SingleOutputStreamOperator jsonObjectStream = sourceStream.map(s -> JSONObject.parseObject(s));
//flink split流已被标记为过期
SplitStream splitStream = jsonObjectStream.split(new OutputSelector() {
@Override
public Iterable select(JSONObject jsonObject) {
List tags = new ArrayList<>();
String type = jsonObject.get("type").toString();
if (type.equals("1") || type.equals("11") || type.equals("12")) {
tags.add("stream1");
} else if (type.equals("2") || type.equals("21") || type.equals("22")) {
tags.add("stream2");
}
return tags;
}
});
DataStream stream1 = splitStream.select("stream1");
DataStream stream2 = splitStream.select("stream2");
SplitStream splitStream1 = stream1.split(new OutputSelector() {
@Override
public Iterable select(JSONObject jsonObject) {
List tags = new ArrayList<>();
String type = jsonObject.get("type").toString();
if (type.equals("11")) {
tags.add("stream11");
} else if (type.equals("12")) {
tags.add("stream12");
}
return tags;
}
});
SplitStream splitStream2 = stream1.split(new OutputSelector() {
@Override
public Iterable select(JSONObject jsonObject) {
List tags = new ArrayList<>();
String type = jsonObject.get("type").toString();
if (type.equals("21")) {
tags.add("stream21");
} else if (type.equals("22")) {
tags.add("stream22");
}
return tags;
}
});
DataStream stream11 = splitStream1.select("stream11");
DataStream stream12 = splitStream1.select("stream12");
DataStream stream21 = splitStream2.select("stream21");
DataStream stream22 = splitStream2.select("stream22");
//数据去向
stream1.map(e -> {e.put("stream", "stream1");return e;}).print();
stream11.map(e -> {e.put("stream", "stream11");return e;}).print();
stream12.map(e -> {e.put("stream", "stream12");return e;}).print();
stream2.map(e -> {e.put("stream", "stream2");return e;}).print();
stream21.map(e -> {e.put("stream", "stream21");return e;}).print();
stream22.map(e -> {e.put("stream", "stream22");return e;}).print();
env.execute("SocketStreamTest");
}
}
输出结果：
Flink 1.9以前可以输出，但是第二次split将会失效；Flink 1.9 如果二次split，直接报错并提示使用side-outputs
Exception in thread "main" java.lang.IllegalStateException: Consecutive multiple splits are not supported. Splits are deprecated. Please use side-outputs.
Side-outputs分流(推荐使用)
步骤：
- 定义 OutputTag
- process 算子中收集数据
- ProcessFunction
- KeyedProcessFunction
- CoProcessFunction
- ProcessWindowFunction
- ProcessAllWindowFunction
代码实现：
public class SideOutputTest {
public static final String TYPE = "type";
public static void main(String[] args) throws Exception{
//获取执行环境
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
final ParameterTool params = ParameterTool.fromArgs(args);
String hostName = params.get("hostname", "localhost");
int port = params.getInt("port", 9000);
DataStream sourceStream = env.socketTextStream(hostName, port, "\n");
SingleOutputStreamOperator jsonObjectStream = sourceStream.map(s -> JSONObject.parseObject(s));
//定义OutputTag
OutputTag outputTag1 = new OutputTag("stream1") {};
OutputTag outputTag2 = new OutputTag("stream2") {};
SingleOutputStreamOperator outputStream = jsonObjectStream.process(new ProcessFunction() {
@Override
public void processElement(JSONObject jsonObject, Context context, Collector collector)
throws Exception {
String type = jsonObject.getString(TYPE);
if (type.equals("1") || type.equals("11") || type.equals("12")) {
context.output(outputTag1, jsonObject);
} else if (type.equals("2") || type.equals("21") || type.equals("22")) {
context.output(outputTag2, jsonObject);
}
}
});
DataStream stream1 = outputStream.getSideOutput(outputTag1);
DataStream stream2 = outputStream.getSideOutput(outputTag2);
//定义OutputTag
OutputTag outputTag11 = new OutputTag("stream11") {};
OutputTag outputTag12 = new OutputTag("stream12") {};
SingleOutputStreamOperator outputStream1 = stream1.process(new ProcessFunction() {
@Override
public void processElement(JSONObject jsonObject, Context context, Collector collector)
throws Exception {
String type = jsonObject.getString(TYPE);
if (type.equals("11")) {
context.output(outputTag11, jsonObject);
} else if (type.equals("12")) {
context.output(outputTag12, jsonObject);
}
}
});
DataStream stream11 = outputStream1.getSideOutput(outputTag11);
DataStream stream12 = outputStream1.getSideOutput(outputTag12);
//定义OutputTag
OutputTag outputTag21 = new OutputTag("stream21") {};
OutputTag outputTag22 = new OutputTag("stream22") {};
SingleOutputStreamOperator outputStream2 = stream2.process(new ProcessFunction() {
@Override
public void processElement(JSONObject jsonObject, Context context, Collector collector)
throws Exception {
String type = jsonObject.getString(TYPE);
if (type.equals("21")) {
context.output(outputTag21, jsonObject);
} else if (type.equals("22")) {
context.output(outputTag22, jsonObject);
}
}
});
DataStream stream21 = outputStream2.getSideOutput(outputTag21);
DataStream stream22 = outputStream2.getSideOutput(outputTag22);
//数据去向
stream1.map(e -> {e.put("stream", "stream1");return e;}).print();
stream11.map(e -> {e.put("stream", "stream11");return e;}).print();
stream12.map(e -> {e.put("stream", "stream12");return e;}).print();
stream2.map(e -> {e.put("stream", "stream2");return e;}).print();
stream21.map(e -> {e.put("stream", "stream21");return e;}).print();
stream22.map(e -> {e.put("stream", "stream22");return e;}).print();
env.execute("SocketStreamTest");
}
}
输出结果(与结果一致)：
3> {"data":"data1","stream":"stream1","type":"1","key":"001"}
4> {"data":"data11","stream":"stream11","type":"11","key":"001"}
4> {"data":"data11","stream":"stream1","type":"11","key":"001"}
5> {"data":"data12","stream":"stream12","type":"12","key":"001"}
5> {"data":"data12","stream":"stream1","type":"12","key":"001"}
6> {"data":"data2","stream":"stream2","type":"2","key":"002"}
7> {"data":"data21","stream":"stream21","type":"21","key":"002"}
7> {"data":"data21","stream":"stream2","type":"21","key":"002"}
8> {"data":"data22","stream":"stream22","type":"22","key":"002"}
8> {"data":"data22","stream":"stream2","type":"22","key":"002"}