【极数系列】Flink项目入门搭建（03）

【极数系列】Flink项目入门搭建（03）

引言

gitee地址：https://gitee.com/shawsongyue/aurora.git
源码直接下载可运行，模块：aurora_flink
Flink 版本：1.18.0
Jdk 版本：11

1.创建mavenx项目

2.包结构

3.引入pom依赖

tips：transformer处写主启动类

    4.0.0

    com.xsy
    aurora_flink
    1.0-SNAPSHOT

    
    
        
        11
        
        3.8.1
        
        UTF-8
        
        UTF-8
        
        1.2.75
        
        2.17.1
        
        1.18.0
        
        2.11
        
        2.17.1
    

    
    

        
        
            com.alibaba
            fastjson
            ${fastjson.version}
        

        
        
            org.apache.flink
            flink-java
            ${flink.version}
        

        
            org.apache.flink
            flink-streaming-scala_2.12
            ${flink.version}
        

        
        
            org.apache.flink
            flink-clients
            ${flink.version}
        


        
        
        
            org.apache.logging.log4j
            log4j-slf4j-impl
            ${log4j.version}
        

        
            org.apache.logging.log4j
            log4j-api
            ${log4j.version}
        

        
            org.apache.logging.log4j
            log4j-core
            ${log4j.version}
        

        
    

    
    
        ${project.name}
        
        
            
                src/main/resources
            
            
                src/main/java
                
                    **/*.xml
                
            
        

        
            
                org.apache.maven.plugins
                maven-shade-plugin
                3.1.1
                
                    
                        package
                        
                            shade
                        
                        
                            
                                
                                    org.apache.flink:force-shading
                                    org.google.code.flindbugs:jar305
                                    org.slf4j:*
                                    org.apache.logging.log4j:*
                                
                            
                            
                                
                                    *:*
                                    
                                        META-INF/*.SF
                                        META-INF/*.DSA
                                        META-INF/*.RSA
                                    
                                
                            
                            
                                
                                    org.xsy.sevenhee.flink.TestStreamJob
                                
                            
                        
                    
                
            
        

        
        
            
                
                
                    org.springframework.boot
                    spring-boot-maven-plugin
                    ${spring.boot.version}
                    
                        true
                        ${project.build.finalName}
                    
                    
                        
                            
                                repackage
                            
                        
                    
                

                
                
                    maven-compiler-plugin
                    ${maven.plugin.version}
                    
                        ${java.version}
                        ${java.version}
                        UTF-8
                        
                            -parameters
                        
                    
                
            
        
    

    
    
        
            aliyun-repos
            https://maven.aliyun.com/nexus/content/groups/public/
            
                false
            
        
    

    
    
        
            aliyun-plugin
            https://maven.aliyun.com/nexus/content/groups/public/
            
                false
            
        
    

4.增加log4j2.properties配置

tips：resource目录下增加该配置，主要用于日志打印

rootLogger.level=INFO
rootLogger.appenderRef.console.ref=ConsoleAppender
appender.console.name=ConsoleAppender
appender.console.type=CONSOLE
appender.console.layout.type=PatternLayout
appender.console.layout.pattern=%d{HH:mm:ss,SSS} %-5p %-60c %x - %m%n
log.file=D:\\tmp

5.创建主启动类

tips：编写了一个简单的有界数据流处理demo程序

• step1：创建flink程序运行所需环境
• step2：创建数据集
• step3：把有限数据集转换为数据源
• step4：简单通过flatmap处理数据
• step5：输出最终结果
• step6：启动任务

package com.aurora;


import org.apache.flink.api.common.functions.FlatMapFunction;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.sink.SinkFunction;
import org.apache.flink.util.Collector;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.util.ArrayList;
import java.util.Random;
import java.util.UUID;

/**
 * @author 浅夏的猫
 * @description 主启动类
 * @date 22:46 2024/1/13
 */
public class Application {

    private static final Logger logger = LoggerFactory.getLogger(Application.class);

    public static void main(String[] args) throws Exception {

        //1.创建flink程序运行所需环境
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

        //2.创建数据集
        ArrayList list = new ArrayList<>();
        list.add("001");
        list.add("002");
        list.add("003");

        //3.把有限数据集转换为数据源
        DataStreamSource dataStreamSource = env.fromCollection(list).setParallelism(1);

        //4.简单通过flatmap处理数据，
        SingleOutputStreamOperator flatMap = dataStreamSource.flatMap(new FlatMapFunction() {
            @Override
            public void flatMap(String record, Collector collector) throws Exception {
                //数据追加随机数
                String uuidRecord=record+ UUID.randomUUID().toString();
                //当前环节处理完需要传递数据给下个环节
                collector.collect(uuidRecord);
            }
        });

        //5.输出最终结果
        flatMap.addSink(new SinkFunction() {
            @Override
            public void invoke(String value) throws Exception {
                logger.info("当前正在处理的数据:{}",value);
            }
        }).setParallelism(1);

        //6.启动任务
        env.execute();
    }
}

6.构建打jar包

7.flinkUI页面部署

1.点击add new上传对应的应用包

2.主类填写com.aurora.Application

3.检查任务running状态，大概几秒钟跑完