Flink java wordcount

前言

各位好，欢迎浏览我的博客，后面将持续更新小编在flink上学习的心得体会，希望越来越多的新同学加入到这个行列中。
本次小编开发是flink的入门程序 wordcount

项目的目录结构

本项目就是一个简答的spring boot的项目，
LiuUtilApplication：是spring boot的项目启动类。
WindowWordCount：是flink实现单词统计的主要功能。
helloword.txt：是一个文本文件，里面都是汉子以空格分隔。

pom文件

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.4.0</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>
    <groupId>com.liulangtao.flink</groupId>
    <artifactId>liu-util</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>liu-util</name>
    <description>Demo project for Spring Boot</description>

    <properties>
        <java.version>1.8</java.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter</artifactId>
            <exclusions>
                <exclusion>
                    <groupId>ch.qos.logback</groupId>
                    <artifactId>logback-classic</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-core</artifactId>
            <version>1.11.2</version>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-java</artifactId>
            <version>1.11.2</version>
        </dependency>


        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-streaming-java_2.11</artifactId>
            <version>1.11.2</version>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-clients_2.12</artifactId>
            <version>1.11.2</version>
        </dependency>


        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-streaming-scala_2.11</artifactId>
            <version>1.11.2</version>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-runtime-web_2.12</artifactId>
            <version>1.11.2</version>
        </dependency>


        <!--<dependency>-->
            <!--<groupId>org.apache.flink</groupId>-->
            <!--<artifactId>flink-connector-kafka_2.12</artifactId>-->
            <!--<version>1.11.2</version>-->
        <!--</dependency>-->


        <!--<dependency>-->
            <!--<groupId>org.apache.flink</groupId>-->
            <!--<artifactId>flink-table-common</artifactId>-->
            <!--<version>1.11.2</version>-->
        <!--</dependency>-->

        <!--#flink-streaming-java_2.11-->
        <!--<dependency>-->
            <!--  <groupId>org.apache.flink</groupId>-->
            <!--  <artifactId>flink-streaming-java_2.11</artifactId>-->
            <!--  <version>1.11.1</version>-->
            <!--  <scope>provided</scope>-->
        <!--</dependency>-->
        <!--#flink-scala_2.11-->
        <!--<dependency>-->
            <!--  <groupId>org.apache.flink</groupId>-->
            <!--  <artifactId>flink-scala_2.11</artifactId>-->
            <!--  <version>1.11.1</version>-->
            <!--  <scope>provided</scope>-->
        <!--</dependency>-->
        <!--#flink-streaming-scala_2.11-->
        <!--<dependency>-->
            <!--  <groupId>org.apache.flink</groupId>-->
            <!--  <artifactId>flink-streaming-scala_2.11</artifactId>-->
            <!--  <version>1.11.1</version>-->
            <!--  <scope>provided</scope>-->
        <!--</dependency>-->

        <!--#flink-streaming-scala_2.11-->
        <!--#flink-connector-kafka-0.10_2.11-->
        <!--<dependency>-->
        <!--    <groupId>org.apache.flink</groupId>-->
        <!--    <artifactId>flink-connector-kafka-0.10_2.11</artifactId>-->
        <!--    <version>1.11.1</version>-->
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>

</project>

WindowWordCount.java

public class WindowWordCount {

    public static void main(String[] args) throws Exception {

        ExecutionEnvironment executionEnvironment =ExecutionEnvironment.getExecutionEnvironment();
       String path="E:\\4-openProject\\20-flink\\src\\main\\resources\\helloword.txt";
        DataSet<String> stringDataSource = executionEnvironment
                .readTextFile(path);
        DataSet< Tuple2<String, Integer>> set =
                stringDataSource.flatMap(new myFlatMap())
        .groupBy(0)
       .sum(1);
        set.print();
    }
    public  static  class myFlatMap implements FlatMapFunction<String, Tuple2<String,Integer>> {
        @Override
        public void flatMap(String s, Collector<Tuple2<String, Integer>> collector) throws Exception {
          String [] words=  s.split(" ");
            for ( String word: words) {
                collector.collect(new Tuple2<>(word,1));
            }
        }
    }
}

helloword.txt文件

hello word
hellp zhangsan
hello wangwu
hellp  lisi
hello 123

运行结果

注意

1.pom文件里面 spring-boot-starter里面要把ch.qos.logback日志插件去掉，不然会打很多日志。

org.springframework.boot
spring-boot-starter


ch.qos.logback
logback-classic



2.不去掉日志 会有下面的报错，暂时由于刚学习还不确定报错的原因。

org.apache.flink.runtime.taskexecutor.slot.SlotNotFoundException: Could not find slot for fecc2c52cd4abfe0b0c7400fce272eee.

初步的认为是spring boot 没有绑定flink数据源，导致springboot
跑flink的程序时候，会查询到flink的节点已经停止