文末附下载方式
组件 | 版本 |
---|---|
elasticseach | 7.13.0 |
kibana | 7.13.0 |
logstash | 7.13.0 |
flink | 1.13.6 |
因为在正常的情况下,Flink的流数据是非常大的,有时候会使用print()打印数据自己查看,有时候为了查找问题会开启debug日志,就会导致日志文件非常大,通过Web UI查看对应的日志文件是会非常卡,所以首先将日志文件按照大小滚动生成文件,我们在查看时不会因为某个文件非常大导致Web UI界面卡,没法查看。
# Allows this configuration to be modified at runtime. The file will be checked every 30 seconds.
monitorInterval=30
# This affects logging for both user code and Flink
rootLogger.level = INFO
rootLogger.appenderRef.file.ref = RollingFileAppender
# Uncomment this if you want to _only_ change Flink's logging
#logger.flink.name = org.apache.flink
#logger.flink.level = INFO
# The following lines keep the log level of common libraries/connectors on
# log level INFO. The root logger does not override this. You have to manually
# change the log levels here.
logger.akka.name = akka
logger.akka.level = INFO
logger.kafka.name= org.apache.kafka
logger.kafka.level = INFO
logger.hadoop.name = org.apache.hadoop
logger.hadoop.level = INFO
logger.zookeeper.name = org.apache.zookeeper
logger.zookeeper.level = INFO
logger.shaded_zookeeper.name = org.apache.flink.shaded.zookeeper3
logger.shaded_zookeeper.level = INFO
# Log all infos in the given file
appender.rolling.name = RollingFileAppender
appender.rolling.type = RollingFile
appender.rolling.append = false
appender.rolling.fileName = ${sys:log.file}
appender.rolling.filePattern = ${sys:log.file}.%i
appender.rolling.layout.type = PatternLayout
appender.rolling.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n
appender.rolling.policies.type = Policies
appender.rolling.policies.size.type = SizeBasedTriggeringPolicy
appender.rolling.policies.size.size = 500MB
appender.rolling.strategy.type = DefaultRolloverStrategy
appender.rolling.strategy.max = 10
# Suppress the irrelevant (wrong) warnings from the Netty channel handler
logger.netty.name = org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline
logger.netty.level = OFF
针对按照日志文件大小滚动生成文件的方式,可能因为某个错误的问题,需要看好多个日志文件,所以可以把日志文件通过KafkaAppender写入到kafka中,然后通过ELK等进行日志搜索甚至是分析告警。
# Allows this configuration to be modified at runtime. The file will be checked every 30 seconds.
monitorInterval=30
# This affects logging for both user code and Flink
rootLogger.level = INFO
rootLogger.appenderRef.kafka.ref = Kafka
rootLogger.appenderRef.file.ref = RollingFileAppender
# Uncomment this if you want to _only_ change Flink's logging
#logger.flink.name = org.apache.flink
#logger.flink.level = INFO
# The following lines keep the log level of common libraries/connectors on
# log level INFO. The root logger does not override this. You have to manually
# change the log levels here.
logger.akka.name = akka
logger.akka.level = INFO
logger.kafka.name= org.apache.kafka
logger.kafka.level = INFO
logger.hadoop.name = org.apache.hadoop
logger.hadoop.level = INFO
logger.zookeeper.name = org.apache.zookeeper
logger.zookeeper.level = INFO
logger.shaded_zookeeper.name = org.apache.flink.shaded.zookeeper3
logger.shaded_zookeeper.level = INFO
# Log all infos in the given file
appender.rolling.name = RollingFileAppender
appender.rolling.type = RollingFile
appender.rolling.append = false
appender.rolling.fileName = ${sys:log.file}
appender.rolling.filePattern = ${sys:log.file}.%i
appender.rolling.layout.type = PatternLayout
appender.rolling.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n
appender.rolling.policies.type = Policies
appender.rolling.policies.size.type = SizeBasedTriggeringPolicy
appender.rolling.policies.size.size = 500MB
appender.rolling.strategy.type = DefaultRolloverStrategy
appender.rolling.strategy.max = 10
# kafka
appender.kafka.type = Kafka
appender.kafka.name = Kafka
appender.kafka.syncSend = true
appender.kafka.ignoreExceptions = false
appender.kafka.topic = flink_logs
appender.kafka.property.type = Property
appender.kafka.property.name = bootstrap.servers
appender.kafka.property.value = localhost:9092
appender.kafka.layout.type = JSONLayout
apender.kafka.layout.value = net.logstash.log4j.JSONEventLayoutV1
appender.kafka.layout.compact = true
appender.kafka.layout.complete = false
#appender.kafka.layout.additionalField1.type = KeyValuePair
#appender.kafka.layout.additionalField1.key = logdir
#appender.kafka.layout.additionalField1.value = ${sys:log.file}
#appender.kafka.layout.additionalField2.type = KeyValuePair
#appender.kafka.layout.additionalField2.key = flink_job_name
#appender.kafka.layout.additionalField2.value = ${sys:flink_job_name}
#appender.kafka.layout.additionalField3.type = KeyValuePair
#appender.kafka.layout.additionalField3.key = yarnContainerId
#appender.kafka.layout.additionalField3.value = ${sys:yarnContainerId}
# Suppress the irrelevant (wrong) warnings from the Netty channel handler
logger.netty.name = org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline
logger.netty.level = OFF
上面的appender.kafka.layout.type
可以使用JSONLayout
,也可以自定义。自定义需要将上面的appender.kafka.layout.type
和appender.kafka.layout.value
修改成如下:
appender.kafka.layout.type = PatternLayout
appender.kafka.layout.pattern = {"log_level":"%p","log_timestamp":"%d{ISO8601}","log_thread":"%t","log_file":"%F","log_line":"%L","log_message":"'%m'","log_path":"%X{log_path}","job_name":"${sys:flink_job_name}"}%n
针对于不同的Flink模式,需要在flink/lib
目录下放入的jar包也有所不同:
# 根据kafka的版本放入kafka-clients
kafka-clients-3.1.0.jar
如果是通过flink on yarn
模式还可以添加自定义字段:
# 日志路径
appender.kafka.layout.additionalField1.type = KeyValuePair
appender.kafka.layout.additionalField1.key = logdir
appender.kafka.layout.additionalField1.value = ${sys:log.file}
# flink-job-name
appender.kafka.layout.additionalField2.type = KeyValuePair
appender.kafka.layout.additionalField2.key = flinkJobName
appender.kafka.layout.additionalField2.value = ${sys:flinkJobName}
# 提交到yarn的containerId
appender.kafka.layout.additionalField3.type = KeyValuePair
appender.kafka.layout.additionalField3.key = yarnContainerId
appender.kafka.layout.additionalField3.value = ${sys:yarnContainerId}
添加以上的自定义字段,需要在flink-conf.yaml
中配置如下:
env.java.opts.taskmanager: -DyarnContainerId=$CONTAINER_ID
env.java.opts.jobmanager: -DyarnContainerId=$CONTAINER_ID
之后进入flink目录提交任务到yarn:
./bin/flink run-application \
-d \
-t yarn-application \
-Dyarn.application.name=TopSpeed \
-Dmetrics.reporter.promgateway.groupingKey="TopSpeed" \
-Dmetrics.reporter.promgateway.jobName=TopSpeed \
-Denv.java.opts="-DflinkJobName=TopSpeed" \
./examples/streaming/TopSpeedWindowing.jar
消费kafka的消息可以看到如下:
{
"instant":{
"epochSecond":1655379271,
"nanoOfSecond":642000000
},
"thread":"main",
"level":"INFO",
"loggerName":"org.apache.flink.runtime.io.network.netty.NettyServer",
"message":"Transport type 'auto': using EPOLL.",
"endOfBatch":false,
"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger",
"threadId":1,
"threadPriority":5,
"logdir":"/yarn/container-logs/application_1655373794581_0005/container_e46_1655373794581_0005_01_000002/taskmanager.log",
"flinkJobName":"TopSpeed",
"yarnContainerId":"container_e46_1655373794581_0005_01_000002"
}
# 根据kafka的版本放入kafka-clients
kafka-clients-3.1.0.jar
# jackson对应的jar包
jackson-annotations-2.13.3.jar
jackson-core-2.13.3.jar
jackson-databind-2.13.3.jar
日志写入Kafka之后可以通过Logstash接入elasticsearch,然后通过kibana进行查询或搜索,安装logstash的步骤可以参考官网。
将以下内容写入config/logstash-sample.conf
文件中
input {
kafka {
bootstrap_servers => ["cdh2:9092,cdh3:9092,cdh4:9092"]
group_id => "logstash-group"
topics => ["flink_logs"]
consumer_threads => 3
type => "flink-logs"
codec => "json"
auto_offset_reset => "latest"
}
}
output {
elasticsearch {
hosts => ["cdh2:9200","cdh3:9200","cdh4:9200"]
index => "flink-log-%{+YYYY-MM-dd}"
}
}
上述配置按天生成新的flink日志索引,之后运行logstash,运行的日志没有报错即可在elasticsearch查看对应的索引日志。
bin/logstash -f ./config/logstash-sample.conf 2>&1 >./logs/logstash.log &
新建索引模式,因为我们上面的flink日志的索引名称为flin-log-%{+YYYY-MM-dd}
,所以我们创建一个匹配flink-log的索引模式即可。
之后点击下一步,选择时间字段为@timestamp
即可。
打开Discover选择刚刚创建的索引模式,在左上角可以添加筛选条件。