6、NIFI综合应用场景-离线同步Mysql数据到HDFS中
Apache NiFi系列文章
1、nifi-1.9.2介绍、单机部署及简单验证
2、NIFI应用示例-GetFile和PutFile应用
3、NIFI处理器介绍、FlowFlie常见属性、模板介绍和运行情况信息查看
4、集群部署及验证、监控及节点管理
5、NiFi FileFlow示例和NIFI模板示例
6、NIFI应用场景-离线同步Mysql数据到HDFS中
7、NIFI综合应用场景-将mysql查询出的json数据转换成txt后存储至HDFS中
8、NIFI综合应用场景-NiFi监控MySQL binlog进行实时同步到hive
9、NIFI综合应用场景-通过NIFI配置kafka的数据同步
文章目录
- Apache NiFi系列文章
- 一、实现流程
-
- 1、模板
-
- 1）、模板1
- 2）、模板2
- 2、处理器流程
-
- 1）、模板1处理流程
- 2）、模板2处理流程
- 二、处理器说明
-
- 1、QueryDatabaseTable
-
- 1）、描述
- 2）、属性配置
- 2、ConvertAvroToJSON
-
- 1）、描述
- 2）、属性配置
- 3、SplitJson
-
- 1）、描述
- 2）、属性配置
- 4、PutHDFS
-
- 1）、描述
- 2）、属性配置
- 三、操作
-
- 1、创建组
- 2、创建并配置QueryDatabaseTable
- 3、创建并配置Mysql连接池
-
- 1）、创建
- 2）、配置
- 3）、启动连接池
- 4、创建并配置ConvertAvroToJSON
-
- 1）、创建配置ConvertAvroToJSON
- 2）、连接
- 3）、负载均衡消费数据
- 5、创建并配置SplitJson
- 6、创建并配置PutHDFS
- 四、验证
-
- 1、启动QueryDatabaseTable，并查看队列中数据
- 2、启动ConvertAvroToJSON，并查看队列中数据
- 3、启动SplitJson，并查看队列中数据
- 4、启动PutHDFS，并查看处理器接收和输出的数据
- 5、查看HDFS数据
本文旨在说明将mysql数据同步至HDFS中，并进行验证。阅读本文前最好是阅读本系列的前面文章关于模板中的介绍。
本文的前提依赖是mysql环境有数据、hadoop、nifi、hive、hue环境是搭建好的。如果hue环境没有，则在hdfs中进行验证。
本文分为四部分，即实现流程、使用的处理器介绍、在nifi中操作和验证结果。
一、实现流程
1、模板
1）、模板1
该模板可能出现异常–在验证中有说明–不同的环境可能存在不同。
<template encoding-version="1.2">
<description>将mysql中的数据导入到Hdfs中，并且使用lzo压缩方式。
存在重复的数据description>
<groupId>2f7d3766-0186-1000-0000-00006e07b64agroupId>
<name>MysqlToHDFSByLzoname>
<snippet>
<connections>
<id>8bacaebe-bce0-31e8-0000-000000000000id>
<parentGroupId>60d38136-211b-3d16-0000-000000000000parentGroupId>
<backPressureDataSizeThreshold>1 GBbackPressureDataSizeThreshold>
<backPressureObjectThreshold>10000backPressureObjectThreshold>
<destination>
<groupId>60d38136-211b-3d16-0000-000000000000groupId>
<id>4cb1eb1d-ca3a-34e0-0000-000000000000id>
<type>PROCESSORtype>
destination>
<flowFileExpiration>0 secflowFileExpiration>
<labelIndex>1labelIndex>
<loadBalanceCompression>COMPRESS_ATTRIBUTES_AND_CONTENTloadBalanceCompression>
<loadBalancePartitionAttribute>loadBalancePartitionAttribute>
<loadBalanceStatus>LOAD_BALANCE_INACTIVEloadBalanceStatus>
<loadBalanceStrategy>ROUND_ROBINloadBalanceStrategy>
<name>Q_Cname>
<selectedRelationships>successselectedRelationships>
<source>
<groupId>60d38136-211b-3d16-0000-000000000000groupId>
<id>c16280cc-6d1d-355c-0000-000000000000id>
<type>PROCESSORtype>
source>
<zIndex>0zIndex>
connections>
<connections>
<id>ce7dcdb2-bcd9-38a8-0000-000000000000id>
<parentGroupId>60d38136-211b-3d16-0000-000000000000parentGroupId>
<backPressureDataSizeThreshold>1 GBbackPressureDataSizeThreshold>
<backPressureObjectThreshold>10000backPressureObjectThreshold>
<destination>
<groupId>60d38136-211b-3d16-0000-000000000000groupId>
<id>26c8401a-8807-3771-0000-000000000000id>
<type>PROCESSORtype>
destination>
<flowFileExpiration>0 secflowFileExpiration>
<labelIndex>1labelIndex>
<loadBalanceCompression>COMPRESS_ATTRIBUTES_AND_CONTENTloadBalanceCompression>
<loadBalancePartitionAttribute>loadBalancePartitionAttribute>
<loadBalanceStatus>LOAD_BALANCE_INACTIVEloadBalanceStatus>
<loadBalanceStrategy>ROUND_ROBINloadBalanceStrategy>
<name>S_Pname>
<selectedRelationships>splitselectedRelationships>
<source>
<groupId>60d38136-211b-3d16-0000-000000000000groupId>
<id>20f76bcb-e978-3263-0000-000000000000id>
<type>PROCESSORtype>
source>
<zIndex>0zIndex>
connections>
<connections>
<id>f5322759-8583-3753-0000-000000000000id>
<parentGroupId>60d38136-211b-3d16-0000-000000000000parentGroupId>
<backPressureDataSizeThreshold>1 GBbackPressureDataSizeThreshold>
<backPressureObjectThreshold>10000backPressureObjectThreshold>
<destination>
<groupId>60d38136-211b-3d16-0000-000000000000groupId>
<id>20f76bcb-e978-3263-0000-000000000000id>
<type>PROCESSORtype>
destination>
<flowFileExpiration>0 secflowFileExpiration>
<labelIndex>1labelIndex>
<loadBalanceCompression>DO_NOT_COMPRESSloadBalanceCompression>
<loadBalancePartitionAttribute>loadBalancePartitionAttribute>
<loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGUREDloadBalanceStatus>
<loadBalanceStrategy>DO_NOT_LOAD_BALANCEloadBalanceStrategy>
<name>C_Sname>
<selectedRelationships>successselectedRelationships>
<source>
<groupId>60d38136-211b-3d16-0000-000000000000groupId>
<id>4cb1eb1d-ca3a-34e0-0000-000000000000id>
<type>PROCESSORtype>
source>
<zIndex>0zIndex>
connections>
<controllerServices>
<id>55bee1a0-0b0c-3a63-0000-000000000000id>
<parentGroupId>60d38136-211b-3d16-0000-000000000000parentGroupId>
<bundle>
<artifact>nifi-dbcp-service-narartifact>
<group>org.apache.nifigroup>
<version>1.9.2version>
bundle>
<comments>comments>
<descriptors>
<entry>
<key>Database Connection URLkey>
<value>
<name>Database Connection URLname>
value>
entry>
<entry>
<key>Database Driver Class Namekey>
<value>
<name>Database Driver Class Namename>
value>
entry>
<entry>
<key>database-driver-locationskey>
<value>
<name>database-driver-locationsname>
value>
entry>
<entry>
<key>kerberos-credentials-servicekey>
<value>
<identifiesControllerService>org.apache.nifi.kerberos.KerberosCredentialsServiceidentifiesControllerService>
<name>kerberos-credentials-servicename>
value>
entry>
<entry>
<key>Database Userkey>
<value>
<name>Database Username>
value>
entry>
<entry>
<key>Passwordkey>
<value>
<name>Passwordname>
value>
entry>
<entry>
<key>Max Wait Timekey>
<value>
<name>Max Wait Timename>
value>
entry>
<entry>
<key>Max Total Connectionskey>
<value>
<name>Max Total Connectionsname>
value>
entry>
<entry>
<key>Validation-querykey>
<value>
<name>Validation-queryname>
value>
entry>
<entry>
<key>dbcp-min-idle-connskey>
<value>
<name>dbcp-min-idle-connsname>
value>
entry>
<entry>
<key>dbcp-max-idle-connskey>
<value>
<name>dbcp-max-idle-connsname>
value>
entry>
<entry>
<key>dbcp-max-conn-lifetimekey>
<value>
<name>dbcp-max-conn-lifetimename>
value>
entry>
<entry>
<key>dbcp-time-between-eviction-runskey>
<value>
<name>dbcp-time-between-eviction-runsname>
value>
entry>
<entry>
<key>dbcp-min-evictable-idle-timekey>
<value>
<name>dbcp-min-evictable-idle-timename>
value>
entry>
<entry>
<key>dbcp-soft-min-evictable-idle-timekey>
<value>
<name>dbcp-soft-min-evictable-idle-timename>
value>
entry>
descriptors>
<name>MySQL_ConnectionPoolname>
<persistsState>falsepersistsState>
<properties>
<entry>
<key>Database Connection URLkey>
<value>jdbc:mysql://192.168.10.44:3306/test?characterEncoding=UTF-8&useSSL=false&allowPublicKeyRetrieval=truevalue>
entry>
<entry>
<key>Database Driver Class Namekey>
<value>com.mysql.jdbc.Drivervalue>
entry>
<entry>
<key>database-driver-locationskey>
<value>/usr/local/bigdata/testdata/mysql-connector-java-5.1.44.jarvalue>
entry>
<entry>
<key>kerberos-credentials-servicekey>
entry>
<entry>
<key>Database Userkey>
<value>rootvalue>
entry>
<entry>
<key>Passwordkey>
entry>
<entry>
<key>Max Wait Timekey>
entry>
<entry>
<key>Max Total Connectionskey>
entry>
<entry>
<key>Validation-querykey>
entry>
<entry>
<key>dbcp-min-idle-connskey>
entry>
<entry>
<key>dbcp-max-idle-connskey>
entry>
<entry>
<key>dbcp-max-conn-lifetimekey>
entry>
<entry>
<key>dbcp-time-between-eviction-runskey>
entry>
<entry>
<key>dbcp-min-evictable-idle-timekey>
entry>
<entry>
<key>dbcp-soft-min-evictable-idle-timekey>
entry>
properties>
<state>ENABLEDstate>
<type>org.apache.nifi.dbcp.DBCPConnectionPooltype>
controllerServices>
<processors>
<id>20f76bcb-e978-3263-0000-000000000000id>
<parentGroupId>60d38136-211b-3d16-0000-000000000000parentGroupId>
<position>
<x>4.0x>
<y>413.5y>
position>
<bundle>
<artifact>nifi-standard-narartifact>
<group>org.apache.nifigroup>
<version>1.9.2version>
bundle>
<config>
<bulletinLevel>WARNbulletinLevel>
<comments>comments>
<concurrentlySchedulableTaskCount>1concurrentlySchedulableTaskCount>
<descriptors>
<entry>
<key>JsonPath Expressionkey>
<value>
<name>JsonPath Expressionname>
value>
entry>
<entry>
<key>Null Value Representationkey>
<value>
<name>Null Value Representationname>
value>
entry>
descriptors>
<executionNode>ALLexecutionNode>
<lossTolerant>falselossTolerant>
<penaltyDuration>30 secpenaltyDuration>
<properties>
<entry>
<key>JsonPath Expressionkey>
<value>$.*value>
entry>
<entry>
<key>Null Value Representationkey>
<value>empty stringvalue>
entry>
properties>
<runDurationMillis>0runDurationMillis>
<schedulingPeriod>0 secschedulingPeriod>
<schedulingStrategy>TIMER_DRIVENschedulingStrategy>
<yieldDuration>1 secyieldDuration>
config>
<executionNodeRestricted>falseexecutionNodeRestricted>
<name>SplitJson_Demoname>
<relationships>
<autoTerminate>trueautoTerminate>
<name>failurename>
relationships>
<relationships>
<autoTerminate>trueautoTerminate>
<name>originalname>
relationships>
<relationships>
<autoTerminate>falseautoTerminate>
<name>splitname>
relationships>
<state>STOPPEDstate>
<style/>
<type>org.apache.nifi.processors.standard.SplitJsontype>
processors>
<processors>
<id>26c8401a-8807-3771-0000-000000000000id>
<parentGroupId>60d38136-211b-3d16-0000-000000000000parentGroupId>
<position>
<x>3.0x>
<y>624.5y>
position>
<bundle>
<artifact>nifi-hadoop-narartifact>
<group>org.apache.nifigroup>
<version>1.9.2version>
bundle>
<config>
<bulletinLevel>WARNbulletinLevel>
<comments>comments>
<concurrentlySchedulableTaskCount>1concurrentlySchedulableTaskCount>
<descriptors>
<entry>
<key>Hadoop Configuration Resourceskey>