nifi的安装
nifi的安装
最近领导突然说要用nifi来进行数据同步和数据迁移，无奈之下，只好研究下nifi这个强大的工具了，百度了一下相关资料，发现网上关于这个的资料太少，干脆就自己写一个
下载源码
网址：http://archive.apache.org/dist/nifi/1.13.2/，版本比较多，我这边就选这个1.13.2这个版本吧
文件有点大1.4G（我这边用的是linux版本）
传输至服务
传输方式自选哈，我这边用的是lrzsz这个工具
[root@localhost ~]# yum -y install lrzsz
已加载插件：fastestmirror
Determining fastest mirrors
* base: mirrors.aliyun.com
* extras: mirrors.aliyun.com
* updates: mirrors.aliyun.com
base | 3.6 kB 00:00:00
extras | 2.9 kB 00:00:00
updates | 2.9 kB 00:00:00
(1/4): base/7/x86_64/group_gz | 153 kB 00:00:00
(2/4): extras/7/x86_64/primary_db | 243 kB 00:00:00
(3/4): base/7/x86_64/primary_db | 6.1 MB 00:00:01
(4/4): updates/7/x86_64/primary_db | 11 MB 00:00:02
正在解决依赖关系
--> 正在检查事务
---> 软件包 lrzsz.x86_64.0.0.12.20-36.el7 将被 安装
--> 解决依赖关系完成
依赖关系解决
=================================================================================================================================================
Package 架构 版本 源 大小
=================================================================================================================================================
正在安装:
lrzsz x86_64 0.12.20-36.el7 base 78 k
事务概要
=================================================================================================================================================
安装 1 软件包
总下载量：78 k
安装大小：181 k
Downloading packages:
警告：/var/cache/yum/x86_64/7/base/packages/lrzsz-0.12.20-36.el7.x86_64.rpm: 头V3 RSA/SHA256 Signature, 密钥 ID f4a80eb5: NOKEY
lrzsz-0.12.20-36.el7.x86_64.rpm 的公钥尚未安装
lrzsz-0.12.20-36.el7.x86_64.rpm | 78 kB 00:00:00
从 file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7 检索密钥
导入 GPG key 0xF4A80EB5:
用户ID : "CentOS-7 Key (CentOS 7 Official Signing Key) "
指纹 : 6341 ab27 53d7 8a78 a7c2 7bb1 24c6 a8a7 f4a8 0eb5
软件包 : centos-release-7-5.1804.el7.centos.x86_64 (@anaconda)
来自 : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
正在安装 : lrzsz-0.12.20-36.el7.x86_64 1/1
验证中 : lrzsz-0.12.20-36.el7.x86_64 1/1
已安装:
lrzsz.x86_64 0:0.12.20-36.el7
完毕！
[root@localhost ~]# ls
anaconda-ks.cfg nifi-1.13.2-bin.tar.gz
解压源码
[root@localhost ~]# tar -xvf nifi-1.13.2-bin.tar.gz
*********省略若干
nifi-1.13.2/docs/html/images/version-states-display.png
nifi-1.13.2/docs/html/images/versioned-flow-imported.png
nifi-1.13.2/docs/html/images/versioned-process-group.png
nifi-1.13.2/docs/html/images/xyz-flow.png
nifi-1.13.2/docs/html/images/xyz-process-group.png
nifi-1.13.2/docs/html/images/zero-leader-cluster-http-access.png
nifi-1.13.2/docs/html/images/zero-leader-cluster.png
nifi-1.13.2/docs/html/images/zero-leader-node.png
nifi-1.13.2/docs/html/nifi-in-depth.html
nifi-1.13.2/docs/html/overview.html
nifi-1.13.2/docs/html/record-path-guide.html
nifi-1.13.2/docs/html/toolkit-guide.html
nifi-1.13.2/docs/html/user-guide.html
nifi-1.13.2/docs/html/walkthroughs.html
[root@localhost ~]#
修改默认端口号
这一步是为了防止端口的冲突，nifi默认的端口号为：8080
nifi的配置文件路径：nifi-1.13.2/conf/nifi.properties
[root@localhost ~]# cp nifi-1.13.2/conf/nifi.properties nifi-1.13.2/conf/nifi.properties.bak
[root@localhost ~]# vim nifi-1.13.2/conf/nifi.properties
******
# web properties #
#############################################
# For security, NiFi will present the UI on 127.0.0.1 and only be accessible through this loopback interface.
# Be aware that changing these properties may affect how your instance can be accessed without any restriction.
# We recommend configuring HTTPS instead. The administrators guide provides instructions on how to do this.
nifi.web.http.host=192.168.211.171 ##本机的ip，如果本地访问可以写成127.0.0.0
nifi.web.http.port=5800 ##默认端口号（http协议的），咱用这个
nifi.web.http.network.interface.default=
#############################################
nifi.web.https.host=
nifi.web.https.port= ###默认端口号（https协议的）
nifi.web.https.network.interface.default=
nifi.web.jetty.working.directory=./work/jetty
nifi.web.jetty.threads=200
nifi.web.max.header.size=16 KB
***********
启动nifi
[root@localhost ~]# nifi-1.13.2/bin/nifi.sh
Usage nifi {start|stop|run|restart|status|dump|diagnostics|install|stateless}
[root@localhost ~]# nifi-1.13.2/bin/nifi.sh start ##出现这种报错的原因是咱的没有安装jdk环境，下一步安装jdk环境去
nifi.sh: JAVA_HOME not set; results may vary
nifi-1.13.2/bin/nifi.sh: 第 139 行:type: java: 未找到
nifi.sh: java command not found
安装部署jdk环境
JDK下载官方网址
JDK下载：http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
[root@localhost ~]# ls
anaconda-ks.cfg jdk-8u60-linux-x64.tar.gz nifi-1.13.2 nifi-1.13.2-bin.tar.gz
##### 部署java环境jdk
#jdk的解压安装
[root@localhost ~]# tar xf jdk-8u60-linux-x64.tar.gz -C /usr/local/
[root@localhost ~]# ln -s /usr/local/jdk1.8.0_60 /usr/local/jdk
#####配置java环境变量
[root@localhost ~]# sed -i.ori '$a export JAVA_HOME=/usr/local/jdk\nexport PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH\nexport CLASSPATH=.$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/lib/tools.jar' /etc/profile
[root@localhost ~]# tail -3 /etc/profile
export JAVA_HOME=/usr/local/jdk
export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH
export CLASSPATH=.$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/lib/tools.jar
####让java环境变量立刻生效
[root@localhost ~]# source /etc/profile
#####检查java环境安装情况
[root@localhost ~]# which java
/usr/local/jdk/bin/java
[root@localhost ~]# java -version #出现以下信息表示部署成功
java version "1.8.0_60"
Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)
关于上面那个sed命令的说明：
sed -i.ori :-i表示对文件本身操作，.ori表示修改的同时备份源文件
表示文件内容的最后一行，a表示在下面进行数据插入
\n :表示插入数据时换行
重新运行nifi
[root@localhost ~]# nifi-1.13.2/bin/nifi.sh start
Java home: /usr/local/jdk
NiFi home: /root/nifi-1.13.2
Bootstrap Config File: /root/nifi-1.13.2/conf/bootstrap.conf
[root@localhost ~]# ss -antup | grep 5800
tcp LISTEN 0 50 127.0.0.1:5800 *:* users:(("java",pid=8919,fd=3191))
[root@localhost ~]# ps -ef | grep nifi
root 8905 1 0 10:35 pts/0 00:00:00 /bin/sh nifi-1.13.2/bin/nifi.sh start
root 8907 8905 0 10:35 pts/0 00:00:00 /usr/local/jdk/bin/java -cp /root/nifi-1.13.2/conf:/root/nifi-1.13.2/lib/bootstrap/* -Xms12m -Xmx24m -Dorg.apache.nifi.bootstrap.config.log.dir=/root/nifi-1.13.2/logs -Dorg.apache.nifi.bootstrap.config.pid.dir=/root/nifi-1.13.2/run -Dorg.apache.nifi.bootstrap.config.file=/root/nifi-1.13.2/conf/bootstrap.conf org.apache.nifi.bootstrap.RunNiFi start
root 8919 8907 57 10:35 pts/0 00:00:52 /usr/local/jdk/bin/java -classpath /root/nifi-1.13.2/./conf:/root/nifi-1.13.2/./lib/javax.servlet-api-3.1.0.jar:/root/nifi-1.13.2/./lib/jetty-schemas-3.1.jar:/root/nifi-1.13.2/./lib/logback-classic-1.2.3.jar:/root/nifi-1.13.2/./lib/logback-core-1.2.3.jar:/root/nifi-1.13.2/./lib/jcl-over-slf4j-1.7.30.jar:/root/nifi-1.13.2/./lib/jul-to-slf4j-1.7.30.jar:/root/nifi-1.13.2/./lib/log4j-over-slf4j-1.7.30.jar:/root/nifi-1.13.2/./lib/slf4j-api-1.7.30.jar:/root/nifi-1.13.2/./lib/nifi-api-1.13.2.jar:/root/nifi-1.13.2/./lib/nifi-framework-api-1.13.2.jar:/root/nifi-1.13.2/./lib/nifi-server-api-1.13.2.jar:/root/nifi-1.13.2/./lib/nifi-runtime-1.13.2.jar:/root/nifi-1.13.2/./lib/nif-nar-utils-1.13.2.jar:/root/nifi-1.13.2/./lib/nifi-properties-1.13.2.jar:/root/nifi-1.13.2/./lib/nifi-stateless-bootstrap-1.13.2.jar:/root/nifi-1.13.2/./lib/nifi-stateless-api-1.13.2.jar -Dorg.apache.jasper.compiler.disablejsr199=true -Xmx512m -Xms512m -Dcurator-log-only-first-connection-issue-as-error-level=true -Djavax.security.auth.useSubjectCredsOnly=true -Djava.security.egd=file:/dev/urandom -Dzookeeper.admin.enableServer=false -Dsun.net.http.allowRestrictedHeaders=true -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -Djava.protocol.handler.pkgs=sun.net.www.protocol -Dnifi.properties.file.path=/root/nifi-1.13.2/./conf/nifi.properties -Dnifi.bootstrap.listen.port=40289 -Dapp=NiFi -Dorg.apache.nifi.bootstrap.config.log.dir=/root/nifi-1.13.2/logs org.apache.nifi.NiFi
root 9011 1328 0 10:37 pts/0 00:00:00 grep --color=auto nifi
防火墙安全配置
由于小编较懒，这里直接关闭防火墙了哈
[root@localhost ~]# systemctl stop firewalld
[root@localhost ~]# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
Active: inactive (dead) since 三 2021-09-15 10:39:25 CST; 6s ago
Docs: man:firewalld(1)
Process: 674 ExecStart=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS (code=exited, status=0/SUCCESS)
Main PID: 674 (code=exited, status=0/SUCCESS)
9月 15 10:00:03 localhost.localdomain systemd[1]: Starting firewalld - dynamic firewall daemon...
9月 15 10:00:04 localhost.localdomain systemd[1]: Started firewalld - dynamic firewall daemon.
9月 15 10:39:24 localhost.localdomain systemd[1]: Stopping firewalld - dynamic firewall daemon...
9月 15 10:39:25 localhost.localdomain systemd[1]: Stopped firewalld - dynamic firewall daemon.
浏览器访问
浏览器输入：http://192.168.211.171:5800/nifi
切记哈，我上面配置文件用的是htpp协议的，如果大家要用https的话就的输入https://192.168.211.171:5800/nifi
oK,安装成功
查看Nifi日志
[root@localhost ~]# tail -f nifi-1.13.2/logs/nifi-app.log
2021-09-15 10:43:34,782 INFO [main] o.eclipse.jetty.server.AbstractConnector Started ServerConnector@4e4c3a38{HTTP/1.1, (http/1.1)}{192.168.211.171:5800}
2021-09-15 10:43:34,783 INFO [main] org.eclipse.jetty.server.Server Started @45749ms
2021-09-15 10:43:34,869 INFO [main] org.apache.nifi.nar.NarAutoLoader Starting NAR Auto-Loader for directory ./extensions ...
2021-09-15 10:43:34,879 INFO [main] org.apache.nifi.nar.NarAutoLoader NAR Auto-Loader started
2021-09-15 10:43:34,879 INFO [main] org.apache.nifi.web.server.JettyServer NiFi has started. The UI is available at the following URLs:
2021-09-15 10:43:34,879 INFO [main] org.apache.nifi.web.server.JettyServer http://192.168.211.171:5800/nifi
2021-09-15 10:43:34,893 INFO [main] org.apache.nifi.BootstrapListener Successfully initiated communication with Bootstrap
2021-09-15 10:43:34,894 INFO [main] org.apache.nifi.NiFi Controller initialization took 30201887601 nanoseconds (30 seconds).
2021-09-15 10:43:49,482 INFO [pool-20-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
2021-09-15 10:43:49,483 INFO [pool-20-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 0 milliseconds
2021-09-15 10:44:09,485 INFO [pool-20-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
2021-09-15 10:44:09,485 INFO [pool-20-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 0 milliseconds
2021-09-15 10:44:29,024 INFO [Cleanup Archive for default] o.a.n.c.repository.FileSystemRepository Successfully deleted 0 files (0 bytes) from archive
2021-09-15 10:44:29,486 INFO [pool-20-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
2021-09-15 10:44:29,486 INFO [pool-20-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 0 milliseconds
2021-09-15 10:44:49,486 INFO [pool-20-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
2021-09-15 10:44:49,487 INFO [pool-20-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 0 milliseconds
2021-09-15 10:45:09,488 INFO [pool-20-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
2021-09-15 10:45:09,488 INFO [pool-20-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 0 milliseconds
2021-09-15 10:45:29,049 INFO [Cleanup Archive for default] o.a.n.c.repository.FileSystemRepository Successfully deleted 0 files (0 bytes) from archive
2021-09-15 10:45:29,167 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@663d59ed checkpointed with 0 Records and 0 Swap Files in 15 milliseconds (Stop-the-world time = 3 milliseconds, Clear Edit Logs time = 1 millis), max Transaction ID -1
2021-09-15 10:45:29,488 INFO [pool-20-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
2021-09-15 10:45:29,489 INFO [pool-20-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 0 milliseconds
2021-09-15 10:45:49,491 INFO [pool-20-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
2021-09-15 10:45:49,491 INFO [pool-20-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 0 milliseconds
2021-09-15 10:46:09,493 INFO [pool-20-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
2021-09-15 10:46:09,493 INFO [pool-20-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 0 milliseconds
2021-09-15 10:46:29,067 INFO [Cleanup Archive for default] o.a.n.c.repository.FileSystemRepository Successfully deleted 0 files (0 bytes) from archive
2021-09-15 10:46:29,494 INFO [pool-20-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
2021-09-15 10:46:29,494 INFO [pool-20-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 0 milliseconds
2021-09-15 10:46:49,496 INFO [pool-20-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
2021-09-15 10:46:49,496 INFO [pool-20-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 0 milliseconds
2021-09-15 10:47:09,498 INFO [pool-20-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
2021-09-15 10:47:09,498 INFO [pool-20-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 0 milliseconds
2021-09-15 10:47:29,086 INFO [Cleanup Archive for default] o.a.n.c.repository.FileSystemRepository Successfully deleted 0 files (0 bytes) from archive
2021-09-15 10:47:29,179 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@663d59ed checkpointed with 0 Records and 0 Swap Files in 3 milliseconds (Stop-the-world time = 1 milliseconds, Clear Edit Logs time = 1 millis), max Transaction ID -1
2021-09-15 10:47:29,512 INFO [pool-20-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
2021-09-15 10:47:29,512 INFO [pool-20-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 0 milliseconds
2021-09-15 10:47:49,513 INFO [pool-20-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
2021-09-15 10:47:49,513 INFO [pool-20-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 0 milliseconds
2021-09-15 10:48:09,516 INFO [pool-20-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
2021-09-15 10:48:09,516 INFO [pool-20-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 0 milliseconds
停止运行nifi
[root@localhost ~]# nifi-1.13.2/bin/nifi.sh stop
Java home: /usr/local/jdk
NiFi home: /root/nifi-1.13.2
Bootstrap Config File: /root/nifi-1.13.2/conf/bootstrap.conf
2021-09-15 10:40:46,576 INFO [main] org.apache.nifi.bootstrap.Command Apache NiFi has accepted the Shutdown Command and is shutting down now
2021-09-15 10:40:46,689 INFO [main] org.apache.nifi.bootstrap.Command Waiting for Apache NiFi to finish shutting down...
2021-09-15 10:40:48,697 INFO [main] org.apache.nifi.bootstrap.Command NiFi has finished shutting down.