操作系统centos 7 ,版本logstash-6.2.4
logstash-input-jdbc配置说明:首先在logstash 目录下创建 logstash-default.conf 文件,然后编辑,
#输入
input {
stdin {
}
jdbc{
jdbc_connection_string => "jdbc:oracle:thin:@localhost:1521:ORADBUAT" # jdbc数据库连接
jdbc_user => "xxxx" #自己数据库用户名
jdbc_password => "xxxx" #自己数据库密码
jdbc_driver_library => "/apps/ojdbc8.jar" # oracle 链接jdbc的 jar包文件路径
jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver" # jdbc数据库驱动
jdbc_paging_enabled => "true"
jdbc_page_size => "5000000" # 同步数据分页设置
#statement => "select * from T_SYS_LOG where TO_DATE(TO_CHAR(OPERATION_TIME,'yyyy-mm-dd HH24:MI:SS'),'yyyy-mm-dd HH24:MI:SS') - interval '8' hour > :sql_last_value" # 也可以直接将sql配置在这里
statement => "select id as u_id,operation_type as operationtype ,operation_person as operationperson,area as area,site as site,business_type as businesstype,operation_time as operationtime,sitename as sitename,ownertable as ownertable ,processname as processname, approvalstatus as approvalstatus ,app as app,recordid as recordid from T_SYS_LOG where id > :sql_last_value" # 也可以直接将sql配置在这里
schedule => "* * * * *" # 各字段含义(由左至右)分、时、天、月、年,全部为*默认含义为每分钟都更新
type => "jdbc" # ES的type类型,相当于数据库中的table,需要配置多个表时,将jdbc整体复制一份,修改对应type
use_column_value => "true"
tracking_column => "u_id" #其他配置位
}
}
filter {
date {
locale => "zh"
timezone => "Asia/Shanghai"
match => [ "OPERATION_TIME", "UNIX_MS" ]
}
}
output {
elasticsearch {
hosts => "10.59.8.48:9200"
index => "logstash-default-%{+YYYY.MM.dd}"
codec => "json"
}
}
以上就是我的配置说明:
特别提醒下:
1、如果statement =>“select * from table_name” 启动之后,默认只会执行一次
2、如果 statement =>“select * from table_name where OPERATION_TIME>:sql_last_value” ,其中”OPERATION_TIME“ 这个字段为自己table中的字段而且是一个日期类型的,这个sql_last_value 就表示,运行之后,新的时间大于最后的是时间值,这样就会查询当前大于最后一次 大于这个时间值的数据,而且是配置为每一分钟执行一次(schedule => "* * * * *" #这个配置自己可以设定)。
3、如果 statement => "select id as u_id,operation_type as operationtype ,operation_person as operationperson,area as area,site as site,business_type as businesstype,operation_time as operationtime,sitename as sitename,ownertable as ownertable ,processname as processname, approvalstatus as approvalstatus ,app as app,recordid as recordid from T_SYS_LOG where id > :sql_last_value" 这种方式,使用id递增主键作为查询,那么配置的时候一定注意,首先的设置
use_column_value => "true"
tracking_column => "u_id" #其他配置位
看我标识为红色地方,这样配置,之后就会根据每次最好一次查询的id,跟新增加的id大于最后一次的id,才会执行查询
比如我这里就是用的 id,启动之后 ,如下图所示:
黄色框:表示执行了,查询操作,
红色框:表示是查询数据库中id>6802开始的数据