【ELK7.6.0】docker es单节点环境搭建及logstash同步mysql数据至ElasticSearch

单节点ES

1 安装elasticsearch

1.1 测试elasticsearch

docker run -d --name elasticsearch -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" elasticsearch:7.6.0

1.3 创建目录

mkdir -p /web/es/config/
mkdir -p /web/es/data/ \

1.3 拷贝文件

docker cp elasticsearch:/usr/share/elasticsearch/config/ /web/es/

1.4 宿主机文件夹授权

chmod -R 777 /web/es/

1.5 正式运行es

docker run --name elasticsearch -v /web/es/config/:/usr/share/elasticsearch/config -v /web/es/data/:/usr/share/elasticsearch/data -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" --privileged=true -d elasticsearch:7.6.0

1.6 测试

http://192.168.100.222:9200/ 出现下图即正常
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-t8CD8ji8-1583393390883)(/uploads/yongfeng_jyx_server_conf/images/m_99bfdf57722754c9d8be078e776b9062_r.png)]

2 安装elasticsearch-head

docker run -d --name es_admin -p 9100:9100 mobz/elasticsearch-head:5

2.1 问题

安装上打开http://192.168.100.222:9100/ 连接 100.222：9200如果没有反应即es没有连通
到elasticsearch /usr/share/elasticsearch/config/elasticsearch.yml 增加下面两行代码 有宿主机映射则在相应映射文件修改

http.cors.enabled: true
http.cors.allow-origin: "*"

有如下集群信息即可

3 安装logstash

3.1 创建目录

mkdir -p /web/logstash/

3.2 测试运行

docker run -d --name=logstash logstash:7.6.0

3.3 复制内容

docker cp logstash:/usr/share/logstash/config/ /web/logstash/
docker cp logstash:/usr/share/logstash/pipeline/ /web/logstash/
docker cp logstash:/usr/share/logstash/logstash-core/lib/jars/ /web/logstash/

3.4 mysql-connector-java-8.0.13.jar 拷贝至

/web/logstash/jars/

3.5 正式运行

docker run -d \
--privileged=true \
--restart=always -m=2.5G --memory-swap=2.5G --cpuset-cpus="16,17" \
-v /web/logstash/config/:/usr/share/logstash/config/ \
-v /web/logstash/pipeline/:/usr/share/logstash/pipeline/ \
-v /web/logstash/jars/:/usr/share/logstash/logstash-core/lib/jars/ \
--name=logstash \
logstash:7.6.0

4 安装kibana

docker run --name kibana --link elasticsearch:elasticsearch -p 5601:5601 -d kibana:7.6.0

5 配置logstash

5.1 修改jvm虚拟机配置 解决内存占用不足OOM 问题

修改/web/logstash/config/jvm.options

-Xms1g
-Xmx2g

5.2 修改目标es链接

/web/es/config/logstash.yml

http.host: "0.0.0.0"
#xpack.monitoring.elasticsearch.hosts: [ "http://elasticsearch:9200" ]
xpack.monitoring.elasticsearch.hosts: [ "http://192.168.2.117:9200" ]

• 没有link elasticsearch的话 就填写es的ip和端口

5.3 配置logstash.conf

修改/web/logstash/pipeline/logstash.conf

input {
 jdbc {
    jdbc_driver_library => "/usr/share/logstash/logstash-core/lib/jars/mysql-connector-java-8.0.13.jar"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    jdbc_connection_string => "jdbc:mysql://192.168.100.222:3306/liyumes_zhengshi_log?useSSL=false&characterEncoding=UTF-8&serverTimezone=GMT%2b8"
    jdbc_user => "root"
    jdbc_password => "123456Aa,"
    jdbc_paging_enabled => "true"
    jdbc_page_size => "5000"
    codec => plain { charset => "UTF-8"}
    tracking_column => "unix_ts_in_secs"
    use_column_value => true
    schedule => "*/10 * * * * *" 
    statement => "SELECT *, UNIX_TIMESTAMP(create_time) AS unix_ts_in_secs FROM liyuoa_journalitem WHERE (UNIX_TIMESTAMP(create_time) > :sql_last_value AND create_time < NOW()) ORDER BY create_time ASC"
  }
}

filter {
  mutate {
    copy => { "id" => "[@metadata][_id]"} 
    remove_field => ["id", "@version", "unix_ts_in_secs"]
  }
}
output {
   elasticsearch {
                 hosts => "192.168.100.222:9200"
                 index => "liyumes_log"
                 timeout => 300
                 document_id => "%{[@metadata][_id]}" 
                 }
}

• 防止内存溢出等异常 mysql采取分页处理
• jdbc_paging_enabled => “true”
• jdbc_page_size => “5000”

6 kibana 配置

6.1 设置index-pattern，之前logstash配置时候的‘index’

http://192.168.100.222:5601/app/kibana

6.2 discover查看 index匹配的