Docker部署ELK（ElasticSearch logstash Kibana）

Docker部署ELK

一 . ElasticSearch安装

1. 新建elasticsearch目录，并再其下新建文件config/elasticsearch.yml，文件内容如下：

   cluster.name: "docker-cluster"
   network.host: 0.0.0.0
   xpack:
     ml.enabled: false
     monitoring.enabled: false
     security.enabled: false
     watcher.enabled: false
   

2. 配置vm.max_map_count参数：

   永久生效：

   vi /etc/sysctl.conf
   #修改
   vm.max_map_count = 262144
   
   sysctl -p
   
   

   临时生效：

   sysctl -w vm.max_map_count=262144
   

3. 从镜像网站拉取镜像

   3.1 搜索Elasticsearch镜像

   docker search elasticsearch
   

   3.2 拉取镜像

   docker pull docker.elastic.co/elasticsearch/elasticsearch:6.8.2
   

   3.3 查看镜像是否下载成功：

   docker images
   

4. 运行镜像：

   docker run --rm -d -it -p 9200:9200 -p 9300:9300 -v "$PWD"/config/elasticsearch.yml:/home/jeon/elasticsearch/config/elasticsearch.yml docker.elastic.co/elasticsearch/elasticsearch:6.8.2
   

5. 查看是否启动成功

   docker ps
   

6. 访问170.290.21.1:9200 如出现一下界面，表示安装成功

二 . logstash安装

1. 拉取镜像

   docker pull docker.elastic.co/logstash/logstash:6.5.4
   

2. 创建文件夹

   mkdir -p   /home/jeon/logstash/conf
   chmod  777  /home/jeon/logstash/conf
   mkdir -p  /home/jeon/logstash/plugin
   chmod  777  /home/jeon/logstash/plugin
   mkdir -p  /home/jeon/logstash/pipeline
   chmod  777  /home/jeon/logstash/pipeline
   

3. mysql数据同步到ElasticSearch中

   3.1 下载Java mysql驱动包 mysql-connector-java

   cd /home/jeon/logstash/plugin
   
   wget http://central.maven.org/maven2/mysql/mysql-connector-java/8.0.16/mysql-connector-java-8.0.16.jar
   

   3.2 配置logstash.conf

   ​ vim /home/jeon/logstash/pipeline/logstash.conf

   ​

   input {
           jdbc {
                   jdbc_driver_library => "/plugin/mysql-connector-java-8.0.16.jar"
                   jdbc_driver_class => "com.mysql.jdbc.Driver"
                   jdbc_connection_string => "jdbc:mysql://17.99.213.73:3306/gpmall?useUnicode=true&characterEncoding=utf8&useSSL=false"
                   jdbc_user => "root"
                   jdbc_password => "123456"
                   schedule => "* * * * *"
                   jdbc_paging_enabled => true
                   statement => "select * from tb_item"
           }
           jdbc {
                   jdbc_driver_library => "/plugin/mysql-connector-java-8.0.16.jar"
                   jdbc_driver_class => "com.mysql.jdbc.Driver"
                   jdbc_connection_string => "jdbc:mysql://17.99.213.73:3306/gpmall?useUnicode=true&characterEncoding=utf8&useSSL=false"
                   jdbc_user => "root"
                   jdbc_password => "123456"
                   schedule => "* * * * *"
                   jdbc_paging_enabled => true
                   #执行语句
                   statement => "select * from tb_item where updated > :sql_last_value"
                   #使用数据库表的列
                   use_column_value => true
                   tracking_column => "updated"
                   tracking_column_type => "timestamp"
                   #这是存放上一次执行之后id的值
                   last_run_metadata_path => "lastUpdated.txt"
           }
   }
   
   output {
           #输入es地址
           elasticsearch {
               hosts => ["17.99.213.73:9200"]
               # 索引名字，必须小写
               index => "tb_item"
               # 数据唯一索引
               document_id => "%{id}"
           }
           stdout { codec => json_lines }
   }
   

   3.3 配置文件logstash.yml

   • vim /home/jeon/logstash/conf/logstash.yml

   http.host: "0.0.0.0"
   path.config: /home/jeon/logstash/pipeline
   xpack.monitoring.elasticsearch.url: http://17.99.213.73:9200
   

   • 放行端口号

     firewall-cmd --zone=public --add-port=5000/tcp --permanent
     firewall-cmd --zone=public --add-port=5044/tcp --permanent
     firewall-cmd --zone=public --add-port=9600/tcp --permanent
     firewall-cmd --reload
     

   3.4运行容器

   docker run -v /home/jeon/logstash/pipeline:/usr/share/logstash/pipeline -v /home/jeon/logstash/conf/logstash.yml:/usr/share/logstash/config/logstash.yml -v /home/jeon/logstash/plugin:/plugin -p 5000:5000 -p 5044:5044 -p 9600:9600   --name logstash --privileged=true -d  docker.elastic.co/logstash/logstash:6.5.4 -f /usr/share/logstash/pipeline/logstash.conf 
   
   

三 . kibana安装

1. 新建kibana文件夹，并在其下新建config/kibana.yml文件，内容如下：

   server.name: kibana
   server.host: "0"
   elasticsearch.url: "http://17.99.213.73:9200"
   xpack.monitoring.ui.container.elasticsearch.enabled: true
   i18n.locale: "zh-CN"
   

2. 拉取镜像

   docker pull docker.elastic.co/kibana/kibana:6.5.4
   

3. 运行容器

   docker run --rm -d -it -p 5601:5601 -v "$PWD"/conf/kibana.yml:/usr/share/kibana/config/kibana.yml docker.elastic.co/kibana/kibana:6.5.4
   

​