基于docker的ELK镜像构建


title: 基于docker搭建ELK镜像构建

本文主要记录下以前搭建ELK的docker镜像过程,记录下docker基础使用,如何基于centos的基础镜像构建应用。

基础镜像 elkbase:v1.0

  • 准备jdk 放在tools文件夹下

  • 准备Dockerfile

      FROM centos:latest
      MAINTAINER Yampery
      VOLUME [ "/opt/product/data/" ]
      RUN  /bin/cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
      RUN /bin/echo -e "ZONE="Asia/Shanghai"\nUTC=false\nRTC=false" > /etc/sysconfig/clock
      RUN mkdir /opt/product/tools/
      ADD ./tools /opt/product/tools/
      ENV JAVA_HOME /opt/product/tools/jdk-9.0.1
      CMD ["/usr/sbin/init"]
    
  • 构建文件结构

      |-elk
      	|-tools
      		|-jdk-9.0.1
      	|--Dockerfile
    
  • 构建镜像

      elk/elkbase$ docker build -t elkbase:v1.0 . 
    

elasticsearch:v1.0

  • 准备elasticsearch

      https://www.elastic.co/downloads/elasticsearch
    
  • 准备Dockerfile

      FROM elkbase:v1.0
      MAINTAINER Yampery
      VOLUME [ "/opt/product/data/" ]
      ADD ./tools /opt/product/
      RUN useradd elk && chown -R elk:elk /opt/product/elasticsearch-6.0.0
      ADD build.sh /root/
      RUN chmod +x /root/build.sh
      EXPOSE 9200
      EXPOSE 9300
      ENTRYPOINT ["/root/build.sh"]
      CMD ["/usr/sbin/init"]
    
  • 准备脚本build.sh

      #!/bin/bash
      echo "* soft nofile 65536" > /etc/security/limits.conf 
      echo "* hard nofile 131072" > /etc/security/limits.conf
      echo "* soft nproc 2048" > /etc/security/limits.conf
      echo "* hard nproc 4096" > /etc/security/limits.conf
      echo "vm.max_map_count=655360" > /etc/sysctl.conf 
      sysctl -p
      cd /opt/product/elasticsearch-6.0.0/config/
      rm -rf elasticsearch.yml
      cp /opt/product/data/elk/elasticsearch.yml .
      chown -R elk:elk /opt/product/elasticsearch-6.0.0
      chown -R elk:elk /opt/product/data/elk/elasticsearchdata
      su - elk <
  • 构建目录结构

      |-elk
      	|-elasticsearch
      		|-tools
      		|--Dockerfile
      		|--build.sh
    
  • 构建镜像

      elk/elasticsearch$ docker build -f elasticsearch:v1.0 .
    
  • 配置

      在/opt/product/data目录下建立elk目录,并拷贝elasticsearch.yml文件到该目录
      path.data: /opt/product/data/elk/elasticsearchdata
      network.host: 0.0.0.0
      在/opt/product/data/elk目录 创建 elasticsearchdata 目录
    
  • 运行容器

      docker run --name elasticsearch --privileged --restart=always -d -ti -v /opt/product/data:/opt/product/data -p 9200:9200 -p 9300:9300 elasticsearch:v1.0 /bin/bash
    

    -> 访问:ip:9200/

logstash


  • 准备logstash

      https://www.elastic.co/downloads/logstash	
    
  • 准备Dockerfile

      FROM elkbase:v1.0
      MAINTAINER Yampery
      VOLUME [ "/opt/product/data/" ]
      ADD ./tools /opt/product/
      ADD build.sh /root/
      RUN chmod +x /root/build.sh
      EXPOSE 5044
      EXPOSE 4560
      EXPOSE 8080
      ENTRYPOINT ["/root/build.sh"]
      CMD ["/usr/sbin/init"]
    
  • 准备脚本 build.sh

      #!/bin/bash
      export JAVA_HOME=/opt/product/tools/jdk-9.0.1
      export PATH=$JAVA_HOME/bin:$PATH
      JAVA_OPTS="$JAVA_OPTS -Dfile.encoding=UTF8  -Duser.timezone=GMT+08"
      cd /opt/product/logstash-6.0.0/config/
      rm -rf logstash.yml
      cp /opt/product/data/elk/logstash.yml logstash.yml
      /opt/product/logstash-6.0.0/bin/logstash -f /opt/product/data/elk/logstash.conf
    
  • 目录结构

      |-elk
      	|-logstash
      		|-tools
      		|--Dockerfile
      		|--build.sh
    
  • 构建镜像

      elk/logstash$ docker build -f logstash:v1.0 .
    
  • 配置logstash

      // 在/opt/product/data/elk目录下创建logstash.conf
      input {     
         beats {
      	port => "5044"
         }
      }
      output {
         elasticsearch {
      	   hosts => ["elasticsearch的ip:9200"]
      	   index => "logstash-tomcat-accesslog-%{+YYYY.MM.dd}"
      	}       
      }
      // 将logstash本身的logstash.yml 拷贝到/opt/product/data/elk目录下
      // 在/opt/product/data/elk目录下建立 logstashdata目录
    
  • 启动

      docker run --name logstash --restart=always -d -ti -v /opt/product/data:/opt/product/data -p 5044:5044 -p 4560:4560 -p 9090:9090 logstash:v1.0 /bin/bash
    

kibana


  • 准备kibana

      https://artifacts.elastic.co/downloads/kibana/kibana-6.0.0-linux-x86_64.tar.gz
    
  • 准备Dockerfile

      FROM elkbase:v1.0
      MAINTAINER Yampery
      VOLUME [ "/opt/product/data/" ]
      ADD ./tools /opt/product/
      ADD build.sh /root/
      RUN chmod +x /root/build.sh
      EXPOSE 5601
      ENTRYPOINT ["/root/build.sh"]
      CMD ["/usr/sbin/init"]
    
  • 准备脚本 build.sh

      #!/bin/bash 
      export JAVA_HOME=/opt/product/tools/jdk-9.0.1
      export PATH=$JAVA_HOME/bin:$PATH
      cd /opt/product/kibana-6.0.0/config/
      rm -rf kibana.yml
      ln -s /opt/product/data/elk/kibana.yml . 
      cd /opt/product/kibana-6.0.0/
      rm -rf data
      ln -s  /opt/product/data/elk/kibanadata /opt/product/kibana-6.0.0/data
      /opt/product/kibana-6.0.0/bin/kibana
    
  • 构建镜像

      elk/kibana$ docker build -t kibana:v1.0 .
    
  • 配置kibana

      // 复制kibana.yml到/opt/product/data/elk 下
      server.port: 5601
      server.host: "0.0.0.0"
      elasticsearch.url: "http://ip:9200"
    
  • 启动

      docker run --name kibana -d -ti -v /opt/product/data:/opt/product/data -p 5601:5601  kibana:v1.0 /bin/bash
    
  • 访问

      ip:5601/
    

FileBeat


  • 准备FileBeat

      https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.0.0-linux-x86_64.tar.gz
    
  • 准备Dockerfile

      FROM elkbase:v1.0
      MAINTAINER Yampery
      VOLUME [ "/opt/product/data/" ]
      ADD ./tools /opt/product/
      ADD build.sh /root/
      RUN chmod +x /root/build.sh
      ENTRYPOINT ["/root/build.sh"]
      CMD ["/usr/sbin/init"]
    
  • 准备脚本 build.sh

      #!/bin/bash
      export JAVA_HOME=/opt/product/tools/jdk-9.0.1
      export PATH=$JAVA_HOME/bin:$PATH
      cd /opt/product/filebeat-6.0.0/
      rm -rf filebeat.yml
      ln -s /opt/product/data/elk/filebeat.yml . 
      rm -rf data 
      ln -s /opt/product/data/elk/filebeatdata /opt/product/filebeat-6.0.0/data
      /opt/product/filebeat-6.0.0/filebeat -e -c filebeat.yml
    
  • 构建镜像

      elk/filebeat$ docker build -t filebeat:v1.0 .
    
  • 配置filebeat

      // 在/opt/product/data/elk目录下 创建filebeat.yml
      filebeat.prospectors:
        - input_type: log
      	document_type: tomcataccess
        paths:
      	 - /opt/product/data/logs/tomcat/localhost_access_log*.txt
      	 - /opt/product/data/epg2logs/tomcat/localhost_access_log*.txt
      output.logstash:
        // The Logstash hosts
        hosts: ["ip:5044"]
       
       // 在/opt/product/data/elk 目录下创建filebeatdata
    
  • 启动

      docker run --name filebeat -d -ti -v /opt/product/data:/opt/product/data filebeat:v1.0 /bin/bash
    

x-pack

  • 安装

      ./elasticsearch-plugin install x-pack
    
  • 设置密码

https://www.elastic.co/guide/en/x-pack/current/setting-up-authentication.html#set-built-in-user-passwords

./setup-passwords interactive
	Enter password for [elastic]:
	Reenter password for [elastic]:
	Enter password for [kibana]:
	Reenter password for [kibana]:
	Enter password for [logstash_system]:
	Reenter password for [logstash_system]:
	Changed password for user [kibana]
	Changed password for user [logstash_system]
	Changed password for user [elastic]
  • logstash配置

      input { stdin { } }
      output {
        elasticsearch {
          hosts => ["192.168.107.23:9200"]
          user => elastic
          password => elastic
        }
        stdout { codec => rubydebug }
      }
    

同理,kibana也可以配置,调用es接口也需要用户密码

说明


脚本和启动项参数中的/opt/product/data均以自己要设定的挂载目录一致

你可能感兴趣的:(docker,docker,elk,elasticsearch,Logstash,Filebeat)