Docker之docker-compose一键部署 Elasticsearch Logstash Kibana

1. 需求背景

Elasticsearch、Logstash、Kibana三个中间件在有点规模的项目中都会使用到，尤其是涉及到站内搜索、日志采集等需求时。在开发和测试环境，我们需要快速部署和管理给开发人员使用，那么 docker-compose一键部署最适合不过了。

2. 部署

端口说明

• Elasticsearch：9200（HTTP通信） 和 9300（TCP通信）
• Kibana：5601
• Logstash：5044 和 9600 （TCP通信）

2.1 目录说明

[root@ecs elk]# tree -L 3
.
├── config
│   ├── kibana.yml
│   └── logstash.conf
├── docker-compose.yml
└── elasticsearch
    └── data
        └── nodes

执行命令：

• 创建elk的容器网络：docker network create elk_net
• mkdir -p elasticsearch/data/ && chmod 777 elasticsearch/data/
• touch kibana.yml logstash.conf (配置如下)
• touch docker-compose.yml (配置如下)

2.2 配置文件

2.2.1 docker-compose.yml

version: '3'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.16.2
    container_name: elasticsearch_server
    restart: always
    environment:
      - discovery.type=single-node
      - discovery.zen.minimum_master_nodes=1
      - ES_JAVA_OPTS=-Xms3g -Xmx3g
    volumes:
      - ./elasticsearch/data:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
      - 9300:9300 
    networks:
      elk_net:     # 指定使用的网络
        aliases:
          - elasticsearch     # 该容器的别名，在 elk_net 网络中的其他容器可以通过别名 elasticsearch 来访问到该容器

  kibana:
    image: docker.elastic.co/kibana/kibana:7.16.2
    container_name: kibana_server
    ports:
      - "5601:5601"
    restart: always
    networks:
      elk_net:
        aliases:
          - kibana
    environment:
      - ELASTICSEARCH_URL=http://elasticsearch:9200
      - SERVER_NAME=kibana
    # 如需具体配置，可以创建./config/kibana.yml，并映射
    # volumes:
    #   - ./config/kibana.yml:/usr/share/kibana/config/kibana.yml
    depends_on:
      - elasticsearch

  logstash:
    image: docker.elastic.co/logstash/logstash:7.16.2
    container_name: logstash_server
    restart: always
    environment:
      - LS_JAVA_OPTS=-Xmx256m -Xms256m
    volumes:
      - ./config/logstash.conf:/etc/logstash/conf.d/logstash.conf
    networks:
      elk_net:
        aliases:
          - logstash
    depends_on:
      - elasticsearch
    entrypoint:
      - logstash
      - -f
      - /etc/logstash/conf.d/logstash.conf
    logging:
      driver: "json-file"
      options:
        max-size: "200m"
        max-file: "3"

networks:
  elk_net:
    external:
      name: elk_net

2.2.2 elasticsearch.yml（容器内 config/elasticsearch.yml）

cluster.name: "docker-cluster"
network.host: 0.0.0.0
http.cors.enabled: true
http.cors.allow-origin: "*"
http.cors.allow-headers: Authorization
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true

2.2.3 kibana.yml

server.host: "0.0.0.0"
server.shutdownTimeout: "5s"
elasticsearch.hosts: [ "http://elasticsearch:9200" ]
monitoring.ui.container.elasticsearch.enabled: true
elasticsearch.username: "elastic"
elasticsearch.password: "123456"

2.2.4 logstash.yml （容器内 config/logstash.yml）

http.host: "0.0.0.0"
xpack.monitoring.elasticsearch.hosts: [ "http://elasticsearch:9200" ]
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: 123456

2.2.5 logstash.conf （取存日志到es）

以下为示例，具体的根据自己需要进行配置。

input {
  file {
    path => "/data/nginx/logs/access.log"
  }
}
 
filter {
  mutate {
    add_field => [ "[fields][path]", "%{[path]}"]
	add_field => [ "message]", "%{[message]}"]
  }
}
 
output {
  elasticsearch {
    hosts => ["http://elasticsearch:9200"]
	index => "test-logstash"
  }
}

2.3 设置 & 配置密码

2.3.1 elasticsearch

使用上述配置重启容器后，进入容器执行以下命令，可配置elastic、kibana、logstash_system 等账号的密码： elasticsearch-setup-passwords interactive

root@9dfeeda019ef:/usr/share/elasticsearch# elasticsearch-setup-passwords interactive
Initiating the setup of passwords for reserved users elastic,apm_system,kibana,kibana_system,logstash_system,beats_system,remote_monitoring_user.
You will be prompted to enter passwords as the process progresses.
Please confirm that you would like to continue [y/N]Y

Enter password for [elastic]: 
Reenter password for [elastic]: 
Enter password for [apm_system]: 
Reenter password for [apm_system]: 
Enter password for [kibana_system]: 
Reenter password for [kibana_system]: 
Enter password for [logstash_system]: 
Reenter password for [logstash_system]: 
Passwords do not match.
Try again.
Enter password for [logstash_system]: 
Reenter password for [logstash_system]: 
Enter password for [beats_system]: 
Reenter password for [beats_system]: 
Enter password for [remote_monitoring_user]: 
Reenter password for [remote_monitoring_user]: 
Changed password for user [apm_system]
Changed password for user [kibana_system]
Changed password for user [kibana]
Changed password for user [logstash_system]
Changed password for user [beats_system]
Changed password for user [remote_monitoring_user]
Changed password for user [elastic]

设置完成后,登录Kibana的账户就是 kibana , elasticsearch的账户为 elastic 。

2.3.2 kibana.yml 设置连接es密码，并重启容器

elasticsearch.username: "elastic"
elasticsearch.password: "123456"

2.3.3 logstash.yml 设置连接es密码，并重启容器

xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: 123456

2.4 遇到的坑

2.4.1 宿主机权限不足

./elasticsearch/data目录权限不够会有以下异常：

elasticsearch_server | uncaught exception in thread [main]
elasticsearch_server | {"type": "server", "timestamp": "2022-02-24T02:16:54,440Z", "level": "ERROR", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "docker-cluster", "node.name": "59a11061b6c4", "message": "uncaught exception in thread [main]", 
elasticsearch_server | "stacktrace": ["org.elasticsearch.bootstrap.StartupException: ElasticsearchException[failed to bind service]; nested: AccessDeniedException[/usr/share/elasticsearch/data/nodes];",
elasticsearch_server | "at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.16.2.jar:7.16.2]",
elasticsearch_server | "at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157) ~[elasticsearch-7.16.2.jar:7.16.2]",
elasticsearch_server | "at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77) ~[elasticsearch-7.16.2.jar:7.16.2]",
elasticsearch_server | "at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) ~[elasticsearch-cli-7.16.2.jar:7.16.2]",
elasticsearch_server | "at org.elasticsearch.cli.Command.main(Command.java:77) ~[elasticsearch-cli-7.16.2.jar:7.16.2]",
elasticsearch_server | "at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122) ~[elasticsearch-7.16.2.jar:7.16.2]",
elasticsearch_server | "at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) ~[elasticsearch-7.16.2.jar:7.16.2]",
elasticsearch_server | "Caused by: org.elasticsearch.ElasticsearchException: failed to bind service",
elasticsearch_server | "at org.elasticsearch.node.Node.(Node.java:1090) ~[elasticsearch-7.16.2.jar:7.16.2]",
elasticsearch_server | "at org.elasticsearch.node.Node.(Node.java:309) ~[elasticsearch-7.16.2.jar:7.16.2]",
elasticsearch_server | "at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:234) ~[elasticsearch-7.16.2.jar:7.16.2]",
elasticsearch_server | "at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234) ~[elasticsearch-7.16.2.jar:7.16.2]",
elasticsearch_server | "at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) ~[elasticsearch-7.16.2.jar:7.16.2]",
elasticsearch_server | "at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) ~[elasticsearch-7.16.2.jar:7.16.2]",

解决：chmod 777 elasticsearch/data/

3. 总结

1. 梳理好部署目录，创建对应的文件，直接docker-compose up -d 启动。
2. 内网使用Elasticsearch、Logstash时，可以不设置密码，但kibana需要设置（可以通过yml配置或者nginx代理实现）。
3. 生产环境可以使用公有云上的 ES产品，开发和测试环境可以一键部署。