elasticsearch+logstash+kafka配置说明

ElasticSearch是一个基于Lucene的搜索服务器。它提供了一个分布式多用户能力的全文搜索引擎，基于RESTful web接口。Elasticsearch是用Java开发的，并作为Apache许可条款下的开放源码发布，是当前流行的企业级搜索引擎。设计用于云计算中，能够达到实时搜索，稳定，可靠，快速，安装使用方便。本文档简述ElasticSearch配合kafka单机部署的一个环境，以及java快速链接ElasticSearch服务器的应用简例。

准备文件：
elasticsearch-6.4.2.tar.gz
kibana-6.4.2-x86_64.rpm
logstash-6.4.2.tar.gz
环境：
JDK 和 NodeJS
可参考：
https://www.extlight.com/2017/09/27/Elasticsearch-%E5%9F%BA%E7%A1%80%E5%85%A5%E9%97%A8/
ELK搭建环境说明：

注意点：1.服务器内的防火墙要设置关闭；
2.编辑修改各内配置文件，请注意修改时候如果在下载到windows上编辑，注意修改了编码格式，可能再上传会识别不了；

一.安装elasticsearch和启动
tar zxvf ./elasticsearch-6.4.2.tar.gz
cd elasticsearch-6.4.2
bin/elasticsearch 或 bin/elasticsearch -d # -d 表示后台启动

1.（非必须，启动有问题可参考）因为 Elasticsearch 可以执行脚本文件，为了安全性，默认不允许通过 root 用户启动服务。我们需要新创建用户名和用户组启动服务
#增加 es 组
groupadd es
#增加 es 用户并附加到 es 组
useradd es -g es -p es
#给目录权限
chown -R es:es elasticsearch-5.6.1
#使用es用户
su es

2.（非必须，启动有问题可参考）默认情况下，Elasticsearch 只允许本机访问，如果需要远程访问，需要修改其配置文件
vim config/elasticsearch.yml
network.host: 0.0.0.0

3.（非必须，启动有问题可参考）启动报错：max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
ulimit -n 65536 #需要设置在root用户下，再设置es用户
或者在/etc/security/limits.conf文件中增加以下几行：

• soft nofile 102400
• hard nofile 102400
• soft memlock unlimited
• hard memlock unlimited

4.（非必须，启动有问题可参考）启动报错：max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
vim /etc/sysctl.conf
vm.max_map_count=262144
或者root用户下执行：sysctl -w vm.max_map_count = 262144

6.验证es启动成功
访问http://10.72.66.151:9200/，返回json在界面

关闭ps aux | grep ‘elastic’ #记住是用es用户
kill -9 XXX

二.安装logstash和启动
tar zxvf ./logstash-6.4.2.tar.gz
cd logstash-6.4.2.tar.gz
bin/logstash -f log2es.conf
bin/logstash -f kafka2es.conf
cat test.conf #需要新增输入输出配置文件XXX.config
./bin/logstash -f test.conf& #启动
重启前可删除data下的rm .lock文件

test.conf:
input {
kafka{
bootstrap_servers => [“10.72.66.151:9092”]
client_id => “DataLog_ganges_CID”
group_id => “DataLog_ganges_GID”
auto_offset_reset => “latest”
consumer_threads => 3
decorate_events => true
topics => [“datalog_ganges”]
type => “datalog_ganges”
}
kafka{
bootstrap_servers => [“10.72.66.151:9092”]
client_id => “OpsLog_ganges_CID”
group_id => “OpsLog_ganges_GID”
auto_offset_reset => “latest”
consumer_threads => 6
decorate_events => true
topics => [“opslog_ganges”]
type => “opslog_ganges”
}
kafka{
bootstrap_servers => [“10.72.66.151:9092”]
client_id => “RunLog_ganges_CID”
group_id => “RunLog_ganges_GID”
auto_offset_reset => “latest”
consumer_threads => 4
decorate_events => true
topics => [“runlog_ganges”]
type => “runlog_ganges”
}
kafka{
bootstrap_servers => [“10.72.66.151:9092”]
client_id => “DataLog_CID”
group_id => “DataLog_GID”
auto_offset_reset => “latest”
consumer_threads => 3
decorate_events => true
topics => [“datalog”]
type => “datalog”
}
kafka{
bootstrap_servers => [“10.72.66.151:9092”]
client_id => “OpsLog_CID”
group_id => “OpsLog_GID”
auto_offset_reset => “latest”
consumer_threads => 6
decorate_events => true
topics => [“opslog”]
type => “opslog”
}
kafka{
bootstrap_servers => [“10.72.66.151:9092”]
client_id => “RunLog_CID”
group_id => “RunLog_GID”
auto_offset_reset => “latest”
consumer_threads => 4
decorate_events => true
topics => [“runlog”]
type => “runlog”
}
kafka{
bootstrap_servers => [“10.72.66.151:9092”]
client_id => “FileBeat_CID”
group_id => “FileBeat_GID”
auto_offset_reset => “latest”
consumer_threads => 4
decorate_events => true
topics => [“filebeatlog”]
type => “filebeatlog”
}
}

filter {
if[type] == “datalog_ganges”{
# grok{
# …
# }
json {
source => “message”
target => “message”
}
}
if[type] == “opslog_ganges”{
json {
source => “message”
target => “message”
}
}
if[type] == “runlog_ganges”{
# mutate{
# …
# }
# json {
# source => “message”
# target => “message”
# }
}
if[type] == “datalog”{
# grok{
# …
# }
json {
source => “message”
target => “message”
}
}
if[type] == “opslog”{
json {
source => “message”
target => “message”
}
}
if[type] == “runlog”{
# mutate{
# …
# }
# json {
# source => “message”
# target => “message”
# }
}
}

output {
#console output//just for debug
stdout {}

    if[type] == "datalog_ganges"{
      elasticsearch{
           hosts => ["10.72.66.151:9200"]
           #index => "%{type}-%{+YYYY.MM.dd}"
           index => "datalog_ganges"
           #timeout => 300
           #user => "elastic"
           #password => "changeme"
      }
    }
    if[type] == "opslog_ganges"{
      elasticsearch{
           hosts => ["10.72.66.151:9200"]
           #index => "%{type}-%{+YYYY.MM.dd}"
           index => "opslog_ganges"
           #timeout => 300
           #user => "elastic"
           #password => "changeme"
      }
    }
    if[type] == "runlog_ganges"{
      elasticsearch{
           hosts => ["10.72.66.151:9200"]
           #index => "%{type}-%{+YYYY.MM.dd}"
           index => "runlog_ganges"
           #timeout => 300
           #user => "elastic"
           #password => "changeme"
      }
    }
    if[type] == "datalog"{
      elasticsearch{
           hosts => ["10.72.66.151:9200"]
           #index => "%{type}-%{+YYYY.MM.dd}"
           index => "datalog"
           #timeout => 300
           #user => "elastic"
           #password => "changeme"
      }
    }
    if[type] == "opslog"{
      elasticsearch{
           hosts => ["10.72.66.151:9200"]
           #index => "%{type}-%{+YYYY.MM.dd}"
           index => "opslog"
           #timeout => 300
           #user => "elastic"
           #password => "changeme"
      }
    }
    if[type] == "runlog"{
      elasticsearch{
           hosts => ["10.72.66.151:9200"]
           #index => "%{type}-%{+YYYY.MM.dd}"
           index => "runlog"
           #timeout => 300
           #user => "elastic"
           #password => "changeme"
      }
    }
    if[type] == "filebeatlog"{
      elasticsearch{
           hosts => ["10.72.66.151:9200"]
           #index => "%{type}-%{+YYYY.MM.dd}"
           index => "filebeatlog"
           #timeout => 300
           #user => "elastic"
           #password => "changeme"
      }
   }

}

#验证成功在kafka发数据的时候，logstash可以接受到数据

三.修改kafka参数
Kafka的安装：
下载:
curl -L -O http://mirrors.cnnic.cn/apache/kafka/0.9.0.0/kafka_2.10-0.9.0.0.tgz
或
wget http://mirrors.shu.edu.cn/apache/kafka/2.0.0/kafka_2.12-2.0.0.tgz
解压：
tar zxvf ./kafka_2.12-2.0.0.tgz

1.修改kafka参数zookeeper.properties文件中的：

the directory where the snapshot is stored.

dataDir=/tmp/zookeeper/dataDir
dataLogDir=/tmp/zookeeper/log

the port at which the clients will connect

clientPort=2181

disable the per-ip limit on the number of connections since this is a non-production config

maxClientCnxns=0
tickTime=2000
initLimit=10
syncLimit=5

#集群部署模式
#server.1=10.72.66.151:2888:3888
#server.2=10.72.66.150:2888:3888

2.修改kafka参数:
server.properties中listeners=PLAINTEXT://yourhostname :9093

bin/zookeeper-server-start.sh config/zookeeper.properties &
bin/kafka-server-start.sh config/server.properties &
创建topic
bin/kafka-topics.sh --create --zookeeper webserver.novalocal:2181 --replication-factor 1 --partitions 1 --topic datalog_ganges
查看topic
bin/kafka-topics.sh --list --zookeeper webserver.novalocal:2181
往topic写数据
bin/kafka-console-producer.sh --broker-list webserver.novalocal:9092 --topic datalog_ganges
从topic读数据
bin/kafka-console-consumer.sh --bootstrap-server webserver.novalocal:9092 --topic datalog_ganges --from-beginning

删除topic
bin/kafka-topics.sh --zookeeper webserver.novalocal:2181 --delete --topic datalog_ganges
bin/kafka-topics.sh --zookeeper webserver.novalocal:2181 --delete --topic opslog_ganges

关闭服务
bin/kafka-server-stop.sh
四.安装kibana（辅助查看ES数据）
详见https://www.elastic.co/guide/cn/kibana/current/rpm.html

五.安装elasticsearch-head-master（辅助查看问题）
一般master如果可以链接上kafka数据，就说嘛logstash和kafka搭建无误。
wget https://github.com/mobz/elasticsearch-head/archive/master.zip
unzip master.zip
cd elasticsearch-head-master
npm install
npm run start
linux下npm安装
curl --silent --location https://rpm.nodesource.com/setup_10.x | bash -
yum install -y nodejs
npm install -g cnpm --registry=https://registry.npm.taobao.org
npm install
npm run build

原始参考资料：
http://note.youdao.com/noteshare?id=1d552c0ad8764b5881e4c6acae448629&sub=043A3E7185884562B8D038E777EE6055

http://note.youdao.com/noteshare?id=dd1674ad9a5e3517b41dc63d50c673fd&sub=DB7392893FA34E01959FB639433620F9