ElasticSearch配置文档(v1.7.2)

ElasticSearch

安装java环境

• sudo apt-get update
• java -version
• sudo apt-get install default-jre
• sudo apt-get install default-jdk
• （或者： yum -y install java-1.7.0-openjdk* ）

设置JAVA_HOME

• sudo update-alternatives --config java（返回YOUR_PATH）
• sudo nano /etc/environment
• JAVA_HOME="YOUR_PATH"
• source /etc/environment
• echo $JAVA_HOME

安装ElasticSearch（以1.7.2为例）

• wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.2.deb(yum 下载rpm包)
• sudo dpkg -i elasticsearch-1.7.2.deb  (sudo  rpm -ivh *.rpm)
• sudo update-rc.d elasticsearch defaults   (chkconfig –-add elasticsearch     ;     chkconfig elasticsearch on )

启动ElasticSearch

• sudo service elasticsearch start
• elasticsearch 结构目录位置： /var/lib/elasticsearch

配置ElasticSearch

• 配置文件地址：/etc/elasticsearch/elasticsearch.yml
• 配置文件详解：http://www.cnblogs.com/sunxucool/p/3799190.html

配置防火墙

• sudo ufw disable
• sudo ufw allow from 192.168.1.141
• sudo ufw enable

或者防火墙

 

#/usr/sbin/iptables -I INPUT -p tcp --dport 80 -j ACCEPT

#/usr/sbin/iptables -I INPUT -p tcp --dport 22 -j ACCEPT

#/usr/sbin/iptables -I INPUT -p tcp --dport 3306 -j ACCEPT

#/usr/sbin/iptables -I INPUT -p tcp --dport 8080 -j ACCEPT

 

/usr/sbin/iptables -I INPUT -p tcp --dport 9200 -j DROP

/usr/sbin/iptables -I INPUT -s 10.44.136.154 -p tcp --dport 9200 -j ACCEPT

/usr/sbin/iptables -I INPUT -s 某个ip -p tcp --dport 9200 -j ACCEPT

/usr/sbin/iptables -I INPUT -s 某个ip -p tcp --dport 9200 -j ACCEPT

 

 

/usr/sbin/service iptables save

创建快照

• elasticsearch.yml添加： path.repo: ["/mount/backups", "/mount/longterm_backups"]
• 创建目录： /mount/backups/my_backup,  /mount/longterm_backups
• 添加权限： sudo chmod -R 777 /mount
• 重启elasticsearch服务： sudo service elasticsearch restart
• 创建仓库，并指定仓库类型：

curl -XPUT 'http://localhost:9200/_snapshot/my_backup' -d '

{ 

"type": "fs", 

"settings": { 

                "location": "/mount/backups/my_backup",

                "compress": true 

}

}'

• 创建第一个快照：curl -XPUT http://localhost:9200/_snapshot/my_backup/snapshot_1?wait_for_completion=true

• 查看所有快照： curl -XGET http://localhost:9200/_snapshot/my_backup/_all
• 删除一个快照： curl -XDELETE http://localhost:9200/_snapshot/my_backup/snapshot_1
• 使用快照： 

 

curl -XPOST http://localhost:9200/_snapshot/my_backup/snapshot_1/_restore -d '

{

"indices": "index_1,index_2", 

"ignore_unavailable": "true",

"include_global_state": false, 

"rename_pattern": "index_(.+)", 

"rename_replacement": "restored_index_$1"

}'

配置ik分词

• 版本参考

IK version ES version

master	2.1.0 -> master
1.6.0	2.1.0
1.5.0	2.0.0
1.4.1	1.7.2
1.4.0	1.6.0
1.3.0	1.5.0
1.2.9	1.4.0
1.2.8	1.3.2
1.2.7	1.2.1
1.2.6	1.0.0
1.2.5	0.90.2
1.2.3	0.90.2
1.2.0	0.90.0
1.1.3	0.20.2
1.1.2	0.19.x
1.0.0	0.16.2 -> 0.19.0

• 克隆ik库： https://github.com/medcl/elasticsearch-analysis-ik.git

• checkout对应tag：  git checkout tags/v1.4.1 -b v1.4.1

• 安装maven： (略)

• 编译jar包： sudo mvn package

• copy ik文件夹到/etc/elasticsearch/: sudo cp config/ik /etc/elasticsearch/ 

• copy jar到/usr/share/elasticsearch/lib/: sudo cp target/*.jar /usr/share/elasticsearch/lib

• 配置elasticsearch.yml

index:
  analysis:
    analyzer:
      ik:
        alias: [ik_analyzer]
        type: org.elasticsearch.index.analysis.IkAnalyzerProvider
      ik_max_word:
        type: ik
        use_smart: false
      ik_smart:
        type: ik
        use_smart: true

index.analysis.analyzer.default.type : "ik"

• 创建index:  

curl -XPUT http://localhost:9200/index

• 创建mapping

curl -XPOST http://localhost:9200/index/fulltext/_mapping -d'
{
 "fulltext": {
 "_all": {
 "analyzer": "ik_max_word",
 "search_analyzer": "ik_max_word",
 "term_vector": "no",
 "store": "false"
 },
 "properties": {
 "content": {
 "type": "string",
 "store": "no",
 "term_vector": "with_positions_offsets",
 "analyzer": "ik_max_word",
 "search_analyzer": "ik_max_word",
 "include_in_all": "true",
 "boost": 8
 }
 }
 }
}'

  mapping:

 

curl -XPUT localhost:9200/feiliwu -d '{

"mappings":{

     "product":{

            "_timestamp":{

                    "enables": true

             }

      }

}

 

}'

• 录入数据

curl -XPOST http://localhost:9200/index/fulltext/1 -d'
{"content":"美国留给伊拉克的是个烂摊子吗"}
'

curl -XPOST http://localhost:9200/index/fulltext/2 -d'
{"content":"公安部：各地校车将享最高路权"}
'

curl -XPOST http://localhost:9200/index/fulltext/3 -d'
{"content":"中韩渔警冲突调查：韩警平均每天扣1艘中国渔船"}
'

curl -XPOST http://localhost:9200/index/fulltext/4 -d'
{"content":"中国驻洛杉矶领事馆遭亚裔男子枪击 嫌犯已自首"}
'

• 查询

curl -XPOST http://localhost:9200/index/fulltext/_search  -d'
{
 "query" : { "term" : { "content" : "中国" }},
 "highlight" : {
 "pre_tags" : ["<tag1>", "<tag2>"],
 "post_tags" : ["</tag1>", "</tag2>"],
 "fields" : {
 "content" : {}
 }
 }
}
'

 

安装Marvel

• run sudo ./bin/plugin -i elasticsearch/marvel/latest
• 浏览器访问：http://localhost:9200/_plugin/marvel/

参考资料

• How To Install Java on Ubuntu with Apt-Get
• How To Install and Configure Elasticsearch on Ubuntu 14.04
• Setting up a Secure Single Node Elasticsearch server behind Nginx：
• Elasticsearch 权威指南（中文版）
• UFW防火墙简单设置
• Ufw使用指南
• Snapshot And Restore
• elasticsearch-analysis-ik
• Linux下安装maven
• ElasticSearch入门笔记
• ElasticSearch中文社区

      Logstash

下载安装

• wget https://download.elastic.co/logstash/logstash/packages/debian/logstash_1.5.4-1_all.deb
• sudo dpkg -i logstash_1.5.4-1_all.deb
• sudo update-rc.d logstash defaults   

配置jdk

• sudo nano  /etc/profile

 

export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64

export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar

export PATH=$PATH:$JAVA_HOME/bin

export LOGSTASH_HOME=/opt/logstash

export LOGSTASH=$LOGSTASH_HOME/bin

export PATH=$LOGSTASH:$PATH

 

• source /etc/profile

  配置logstash

• sudo nano /etc/logstash/conf.d/logstash-simple.conf

 

 

 

input {

 

  tcp {

    port => 5000

    type => syslog

  }

  udp {

    port => 5000

    type => syslog

  }

}

filter {

  if [type] == "syslog" {

    grok {

      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }

      add_field => [ "received_at", "%{@timestamp}" ]

      add_field => [ "received_from", "%{host}" ]

    }

    syslog_pri { }

    date {

      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]

    }

  }

}

output {

    elasticsearch {

        protocol => "http"

        host => "localhost:9200"

    }

    stdout { }

}

 

 

 

 

启动logstash

• sudo service logstash status
• sudo service logstash start

查看log

• tail -f /var/log/logstash/logstash.log

参考资料

• ELKstack 中文指南
• logstash&node.js&elk&日志分析
• logstash download

Kibana

建立使用kibana的分组

• sudo groupadd -g 999 kibana 
• sudo useradd -u 999 -g 999 kibana
• If those commands fail because the999GID or UID already exist, replace the number with IDs that are free

下载安装

• cd ~ 
• wget https://download.elastic.co/kibana/kibana/kibana-4.1.2-linux-x64.tar.gz
  
• tar xvf kibana-*.tar.gz

配置

• sudo nano ~/kibana-4*/config/kibana.yml（server.host: "localhost"）
• sudo mkdir -p /opt/kibana
• sudo cp -R ~/kibana-4*/* /opt/kibana/
  
• sudo chown -R kibana: /opt/kibana

run as a service

• cd /etc/init.d && sudo curl -o kibana https://gist.githubusercontent.com/thisismitch/8b15ac909aed214ad04a/raw/fc5025c3fc499ad8262aff34ba7fde8c87ead7c0/kibana-4.x-init
• cd /etc/default && sudo curl -o kibana https://gist.githubusercontent.com/thisismitch/8b15ac909aed214ad04a/raw/fc5025c3fc499ad8262aff34ba7fde8c87ead7c0/kibana-4.x-default
• sudo chmod +x /etc/init.d/kibana
• sudo update-rc.d kibana defaults 96 9
  
• sudo service kibana start
• visit localhost:5601

Install Nginx

• sudo apt-get install nginx apache2-utils(sudo yum install nginx httpd)
• sudo htpasswd -c /etc/nginx/htpasswd.users kibanaadmin 
  
• sudo vi /etc/nginx/sites-available/default

 

 

 

server {

listen 80;

server_name localhost;

auth_basic "Restricted Access";

auth_basic_user_file /etc/nginx/htpasswd.users;

location / {

proxy_pass http://localhost:5601;

proxy_http_version 1.1;

proxy_set_header Upgrade $http_upgrade;

proxy_set_header Connection 'upgrade';

proxy_set_header Host $host;

proxy_cache_bypass $http_upgrade;

}

}

• sudo service nginx restart

Configure An Index Pattern

• Go to Settings → Advanced.
• Edit the metaFields and add "_timestamp". Hit save.
• Now go back to Settings → Indices 

• create 

参考资料

• kibana download
• How To Install Elasticsearch, Logstash, and Kibana (ELK Stack) on Ubuntu 14.04
• configure an index pattern