2、Elasticsearch介绍
2.1、什么是搜索?
- 百度
- 京东
2.2、如果用数据库做搜索会怎样
- 天气
- 关键词
- 全表扫描
2.3、什么是全文搜索、倒排索引和lucene?
1.老男孩教育
2.老男孩教育linux学院
3.老男孩教育python学院
4.老男孩教育DBA
5.老男孩教育oldzhang
老 1,2,3,4,5
老男孩 1,2,3,4,5
教育 1,2,3,4,5
学院 2,3
linux 2
python 3
DBA 4
oldzhang 5
2.4、Elasticsearch功能
1)分布式的搜索引擎和数据分析引擎
2)针对电商搜索
3)对海量数据进行近实时处理
2.5、特点
1)搜索
2)全文搜索
3)分析数据
4)处理海量数据PB
5)高可用高性能分布式搜索引擎数据库
2.6、应用场景
1)网页搜索
2)新闻搜索
3)商品标签
4)日志收集分析展示
3、安装部署
1)安装java
yum install -y java-1.8.0-openjdk.x86_64
2) 下载安装软件
mkdir -p /data/es_soft/
cd /data/es_soft/
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.6.0.rpm
#yum install elasticsearch-6.6.0.rpm -y
rpm -ivh elasticsearch-6.6.0.rpm
3) 配置启动
systemctl daemon-reload
systemctl enable elasticsearch.service
systemctl start elasticsearch.service
systemctl status elasticsearch.service
systemctl is-active elasticsearch.service
4)检查是否启动成功
ps -ef|grep elastic
lsof -i:9200
curl localhost:9200
5)查看配置文件位置
rpm -qc elasticsearch
/etc/elasticsearch/elasticsearch.yml ---- 主配置文件
/etc/elasticsearch/jvm.options -----java虚拟机配置
/etc/init.d/elasticsearch ----init.d启动脚本
/etc/sysconfig/elasticsearch ----环境变量相关信息,不需要修改
/usr/lib/sysctl.d/elasticsearch.conf ----环境变量相关
/usr/lib/systemd/system/elasticsearch.service ----systemd启动脚本
6) 调整jvm配置
/etc/elasticsearch/jvm.options
-Xms1g
-Xmx1g
7)调整elasticsearch配置
/etc/elasticsearch/elasticsearch.yml
[root@db01 ~]# grep "^[a-z]" /etc/elasticsearch/elasticsearch.yml
node.name: node-1
path.data: /data/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 10.0.0.51
http.port: 9200
mkdir /data/elasticsearch/ -p
chown -R elasticsearch:elasticsearch /data/elasticsearch/
systemctl edit elasticsearch
[Service]
LimitMEMLOCK=infinity
systemctl daemon-reload
systemctl start elasticsearch
4、名词概念
index 库
type 表
fields 字段
doc 行数据
5、交互方式
RESTful API
curl命令:
最繁琐
最复杂
最容易出错
不需要安装任何软件,只需要有curl命令
es-head插件
查看数据方便
操作相对容易
需要node环境
kibana
查看数据以及报表格式丰富
操作很简单
需要java环境和安装配置kibana
6、es-head安装部署
yum install nodejs npm openssl screen -y
node -v
npm -v
npm install -g cnpm --registry=https://registry.npm.taobao.org
tar zxvf elasticsearch-head.tar.gz -C /opt/
cd /opt/elasticsearch-head
npm run start &
修改es配置文件
vim /etc/elasticsearch/elasticsearch.yml
http.cors.enabled: true
http.cors.allow-origin: "*"
重启es
systemctl restart elasticsearch
网页打开
10.0.0.51:9100
输入
http://10.0.0.51:9200
6.1、插入数据
curl -XPUT 'localhost:9200/vipinfo/user/2?pretty' -H 'Content-Type: application/json' -d' {
"first_name": "Jane",
"last_name" : "Smith",
"age" : 32,
"about" : "I like to collect rock albums", "interests": [ "music" ]
}'
curl -XPUT 'localhost:9200/vipinfo/user/3?pretty' -H 'Content-Type: application/json' -d' {
"first_name": "Jane",
"last_name" : "Smith",
"age" : 32,
"about" : "I like to collect rock albums", "interests": [ "music" ]
}'
curl -XPOST 'localhost:9200/vipinfo/user?pretty' -H 'Content-Type: application/json' -d' {
"first_name": "Jane",
"last_name" : "Smith",
"age" : 32,
"about" : "I like to collect rock albums", "interests": [ "music" ]
}'
curl -XPUT 'localhost:9200/vipinfo/user/2?pretty' -H 'Content-Type: application/json' -d' {
"first_name": "zhang",
"last_name" : "ya",
"age" : 18,
"about" : "I like to collect rock albums", "interests": [ "music" ]
}'
curl -XPUT 'localhost:9200/vipinfo/user/3?pretty' -H 'Content-Type: application/json' -d' {
"first_name": "ya",
"last_name" : "zhang",
"age" : 28,
"about" : "I like to collect rock albums", "interests": [ "music" ]
}'
curl -XPOST 'localhost:9200/vipinfo/user?pretty' -H 'Content-Type: application/json' -d' {
"first_name": "老男孩Linux",
"last_name" : "学院",
"age" : 8,
"about" : "I like to collect rock albums", "interests": [ "music" ]
}'
6.2、条件查询数据
1. 查询索引中所有的
curl -XGET 'localhost:9200/vipinfo/user/1?pretty'
2. 查询指定文档数据
curl -XGET 'localhost:9200/vipinfo/user/1?pretty'
curl -XGET 'localhost:9200/vipinfo/user/2?pretty’
3. 按条件查询文档数据
curl -XGET 'localhost:9200/vipinfo/user/_search?q=last_name:ya&pretty'
curl -XGET 'localhost:9200/vipinfo/user/_search?q=age:28&pretty'
4. 使用Query-string查询
curl -XGET 'localhost:9200/vipinfo/user/_search?pretty' -H 'Content-Type: application/json' -d'
{
"query" : {
"match" : {
"last_name" : "ya"
}
}
}
'
5.更新数据的两种方式
1)PUT更新,需要填写完整的信息
curl -XPUT 'localhost:9200/vipinfo/user/4?pretty' -H 'Content-Type: application/json' -d'
{
"first_name" : "zhang",
"last_name": "ya",
"age" : 27,
"about" : "i love my cat", "interests": [ "sports", "music" ]
}
2)POST更新,只需要填写更改的信息
curl -XPOST 'localhost:9200/vipinfo/user/4?pretty' -H 'Content-Type: application/json' -d'
{
"age" : 29
}
curl -XPOST 'localhost:9200/linux58/linux?pretty' -H 'Content-Type: application/json' -d' {
"id" : 1 ,
"first_name": "zhang",
"last_name" : "ya",
"age" : 18,
"group" : 1
}'
curl -XPOST 'localhost:9200/linux58/linux?pretty' -H 'Content-Type: application/json' -d' {
"id" : 2 ,
"first_name": "li",
"last_name" : "banzhang",
"age" : 18,
"group" : 10
}'
curl -XPOST 'localhost:9200/linux58/linux?pretty' -H 'Content-Type: application/json' -d' {
"id" : 3 ,
"first_name": "what",
"last_name" : "ao",
"age" : 17,
"group" : 13
}'
6.3、删除数据
curl -XDELETE 'localhost:9200/vipinfo/user/1?pretty’
{
"_index" : "vipinfo",
"_type" : "user",
"_id" : "1",
"_version" : 2,
"result" : "deleted",
"_shards" : {
"total" : 2,
"successful" : 1,
"failed" : 0
},
"_seq_no" : 1,
"_primary_term" : 2
}
6.4、删除索引
curl -XDELETE 'localhost:9200/vipinfo?pretty'
7、集群(所有节点执行)
1) 下载安装软件
mkdir -p /data/es_soft/
cd /data/es_soft/
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.6.0.rpm
#yum install elasticsearch-6.6.0.rpm -y
rpm -ivh elasticsearch-6.6.0.rpm
2)配置配置文件
[root@db01 /server/tools]# cat /etc/elasticsearch/elasticsearch.yml
cluster.name: linux58
node.name: node-1
path.data: /data/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 10.0.0.51,127.0.0.1
http.port: 9200
discovery.zen.ping.unicast.hosts: ["10.0.0.51", "10.0.0.52"]
discovery.zen.minimum_master_nodes: 1
http.cors.enabled: true
http.cors.allow-origin: "*"
[root@db02 /etc/elasticsearch]# cat /etc/elasticsearch/elasticsearch.yml
cluster.name: linux58
node.name: node-2
path.data: /data/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 10.0.0.52,127.0.0.1
http.port: 9200
discovery.zen.ping.unicast.hosts: ["10.0.0.51", "10.0.0.52"]
discovery.zen.minimum_master_nodes: 1
http.cors.enabled: true
http.cors.allow-origin: "*"
[root@db03 /etc/elasticsearch]# cat /etc/elasticsearch/elasticsearch.yml
cluster.name: linux58
node.name: node-3
path.data: /data/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 10.0.0.53,127.0.0.1
http.port: 9200
discovery.zen.ping.unicast.hosts: ["10.0.0.51", "10.0.0.52", "10.0.0.53"]
discovery.zen.minimum_master_nodes: 1
http.cors.enabled: true
http.cors.allow-origin: "*"
3)创建数据目录并授权
mkdir /data/elasticsearch/ -p
chown -R elasticsearch:elasticsearch /data/elasticsearch/
4) 配置启动
systemctl daemon-reload
systemctl enable elasticsearch.service
systemctl start elasticsearch.service
systemctl status elasticsearch.service
systemctl is-active elasticsearch.service
5)检查是否启动成功
ps -ef|grep elastic
lsof -i:9200
curl localhost:9200
6)添加
systemctl edit elasticsearch
[Service]
LimitMEMLOCK=infinity
systemctl daemon-reload
systemctl restart elasticsearch
curl 10.0.0.51:9200
curl 10.0.0.52:9200
curl 10.0.0.53:9200
7.1、调整限制
限定条件
1.索引一旦创建以后,分片数就固定死了,不能调整了
2.但是副本数可以动态调整
curl -XPUT 'localhost:9200/index2/_settings?pretty' -H 'Content-Type: application/json' -d'
{
"settings" : {
"number_of_replicas" : 2
}
}'
curl -XPUT 'localhost:9200/index2/_settings?pretty' -H 'Content-Type: application/json' -d'
{
"settings" : {
"number_of_replicas" : 0
}
}'
7.2、查看节点数
curl -XGET 'http://localhost:9200/_cat/nodes?human&pretty'
增加index和设置分片、副本
curl -XPUT 'localhost:9200/index2?pretty' -H 'Content-Type: application/json' -d'
{
"settings" : {
"number_of_shards" : 3,
"number_of_replicas" : 1
}
}'
8、安装kibana
1)下载安装
2)修改配置文件
[root@db01 /data/soft]# grep "^[a-z]" /etc/kibana/kibana.yml
server.port: 5601
server.host: "10.0.0.51"
server.name: "db01"
elasticsearch.hosts: ["http://localhost:9200"]
3)网站连接elasticsearch
8.1、安装分词器(1.版本要对应 2.所有的ES节点都需要安装 3.所有es节点都需要重启)
未使用分词器之前查询中文数据
curl -XPOST http://localhost:9200/index2/fulltext/1 -H 'Content-Type:application/json' -d'
{"content":"美国留给伊拉克的是个烂摊子吗"}
'
curl -XPOST http://localhost:9200/index2/fulltext/2 -H 'Content-Type:application/json' -d'
{"content":"公安部:各地校车将享最高路权"}
'
curl -XPOST http://localhost:9200/index2/fulltext/3 -H 'Content-Type:application/json' -d'
{"content":"中韩渔警冲突调查:韩警平均每天扣1艘中国渔船"}
'
curl -XPOST http://localhost:9200/index2/fulltext/4 -H 'Content-Type:application/json' -d'
{"content":"中国驻洛杉矶领事馆遭亚裔男子枪击 嫌犯已自首"}
'
curl -XPOST http://localhost:9200/index2/fulltext/_search?pretty -H 'Content-Type:application/json' -d'
{
"query" : { "match" : { "content" : "中国" }},
"highlight" : {
"pre_tags" : ["", ""],
"post_tags" : ["", ""],
"fields" : {
"content" : {}
}
}
}
1)安装
cd /usr/share/elasticsearch/bin
./elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v6.6.0/elasticsearch-analysis-ik-6.6.0.zip
2)重启
systemctl restart elasticsearch
8.1.1、分词器测试
创建索引
curl -XPUT http://localhost:9200/index
创建映射
curl -XPOST http://localhost:9200/index/fulltext/_mapping -H 'Content-Type:application/json' -d'
{
"properties": {
"content": {
"type": "text",
"analyzer": "ik_max_word",
"search_analyzer": "ik_max_word"
}
}
}'
创建一些文档
curl -XPOST http://localhost:9200/index/fulltext/1 -H 'Content-Type:application/json' -d'
{"content":"美国留给伊拉克的是个烂摊子吗"}
'
curl -XPOST http://localhost:9200/index/fulltext/2 -H 'Content-Type:application/json' -d'
{"content":"公安部:各地校车将享最高路权"}
'
curl -XPOST http://localhost:9200/index/fulltext/3 -H 'Content-Type:application/json' -d'
{"content":"中韩渔警冲突调查:韩警平均每天扣1艘中国渔船"}
'
curl -XPOST http://localhost:9200/index/fulltext/4 -H 'Content-Type:application/json' -d'
{"content":"中国驻洛杉矶领事馆遭亚裔男子枪击 嫌犯已自首"}
'
查询
curl -XPOST http://localhost:9200/index/fulltext/_search?pretty -H 'Content-Type:application/json' -d'
{
"query" : { "match" : { "content" : "中国" }},
"highlight" : {
"pre_tags" : ["", ""],
"post_tags" : ["", ""],
"fields" : {
"content" : {}
}
}
}
'