ElasticSearch ik,elasticsearch-jdbc 使用 和 yii2 实例

为什么80%的码农都做不了架构师？>>>   

1.ES安装+ik插件

mac安装和使用:

a.由于ES为java开发 所以首先安装下载jdk,由于最新版本对jdk要求>1.8.0,所以可以通过brew 升级jdk

brew cask install caskroom/versions/java8

b.brew 安装ES

$ brew update
$ brew search elasticsearch
elasticsearch            [email protected]        [email protected]        
$ brew install [email protected]  #版本不需要安装太高,不然有的插件用不了
....
$ brew info [email protected] #安装成功后信息
[email protected]: stable 2.4.6 [keg-only]
Distributed search & analytics engine
https://www.elastic.co/products/elasticsearch
/usr/local/Cellar/[email protected]/2.4.6 (78 files, 38.5MB)
  Built from source on 2017-12-15 at 11:49:26
From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/[email protected]
==> Requirements
Required: java >= 1.7 ✔
==> Caveats
Data:    /usr/local/var/elasticsearch/elasticsearch_lph/
Logs:    /usr/local/var/log/elasticsearch/elasticsearch_lph.log
Plugins: /usr/local/opt/[email protected]/libexec/plugins/
Config:  /usr/local/etc/elasticsearch/
plugin script: /usr/local/opt/[email protected]/libexec/bin/plugin

This formula is keg-only, which means it was not symlinked into /usr/local,
because this is an alternate version of another formula.

If you need to have this software first in your PATH run:
  echo 'export PATH="/usr/local/opt/[email protected]/bin:$PATH"' >> ~/.bash_profile


To have launchd start [email protected] now and restart at login:
  brew services start [email protected]
Or, if you don't want/need a background service you can just run:
  elasticsearch

$ elasticsearch #启动elasticsearch

启动 elasticsearch 后,在浏览器地址:127.0.0.1:9200,浏览器会显示一段json数据

这里出现一个错误: 再次用brew 启动 elasticsearch时候

 $ brew services start [email protected] 
 或者
 $ elasticsearch
 #报错
 ...
 failed to obtain node locks, tried [[/usr/local/var/lib/elasticsearch/elasticsearch_lph]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
 ...

127.0.0.1:9200不能进行访问,这是因为已经有个节点在线程中跑了,可以查看线程

$ ps aux | grep "elastics"
lph              86699   0.0  0.0  2442020    816 s000  S+    2:02下午   0:00.00 grep elastics
lph              86662   0.0 11.0  5079720 920492   ??  S     2:01下午   0:19.68 /Library/Java/JavaVirtualMachines/jdk1.8.0_152.jdk/Contents/Home....
$ kill -9 86662 #杀死jdk跑的elasticsearch 线程

之后 elasticsearch就能正常启动了

 $ brew services start [email protected]

修改elasticsearch配置

在安装完成后 brew info elasticsearch,显示了对于配置文件目录位置 /usr/local/etc/elasticsearch 打开elasticsearch.yml

$ vim /usr/local/etc/elasticsearch/elasticsearch.yml
...
...
cluster.name : test  #集群名称
...
node.name: master-1   #当前节点
...
network.host: 127.0.0.1  

 

保存修改后,启动elasticsearch

$ brew services start [email protected]

外网访问elasticsearch配置

1.elasticsearch.yml 中network.host: 127.0.0.1  改为 network.host: 0.0.0.0

2.开启unbuntu 防火墙对9200的允许 

$ sudo ufw status #查看防火墙状态
22                         ALLOW       Anywhere
80                         ALLOW       Anywhere

$ sudo ufw allow from youallowip to any port 9200 #youallowip为你允许操作的ip,如果没有ip操作限制直接写成:sudo ufw allow 9200

3.用云平台 的朋友需要考虑云端是否设置了9200端口是否被允许访问

 

安装中文分词插件  elasticsearch-analysis-ik 

Versions

IK version	ES version
master	6.x -> master
6.0.0	6.0.0
5.6.4	5.6.4
5.5.3	5.5.3
5.4.3	5.4.3
5.3.3	5.3.3
5.2.2	5.2.2
5.1.2	5.1.2
1.10.6	2.4.6
1.9.5	2.3.5
1.8.1	2.2.1
1.7.0	2.1.1
1.5.0	2.0.0
1.2.6	1.0.0
1.2.5	0.90.x
1.1.3	0.20.x
1.0.0	0.16.2 -> 0.19.0

本例子安装ES 版本为2.4.6,所以对应下载1.10.6插件, 点击releases 查看v1.10.6版本复制下载地址

$ cd /usr/local/opt/[email protected]/libexec/plugins/ #切换到elasticsearch插件位置
$ cd mkdir ik #创建ik目录存放插件
$ cd ik
$ wget https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v1.10.6/elasticsearch-analysis-ik-1.10.6.zip #下载安装包
$ unzip elasticsearch-analysis-ik-1.10.6.zip #解压安装包
$ rm elasticsearch-analysis-ik-1.10.6.zip #删除压缩包
$ brew services restart [email protected]  #重启[email protected]

重启之后查看log   表示插件加载成功 plugins [analysis-ik]

[2017-12-15 13:28:50,371][INFO ][plugins                  ] [master1] modules [reindex, lang-expression, lang-groovy], plugins [analysis-ik], sites []

对比 ik 和自带 对比测试

$ curl -XPOST "http://127.0.0.1:9200/_analyze?analyzer=ik&pretty" -d "你好世界"
{
  "tokens" : [ {
    "token" : "你好",
    "start_offset" : 0,
    "end_offset" : 2,
    "type" : "CN_WORD",
    "position" : 0
  }, {
    "token" : "世界",
    "start_offset" : 2,
    "end_offset" : 4,
    "type" : "CN_WORD",
    "position" : 1
  } ]
}

$ curl -XPOST "http://127.0.0.1:9200/_analyze?analyzer=standard&pretty" -d "你好世界"
{
  "tokens" : [ {
    "token" : "你",
    "start_offset" : 0,
    "end_offset" : 1,
    "type" : "",
    "position" : 0
  }, {
    "token" : "好",
    "start_offset" : 1,
    "end_offset" : 2,
    "type" : "",
    "position" : 1
  }, {
    "token" : "世",
    "start_offset" : 2,
    "end_offset" : 3,
    "type" : "",
    "position" : 2
  }, {
    "token" : "界",
    "start_offset" : 3,
    "end_offset" : 4,
    "type" : "",
    "position" : 3
  } ]
}

 

ubuntu安装:

首先安装jdk

sudo add-apt-repository ppa:webupd8team/java
##报错:
The program 'add-apt-repository' is currently not installed. You can install it by typing:
apt-get install software-properties-common
需要先执行:apt-get install software-properties-common 再执行sudo add-apt-repository ppa:webupd8team/java


sudo apt-get update
sudo apt-get install oracle-java8-installer #都选yes
sudo apt-get install oracle-java8-set-default   #都选yes
java -version
whereis java
which java （java執行路徑）
echo $JAVA_HOME

导入 Elasticsearch 公共 GPG 密钥到 apt：

$ wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

创建  Elasticsearch 资源列表：

$ echo "deb http://packages.elastic.co/elasticsearch/2.x/debian stable main" | sudo tee -a /etc/apt/sources.list.d/elasticsearch-2.x.list

安装Elasticsearch

$ sudo apt-get update
$ sudo apt-get -y install elasticsearch
$ service elasticsearch start  #启动elasticsearch

运行以下命令将在系统启动时启动 Elasticsearch：

$ sudo update-rc.d elasticsearch defaults 95 10

查看版本

root@iZwz92p0vf7mrdbn3qwl40Z:~# curl -XGET localhost:9200
{
  "name" : "masterr-1",
  "cluster_name" : "deraceur",
  "cluster_uuid" : "VV0D289YTbamI939oKzkOA",
  "version" : {
    "number" : "2.4.6",#为es版本号 ik 对应版本1.10.6
    "build_hash" : "5376dca9f70f3abef96a77f4bb22720ace8240fd",
    "build_timestamp" : "2017-07-18T12:17:44Z",
    "build_snapshot" : false,
    "lucene_version" : "5.5.4"
  },
  "tagline" : "You Know, for Search"
}

 

安装maven (ik需要打包 需要先按照maven)

1 下载安装

$ wget http://mirror.bit.edu.cn/apache/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz 
$ tar -zxvf apache-maven-3.3.9-bin.tar.gz 
$ sudo mv apache-maven-3.3.9 /usr/local/

2 为maven配置环境变量

将如下内容添加到/etc/profile和/root/.bashrc中

export M2_HOME=/usr/local/apache-maven-3.3.9 
export PATH=$PATH:$M2_HOME/bin

执行如下指令使其上述配置生效或者重启系统：

source /etc/profile

检查安装是否生效

$ mvn -v 
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-11T00:41:47+08:00) 
Maven home: /usr/local/apache-maven-3.3.9 
Java version: 1.8.0_65, vendor: Oracle Corporation 
Java home: /usr/local/jdk/jdk1.8.0_65/jre 
Default locale: en_US, platform encoding: UTF-8 
OS name: “linux”, version: “3.19.0-25-generic”, arch: “amd64”, family: “unix”

安装ik插件

git clone https://github.com/medcl/elasticsearch-analysis-ik.git
cd elasticsearch-analysis-ik/
git checkout -b v1.10.6
/usr/local/apache-maven-3.3.9/bin/mvn package  #mvn执行
mkdir /usr/share/elasticsearch/plugins/ik #创建ik
cp  target/releases/elasticsearch-analysis-ik-1.10.6.zip  /usr/share/elasticsearch/plugins/ik
cd  /usr/share/elasticsearch/plugins/ik
unzip elasticsearch-analysis-ik-1.10.6.zip

$ service elasticsearch restart  #重新启动elasticsearch

修改elasticsearch配置

vim /etc/elasticsearch/elasticsearch.yml

# 添加以下内容
index.analysis.analyzer.default.type: ik

测试

root@dsfsd:~# curl -XGET 'http://127.0.0.1:9200/_analyze?pretty&analyzer=ik_max_word' -d '电动车不贴反光膜是违法行为'
{
  "tokens" : [ {
    "token" : "电动车",
    "start_offset" : 0,
    "end_offset" : 3,
    "type" : "CN_WORD",
    "position" : 0
  }, {
    "token" : "电动",
    "start_offset" : 0,
    "end_offset" : 2,
    "type" : "CN_WORD",
    "position" : 1
  }, {
    "token" : "车",
    "start_offset" : 2,
    "end_offset" : 3,
    "type" : "CN_CHAR",
    "position" : 2
  }, {
    "token" : "不",
    "start_offset" : 3,
    "end_offset" : 4,
    "type" : "CN_CHAR",
    "position" : 3
  }, {
    "token" : "贴反",
    "start_offset" : 4,
    "end_offset" : 6,
    "type" : "CN_WORD",
    "position" : 4
  }, {
    "token" : "贴",
    "start_offset" : 4,
    "end_offset" : 5,
    "type" : "CN_WORD",
    "position" : 5
  }, {
    "token" : "反光膜",
    "start_offset" : 5,
    "end_offset" : 8,
    "type" : "CN_WORD",
    "position" : 6
  }, {
    "token" : "反光",
    "start_offset" : 5,
    "end_offset" : 7,
    "type" : "CN_WORD",
    "position" : 7
  }, {
    "token" : "膜",
    "start_offset" : 7,
    "end_offset" : 8,
    "type" : "CN_WORD",
    "position" : 8
  }, {
    "token" : "违法行为",
    "start_offset" : 9,
    "end_offset" : 13,
    "type" : "CN_WORD",
    "position" : 9
  }, {
    "token" : "违法",
    "start_offset" : 9,
    "end_offset" : 11,
    "type" : "CN_WORD",
    "position" : 10
  }, {
    "token" : "违",
    "start_offset" : 9,
    "end_offset" : 10,
    "type" : "CN_WORD",
    "position" : 11
  }, {
    "token" : "法",
    "start_offset" : 10,
    "end_offset" : 11,
    "type" : "CN_CHAR",
    "position" : 12
  }, {
    "token" : "行为",
    "start_offset" : 11,
    "end_offset" : 13,
    "type" : "CN_WORD",
    "position" : 13
  } ]
}

 

4.通过elasticsearch-jdbc操作数据(不推荐使用,这边我只是记录下)

使用elasticsearch-jdbc将现有MySQL数据批量导入至ElasticSearch(ubuntu环境运行)

下载并生成 mysql-import-product.sh

# wget http://xbib.org/repository/org/xbib/elasticsearch/importer/elasticsearch-jdbc/2.3.4.0/elasticsearch-jdbc-2.3.4.0-dist.zip  #如果下载速度很慢 可以尝试迅雷下载
# unzip elasticsearch-jdbc-2.3.4.0-dist.zip #解压
# cd elasticsearch-jdbc-2.3.4.0-dist
# cd bin
# cp mysql-blog.sh mysql-import-product.sh
# vim mysql-import-product.sh

修改 mysql-import-product.sh

#!/bin/sh

DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
bin=${DIR}/../bin
lib=${DIR}/../lib

echo '
{
    "type" : "jdbc",
    "jdbc" : {
        "url" : "jdbc:mysql://localhost:3306/test", #目标数据库为test
        "schedule" : "0 0-59 0-23 ? * *", #定时任务
        "user" : "root", #数据库user
        "password" : "12345678",
        "sql" : "select productid,title from product" #导入数据语句
        "index" : "test", 
        "type" : "products",
       
        "elasticsearch" : {
             "cluster" : "test",#为 elasticsearch config 中的 cluster.name
             "host" : "localhost",
             "port" : 9300 
        }   
    }
}
' | java \
    -cp "${lib}/*" \
    -Dlog4j.configurationFile=${bin}/log4j2.xml \
    org.xbib.tools.Runner \
    org.xbib.tools.JDBCImporter

保存后 执行该脚本

# cd ../..
# bin/mysql-import-product.sh
bin/mysql-import-product.sh: 3: bin/mysql-import-product.sh: Bad substitution
bin/mysql-import-product.sh: 4: bin/mysql-import-product.sh: num: not found

#报错 为路劲错误,google很多说是jdk path问题,这边没有时间研究,直接修改路劲
# pwd  #显示当前路劲
/root/elasticsearch-jdbc-2.3.4.0
# vim bin/mysql-import-product.sh  #修改如下
...
 bin=/root/elasticsearch-jdbc-2.3.4.0/bin
lib=/root/elasticsearch-jdbc-2.3.4.0/lib
....
# 退出保存
# bin/mysql-import-product.sh #继续报错 google后 为jdk版本>1.8要求,本机为1.7
Exception in thread "main" java.lang.UnsupportedClassVersionError: org/xbib/tools/Runner : Unsupported major.minor version 52.0
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:803)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:442)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:64)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:354)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:348)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:347)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:312)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
	at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:482)
# .... #papapa~ 升级jdk 升级方式请参考 http://blog.csdn.net/cuiaamay/article/details/51822308
# chmod 777 bin/mysql-import-product.sh #添加执行权限
# bin/mysql-import-product.sh 
# curl -XGET "http://127.0.0.1:9200/test/_search?pretty"
 #验证导入时候成功
{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 378,
  .......
  #有数据表示成功

指令记录:

$  curl -XPOST "http://127.0.0.1:9200/test/_search?pretty"   #获取所有
$  curl -XPOST "http://127.0.0.1:9200/test/_search?pretty" -d”@search.json”   #通过search.json检索数据
$  curl -XDELETE "http://127.0.0.1:9200/test"    #删除数据

查看更多 es指令操作

 

elasticsearch-jdbc自动导入增量数据

 

 

# cp mysql-import-product.sh mysql-delta-import-product.sh
# vim mysql-delta-import-product.sh

#!/bin/sh

DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
bin=${DIR}/../bin
lib=${DIR}/../lib

echo '
{
    "type" : "jdbc",
    "jdbc" : {
        "url" : "jdbc:mysql://localhost:3306/test", #目标数据库为test
        "schedule" : "0 0-59 0-23 ? * *", #定时任务
        "user" : "root", #数据库user
        "password" : "12345678",
        "sql" :[{
                   "statement": "select productid,title from product.updated_at > unix_timestamp(?) group by bshop_product_lang.productid",#当updated_at时间大于当前时间 执行数据导入
                    "parameter" : ["$metrics.lastexecutionstart"] #当前时间戳
        }],
        "index" : "test", 
        "type" : "products",
       
        "elasticsearch" : {
             "cluster" : "test",#为 elasticsearch config 中的 cluster.name
             "host" : "localhost",
             "port" : 9300 
        }   
    }
}
' | java \
    -cp "${lib}/*" \
    -Dlog4j.configurationFile=${bin}/log4j2.xml \
    org.xbib.tools.Runner \
    org.xbib.tools.JDBCImporter

其中updated_at为product更新或者创建时候的记录时间戳.

添加mysql-delta-import-product.sh 脚本权限和写进开机执行脚本

# chmod -X  mysql-delta-import-product.sh
# vim /etc/rc.local
....
sh /youpath/bin/mysql-delta-import-product.sh #ES数据增量 youpath为文件绝对位置
exit 0
....

这样开机将自动执行该脚本

 

5.通过 yii2-elasticsearch操作数据(推荐方法)

打开 yii2 根目录下 composer.json

...
   "require": {
        ....
        "yiisoft/yii2-elasticsearch": "^2.0"
    },
...

保存后

$ composer update

完成后在yii2 配置文件中写入

 'components' => [
        'elasticsearch' => [
            'class' => 'yii\elasticsearch\Connection',
            'nodes' => [
                ['http_address' => '127.0.0.1:9200'],
                // configure more hosts if you have a cluster
            ],
        ], 

创建common\models\ESProduct类 继承 yii\elasticsearch\ActiveRecord

 [
                'productid'  => ['type' => 'double'],
                
                'title' => ['type' => 'string','analyzer'=>'ik', 'searchAnalyzer' => 'ik']
            ]
        ];
    }



    public static function mapping()
    {
        return [
            static::type() => self::mapConfig(),
        ];
    }



    //索引名称
    public static function index()
    {
        return "test";
    }
    //索引类型
    public static function type()
    {
        return "products";
    }

    /**
     * Set (update) mappings for this model
     */
    public static function updateMapping()
    {
        $db = static::getDb();
        $command = $db->createCommand();
        $command->setMapping(static::index(), static::type(), static::mapping());
    }


    /**
     * Create this model's index
     */
    public static function createIndex()
    {
        $db = static::getDb();
        $command = $db->createCommand();
        $command->createIndex(static::index(), [
            'settings' => [ 'index' => ['refresh_interval' => '1s'] ],
            'mappings' => static::mapping(),
            //'warmers' => [ /* ... */ ],
            //'aliases' => [ /* ... */ ],
            //'creation_date' => '...'
        ]);
    }


    /**
     * Delete this model's index
     */
    public static function deleteIndex()
    {
        $db = static::getDb();
        $command = $db->createCommand();
        $command->deleteIndex(static::index(), static::type());
    }

    public static function updateRecord($productid, $columns){
        try {
            $record = self::get($productid);
            foreach($columns as $key => $value){
                $record->$key = $value;
            }
            return $record->update();
        }catch (\Exception $e){
            return false;
        }
    }

    //删除记录
    public static function deleteRecord($productid){
        try {
            $record = self::get($productid);
            $record->delete();
            return true;
        }catch (\Exception $e){
            return false;
        }
    }

    //保存数据(添加 || 更新)
    public function saveRecord($data){

        //加载数据
        if ($this->load($data) && $this->validate()){
            $isExist = false;
            try {
                $record = self::get($this->productid);
                //不存在 为更新
                if (!$record){
                    $record = new self();
                    $record->setPrimaryKey($this->productid);
                }else{
                    $isExist = true;
                }
            }catch (\Exception $e){
                $record = new self();
                $record->setPrimaryKey($this->productid);
            }
            //遍历数据到对象
            foreach ($data['ESProductSearch'] as $key => $value) {
                $record->$key = $value;
            }

            try {

                if (!$isExist){
                    //插入
                    $result = $record->insert();
                }else{
                    //更新
                    $result = $record->update();
                }

            }catch (\Exception $e){
                return false;

            }
            return true;
        }
    }



   


}

创建测试TestControlle.php

query([
            "multi_match" => [
                "query" => $keyword,
                "fields" => ["title"] //检索属性
            ],
        ])->all();
        foreach ($res as $k){
            var_dump($k->productid);//输出id
        }
        exit;
    }

    //创建 或更新
    public function actionCreateES(){
        $ESArray = [
          'ESProductSearch' => [
                'productid' => '1',
                'title'     => 'hello world'
           ]
        ];

        $eSProductSearch = new ESProductSearch();
         if(!$eSProductSearch -> saveRecord($ESArray)){
                   throw new \Exception(Common::getLayoutMessageHtml("操作失败"));
        }        
       
    }
    
    //删除
    public function actionDeteleES($id){
        
        $eSProductSearch = new ESProductSearch();
         if(!$eSProductSearch->deleteByProduct($id)){
                   throw new \Exception(Common::getLayoutMessageHtml("删除失败"));
        }        
       
    }

}

 

转载于:https://my.oschina.net/oslph/blog/1590783