项目中有一个功能是数据搜索,要求可以根据用户id、帐户名、邮箱、手机号、昵称、中英文姓名等来精确或模糊查询用户,并且支持按以上查询类型排序,且支持分页;由于当时设计用户表时只有userId为主键,其他几乎没有索引,所以如果按照sql来实现这个功能,那性能可想而知。项目已经上线,为那些字段加索引也不太好,况且不知道哪个字段查询的最频繁,索引加多也浪费;在网上得知一些著名的开放平台像这样的接口都是使用搜索引擎。
于是我开始研究elasticsearch,因为项目用的是springboot框架,自然想到了springboot集成elasticsearch的方式,就是这个依赖
<dependency>
<groupId>org.springframework.bootgroupId>
<artifactId>spring-boot-starter-data-elasticsearchartifactId>
dependency>
以上说的数据搜索接口,举个例子:查询帐户名以“test”开头的、性别为“男性”的、年龄为“20岁”的、按“帐户名”降序排序取前5条数据。用sql写出来为:
select * from user where account like 'test%' and gender=1 and age=20 order by account desc limit 5;用springboot集成elasticsearch的方式写出来为:
SearchQuery searchQuery=new NativeSearchQueryBuilder()
.withQuery(new BoolQueryBuilder()
.must(QueryBuilders.termQuery("gender",1))
.must(QueryBuilders.termQuery("age",20))
.must(QueryBuilders.prefixQuery("account","test"))) //前缀查询以“test”开头的
.withSort(new FieldSortBuilder("account").order(SortOrder.DESC))
.withPageable(new PageRequest(0,5)).build();
List list=elasticsearchTemplate.queryForList(searchQuery,User.class);
这里用的elasticsearchTemplate为集成方式自带的模板,只要在application.properties中配置了elasticsearch集群信息,就可以直接注入模板使用,如下:
@Autowired public ElasticsearchTemplate elasticsearchTemplate;在这里数据搜索功能介绍完了,回到正题。elasticsearch安全很重要,不能让别人知道了你的ip就可以连接上,所以要加安全验证;elasticsearch可以使用shield插件,基本的用户名密码验证功能是不收费的,其他功能收费;但是我们公司maven仓库没有shield依赖,这就麻烦了;后来得知现在都用x-pack来管理,但是x-pack只支持elasticsearch5.0以上版本,而spring-data集成elasticsearch只支持到elasticsearch2.4版本,spring-data集成方式很方便,但是安全线上用不了;改用elasticsearch5.0以上版本,要自己创建客户端连接,最后还是改用了5.0以上采用x-pack管理。
这里版本分别为elasticsearch-5.5.3、x-pack-4.2.1、logstash-5.5.3,之所以要用logstash,是因为我要把mysql数据同步到elasticsearch,用的是logstash-input-jdbc插件。这种方式集成到项目里的方式为:
@Configuration public class ElasticsearchConfiguration implements FactoryBean你也可以不使用连接池,因为TransportClient对es的操作都是异步的。此种连接方式所需要的依赖为:, InitializingBean, DisposableBean { private static final Logger logger = LoggerFactory.getLogger(ElasticsearchConfiguration.class); //由于项目从2.2.4配置的升级到 5.5.3版本 原配置文件不想动还是指定原来配置参数 @Value("${spring.data.elasticsearch.cluster-nodes}") private String clusterNodes ; @Value("${spring.data.elasticsearch.cluster-name}") private String clusterName; private TransportClient client; @Override public void destroy() throws Exception { try { logger.info("Closing elasticSearch client"); if (client != null) { client.close(); } } catch (final Exception e) { logger.error("Error closing ElasticSearch client: ", e); } } @Override public TransportClient getObject() throws Exception { return client; } @Override public Class getObjectType() { return TransportClient.class; } @Override public boolean isSingleton() { return false; } @Override public void afterPropertiesSet() throws Exception { buildClient(); } protected void buildClient() { try { PreBuiltXPackTransportClient preBuiltXPackTransportClient = new PreBuiltXPackTransportClient(settings()); if (!"".equals(clusterNodes)){ for (String nodes:clusterNodes.split(",")) { String InetSocket [] = nodes.split(":"); String Address = InetSocket[0]; Integer port = Integer.valueOf(InetSocket[1]); preBuiltXPackTransportClient.addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName(Address),port )); } client = preBuiltXPackTransportClient; } } catch (UnknownHostException e) { logger.error(e.getMessage()); } } /** * 初始化默认的client */ private Settings settings(){ Settings settings = Settings.builder() .put("cluster.name",clusterName) .put("xpack.security.transport.ssl.enabled", false) .put("xpack.security.user", "elastic:changeme") .put("client.transport.sniff", true).build(); return settings; } }
<dependency>
<groupId>org.springframework.bootgroupId>
<artifactId>spring-boot-starter-webartifactId>
dependency>
<dependency>
<groupId>org.elasticsearchgroupId>
<artifactId>elasticsearchartifactId>
<version>5.5.3version>
dependency>
<dependency>
<groupId>org.elasticsearch.clientgroupId>
<artifactId>transportartifactId>
<version>5.5.3version>
dependency>
<dependency>
<groupId>org.elasticsearch.clientgroupId>
<artifactId>x-pack-transportartifactId>
<version>5.5.3version>
dependency>
<dependency> @Autowired public TransportClient client;
没有了spring-data对elasticsearch的封装,就只能使用最原始的Java api了:
BoolQueryBuilder builder=new BoolQueryBuilder();
builder.must(QueryBuilders.termQuery("gender",1))
.must(QueryBuilders.termQuery("age",20))
.must(QueryBuilders.prefixQuery("account","test"));
SearchResponse response=client.prepareSearch("my_index")//可添加多个index,逗号隔开
.setTypes("my_type")//可添加多个type,逗号隔开
.setQuery(builder)
.setFetchSource(new String[]{"account","gender","age"},null)//自定义返回的字段
.addSort("account", SortOrder.DESC)
.setFrom(0)
.setSize(5)
.setExplain(true)//按查询匹配度排序
.get();
for (SearchHit hit:response.getHits()){
System.out.println(hit.getSourceAsString());
}
elasticsearch安装了x-pack,logstash也安装了x-pack,这时使用logstash-input-jdbc插件同步数据到elasticsearch就会报错:
Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#xpack.monitoring.elasticsearch.url: "http://192.168.134.222:9200"
xpack.monitoring.elasticsearch.username: "elastic"
xpack.monitoring.elasticsearch.password: "changeme"
使用logstash-input-jdbc插件同步数据到elasticsearch,会自动创建映射;然后你再运行项目,调接口执行上述命令时,控制台报错:Fielddata is disabled on text fields by default. Set fielddata=true on [interests] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory,Alternatively use a keyword field instead.
解决方法:更新如下映射,等数据同步完,映射已经创建,这时就会更新失败,所以使用插件同步前,先把映射建好吧
curl -XPUT http://localhost:9200/index -d '{
"mappings": {
"type": {
"properties": {
"publisher": {
"type": "text",
"fielddata": true }
}
}
}
}'
采用原生客户端连接elasticsearch,一定要把原来的spring-boot-starter-data-elasticsearch注掉或去掉,不然会报错:Caused by: java.lang.ClassNotFoundException: org.elasticsearch.action.count.CountRequestBuilder
如果只添加了transport和x-pack-transport,没有添加elasticsearch-5.5.3的依赖,则会报:Caused by: java.lang.ClassNotFoundException: org.elasticsearch.plugins.NetworkPlugin
最好是在依赖里面添加上log4j的依赖,要不然控制台会显示一行红颜色的字,不加也不影响elasticsearch的功能,最好加上:ERROR StatusLogger Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console...
x-pack默认用户名为elastic,密码为changeme;修改密码方式:
curl -XPUT -u elastic 'http://localhost:9200/_xpack/security/user/elastic/_password' -d '{
"password" : "新密码"
}'s
curl -XPUT -u elastic 'http://localhost:9200/_xpack/security/user/kibana/_password' -d '{
"password" : "新密码"
}'
其实已经有人催促官方赶快将spring-data-elasticsearch更新到elasticsearch5.0,但是官方说太忙了,会尽力的,这是讨论地址https://github.com/spring-projects/spring-data-elasticsearch/pull/170
参考地址:http://www.jianshu.com/p/d3d4b5497010
http://blog.csdn.net/peterwanghao/article/details/75230962
http://blog.csdn.net/wangzi19933/article/details/77407702
https://elasticsearch.cn/question/532
http://blog.csdn.net/Q772363685/article/details/74755931
http://blog.csdn.net/pistolove/article/details/53838138