Spring Boot 集成 Elasticsearch 7.x手记
前言
生产环境采部署的7.5.0版本的es集群，且使用自带的xpack验证，不能升级springboot版本，以免影响现有业务，只能手动兼容。
相关的版本
- SpringCloud Greenwich.SR2
- SpringBoot 2.1.7.RELEASE
项目配置
maven依赖
org.springframework.boot
spring-boot-starter-data-elasticsearch
org.springframework.data
spring-data-elasticsearch
3.2.6.RELEASE
上述spring-data-elasticsearch需要重新引入更新的版本，后面需要使用restclient连接es集群
使用intellij或者maven命令查看spring-data-elasticsearch包的依赖关系，可以知晓几个重要包的版本。
es连接加载配置
网上流传一种配置，只需要修改yml配置，就可直接使用，试了一下没有成功。
elasticsearch:
rest:
username: elastic
password: 123456
uris: ["*.*.*.*:9200","*.*.*.*:9200"]
于是作出以下配置
- yml
elasticsearch:
rest:
username: elastic
password: 123456
url: "*.*.*.*:9200,*.*.*.*:9200"
- config
import lombok.extern.slf4j.Slf4j;
import org.elasticsearch.client.RestHighLevelClient;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.elasticsearch.client.ClientConfiguration;
import org.springframework.data.elasticsearch.client.RestClients;
import org.springframework.data.elasticsearch.config.AbstractElasticsearchConfiguration;
import java.util.List;
@Configuration
@Slf4j
public class RestClientConfig extends AbstractElasticsearchConfiguration {
// @Value("${spring.elasticsearch.rest.uris}")
private List<String> uris;
@Value("${spring.elasticsearch.rest.username}")
private String username;
@Value("${spring.elasticsearch.rest.password}")
private String password;
@Value("#{'${spring.elasticsearch.rest.url}'.split(',')}")
private String[] url;
@Override
@Bean
public RestHighLevelClient elasticsearchClient() {
log.info("[RestClientConfig] ----------> url={}", url);
final ClientConfiguration clientConfiguration = ClientConfiguration.builder()
.connectedTo(url)
.withBasicAuth(username, password)
.build();
return RestClients.create(clientConfiguration).rest();
}
}
- Document
@Data
@Builder
@AllArgsConstructor
@NoArgsConstructor
@Document(indexName = "elslog", type = "_doc")
public class ElsLog {
@Id
private String id;
private String orderId;
private String content;
private Date createTime;
}
- Repository
import org.springframework.data.elasticsearch.repository.ElasticsearchRepository;
import org.springframework.stereotype.Repository;
@Repository
public interface ElsLogRepository extends ElasticsearchRepository<ElsLog, String> {
}
- Test
@Autowired
private ElsLogRepository elsLogRepository;
public void testSave(String... args) throws Exception {
ElsLog log = ElsLog.builder()
.context("test")
.createTime(new Date())
.build();
elsLogRepository.save(log);
System.out.println("------------------------------> save log to els over.");
}
public void testQuery() {
// NativeSearchQueryBuilder nativeSearchQueryBuilder = new NativeSearchQueryBuilder();
// NativeSearchQuery searchQuery = nativeSearchQueryBuilder.withSort(SortBuilders.fieldSort("createTime").order(SortOrder.DESC)).build();
// List elsLogs = elasticsearchRestTemplate.queryForList(searchQuery, ElsLog.class);
// System.out.println(new Gson().toJson(elsLogs));
}
查询报错解决
问题 java.lang.NoSuchMethodError: org.elasticsearch.search.SearchHits.getTotalHits()J
解决
- 重写 ResultMapper
需要修改方法 mapResults方法中的“response.getHits().getTotalHits()”，具体代码如下：
import com.fasterxml.jackson.core.JsonEncoding;
import com.fasterxml.jackson.core.JsonFactory;
import com.fasterxml.jackson.core.JsonGenerator;
import org.elasticsearch.action.get.GetResponse;
import org.elasticsearch.action.get.MultiGetItemResponse;
import org.elasticsearch.action.get.MultiGetResponse;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.common.document.DocumentField;
import org.elasticsearch.search.SearchHit;
import org.springframework.core.convert.ConversionService;
import org.springframework.core.convert.support.DefaultConversionService;
import org.springframework.data.domain.Pageable;
import org.springframework.data.elasticsearch.ElasticsearchException;
import org.springframework.data.elasticsearch.annotations.Document;
import org.springframework.data.elasticsearch.annotations.ScriptedField;
import org.springframework.data.elasticsearch.core.AbstractResultMapper;
import org.springframework.data.elasticsearch.core.DefaultEntityMapper;
import org.springframework.data.elasticsearch.core.EntityMapper;
import org.springframework.data.elasticsearch.core.aggregation.AggregatedPage;
import org.springframework.data.elasticsearch.core.aggregation.impl.AggregatedPageImpl;
import org.springframework.data.elasticsearch.core.mapping.ElasticsearchPersistentEntity;
import org.springframework.data.elasticsearch.core.mapping.ElasticsearchPersistentProperty;
import org.springframework.data.elasticsearch.core.mapping.SimpleElasticsearchMappingContext;
import org.springframework.data.mapping.PersistentPropertyAccessor;
import org.springframework.data.mapping.context.MappingContext;
import org.springframework.data.mapping.model.ConvertingPropertyAccessor;
import org.springframework.lang.Nullable;
import org.springframework.util.Assert;
import org.springframework.util.StringUtils;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.lang.reflect.Field;
import java.nio.charset.Charset;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Iterator;
import java.util.List;
public class MyResultMapper extends AbstractResultMapper {
private final MappingContext<? extends ElasticsearchPersistentEntity<?>, ElasticsearchPersistentProperty> mappingContext;
private final ConversionService conversionService;
public MyResultMapper() {
this((MappingContext)(new SimpleElasticsearchMappingContext()));
}
public MyResultMapper(MappingContext<? extends ElasticsearchPersistentEntity<?>, ElasticsearchPersistentProperty> mappingContext) {
this(mappingContext, initEntityMapper(mappingContext));
}
public MyResultMapper(EntityMapper entityMapper) {
this(new SimpleElasticsearchMappingContext(), entityMapper);
}
public MyResultMapper(MappingContext<? extends ElasticsearchPersistentEntity<?>, ElasticsearchPersistentProperty> mappingContext, @Nullable EntityMapper entityMapper) {
super(entityMapper != null ? entityMapper : initEntityMapper(mappingContext));
this.conversionService = new DefaultConversionService();
this.mappingContext = mappingContext;
}
private static EntityMapper initEntityMapper(MappingContext<? extends ElasticsearchPersistentEntity<?>, ElasticsearchPersistentProperty> mappingContext) {
Assert.notNull(mappingContext, "MappingContext must not be null!");
return new DefaultEntityMapper(mappingContext);
}
@Override
public <T> AggregatedPage<T> mapResults(SearchResponse response, Class<T> clazz, Pageable pageable) {
long totalHits = response.getHits().getTotalHits().value;
float maxScore = response.getHits().getMaxScore();
List<T> results = new ArrayList();
Iterator var8 = response.getHits().iterator();
while(var8.hasNext()) {
SearchHit hit = (SearchHit)var8.next();
if (hit != null) {
T result = null;
String hitSourceAsString = hit.getSourceAsString();
if (!StringUtils.isEmpty(hitSourceAsString)) {
result = this.mapEntity(hitSourceAsString, clazz);
} else {
result = this.mapEntity(hit.getFields().values(), clazz);
}
this.setPersistentEntityId(result, hit.getId(), clazz);
this.setPersistentEntityVersion(result, hit.getVersion(), clazz);
this.setPersistentEntityScore(result, hit.getScore(), clazz);
this.populateScriptFields(result, hit);
results.add(result);
}
}
return new AggregatedPageImpl(results, pageable, totalHits, response.getAggregations(), response.getScrollId(), maxScore);
}
private <T> void populateScriptFields(T result, SearchHit hit) {
if (hit.getFields() != null && !hit.getFields().isEmpty() && result != null) {
Field[] var3 = result.getClass().getDeclaredFields();
int var4 = var3.length;
for(int var5 = 0; var5 < var4; ++var5) {
Field field = var3[var5];
ScriptedField scriptedField = (ScriptedField)field.getAnnotation(ScriptedField.class);
if (scriptedField != null) {
String name = scriptedField.name().isEmpty() ? field.getName() : scriptedField.name();
DocumentField searchHitField = (DocumentField)hit.getFields().get(name);
if (searchHitField != null) {
field.setAccessible(true);
try {
field.set(result, searchHitField.getValue());
} catch (IllegalArgumentException var11) {
throw new ElasticsearchException("failed to set scripted field: " + name + " with value: " + searchHitField.getValue(), var11);
} catch (IllegalAccessException var12) {
throw new ElasticsearchException("failed to access scripted field: " + name, var12);
}
}
}
}
}
}
private <T> T mapEntity(Collection<DocumentField> values, Class<T> clazz) {
return this.mapEntity(this.buildJSONFromFields(values), clazz);
}
private String buildJSONFromFields(Collection<DocumentField> values) {
JsonFactory nodeFactory = new JsonFactory();
try {
ByteArrayOutputStream stream = new ByteArrayOutputStream();
JsonGenerator generator = nodeFactory.createGenerator(stream, JsonEncoding.UTF8);
generator.writeStartObject();
Iterator var5 = values.iterator();
while(true) {
while(var5.hasNext()) {
DocumentField value = (DocumentField)var5.next();
if (value.getValues().size() > 1) {
generator.writeArrayFieldStart(value.getName());
Iterator var7 = value.getValues().iterator();
while(var7.hasNext()) {
Object val = var7.next();
generator.writeObject(val);
}
generator.writeEndArray();
} else {
generator.writeObjectField(value.getName(), value.getValue());
}
}
generator.writeEndObject();
generator.flush();
return new String(stream.toByteArray(), Charset.forName("UTF-8"));
}
} catch (IOException var9) {
return null;
}
}
public <T> T mapResult(GetResponse response, Class<T> clazz) {
T result = this.mapEntity(response.getSourceAsString(), clazz);
if (result != null) {
this.setPersistentEntityId(result, response.getId(), clazz);
this.setPersistentEntityVersion(result, response.getVersion(), clazz);
}
return result;
}
public <T> List<T> mapResults(MultiGetResponse responses, Class<T> clazz) {
List<T> list = new ArrayList();
MultiGetItemResponse[] var4 = responses.getResponses();
int var5 = var4.length;
for(int var6 = 0; var6 < var5; ++var6) {
MultiGetItemResponse response = var4[var6];
if (!response.isFailed() && response.getResponse().isExists()) {
T result = this.mapEntity(response.getResponse().getSourceAsString(), clazz);
this.setPersistentEntityId(result, response.getResponse().getId(), clazz);
this.setPersistentEntityVersion(result, response.getResponse().getVersion(), clazz);
list.add(result);
}
}
return list;
}
private <T> void setPersistentEntityId(T result, String id, Class<T> clazz) {
if (clazz.isAnnotationPresent(Document.class)) {
ElasticsearchPersistentEntity<?> persistentEntity = (ElasticsearchPersistentEntity)this.mappingContext.getRequiredPersistentEntity(clazz);
ElasticsearchPersistentProperty idProperty = (ElasticsearchPersistentProperty)persistentEntity.getIdProperty();
PersistentPropertyAccessor<T> accessor = new ConvertingPropertyAccessor(persistentEntity.getPropertyAccessor(result), this.conversionService);
if (idProperty != null && idProperty.getType().isAssignableFrom(String.class)) {
accessor.setProperty(idProperty, id);
}
}
}
private <T> void setPersistentEntityVersion(T result, long version, Class<T> clazz) {
if (clazz.isAnnotationPresent(Document.class)) {
ElasticsearchPersistentEntity<?> persistentEntity = (ElasticsearchPersistentEntity)this.mappingContext.getPersistentEntity(clazz);
ElasticsearchPersistentProperty versionProperty = persistentEntity.getVersionProperty();
if (versionProperty != null && versionProperty.getType().isAssignableFrom(Long.class)) {
Assert.isTrue(version != -1L, "Version in response is -1");
persistentEntity.getPropertyAccessor(result).setProperty(versionProperty, version);
}
}
}
private <T> void setPersistentEntityScore(T result, float score, Class<T> clazz) {
if (clazz.isAnnotationPresent(Document.class)) {
ElasticsearchPersistentEntity<?> entity = (ElasticsearchPersistentEntity)this.mappingContext.getRequiredPersistentEntity(clazz);
if (!entity.hasScoreProperty()) {
return;
}
entity.getPropertyAccessor(result).setProperty(entity.getScoreProperty(), score);
}
}
}
ElasticSearch的几种client
- ES有4种客户端：Jest client、Rest client、Transport client、Node client
- ES支持两种协议 ：
- HTTP协议，支持的客户端有Jest client和Rest client；
- Native Elasticsearch binary协议，也就是Transport client和Node client
- Jest client和Rest client区别：
Jest client非官方支持，在ES5.0之前官方提供的客户端只有Transport client、Node client。在5.0之后官方发布Rest client，并大力推荐。 - Transport client和Node client区别：
Transport client（7.0弃用）和Node client（2.3弃用）区别：最早的两个客户端，Transport client是不需要单独一个节点。Node client需要单独建立一个节点，连接该节点进行操作，ES2.3之前有独立的API，ES2.3之后弃用该API，推荐用户创建一个节点，并用Transport client连接进行操作。