项目中使用到了Spring data jpa技术,调用 JpaRepository.saveAll()/deleteAll()方法对list中的数据进行插入/删除时,发现速度特别慢,数据量5k+的数据大概需要话费20s+,导致请求时间过长,前端报错请求超时。
saveAll底层源码:
@Transactional
public <S extends T> List<S> saveAll(Iterable<S> entities) {
Assert.notNull(entities, "The given Iterable of entities not be null!");
List<S> result = new ArrayList<S>();
for (S entity : entities) {
result.add(save(entity));
}
return result;
}
save方法源码:
@Transactional
public <S extends T> S save(S entity) {
if (entityInformation.isNew(entity)) {
em.persist(entity);
return entity;
} else {
return em.merge(entity);
}
}
deleteAll()源码:
@Transactional
public void deleteAll() {
for (T element : findAll()) {
delete(element);
}
}
查看Spring data jpa 底层==savaAll()==方法,发现,每次保存都是一条条执行,先查询出来,判断是否存在,存在做编辑操作,不存在则新增,导致耗时很久。==deleteAll()==方法也是先查询出所有结果,然后一条条执行删除操作。
1.之前看网上有人说在ymal配置文件中配置可以解决,但是测试下来发现并没有用,配置文件如下:
spring:
jpa:
properties:
hibernate:
jdbc:
batch_size: 500
batch_versioned_data: true
order_inserts: true
order_updates: true
此方法并没有解决我的问题
2.使用EntityManager执行自定义sql
1.引入对象
@PersistenceContext
private EntityManager entityManager;
2.重写saveAll实现批量插入
entityManager.createNativeQuery(sb.toString()+";").executeUpdate()
@Override
public List<DistributeSuccessPlaybill> saveAll(List<DistributeSuccessPlaybill> list) {
StringBuilder sb = into();
int i=0;
for (DistributeSuccessPlaybill dis : list) {
i++;
if(i>1){
sb.append(",");
}
sb.append(String.format(" ('%s','%s','%s','%s','%s','%s',"
+ "'%s','%s','%s','%s','%s','%s','%s','%s','%s','%s','%s','%s')",
dis.getLayoutId(),
dis.getOutChannelId(),
dis.getContentId(),
dis.getAssetId(),
dis.getType(),
dis.getPlayUrl(),
dis.getName(),
dis.getDisplayName(),
dis.getPlayType(),
dis.getStartTime(),
dis.getEndTime(),
dis.getDuration(),
dis.getInsertMode(),
dis.getDistributeStatus(),
dis.getCreatedBy(),
dis.getCreatedTime(),
dis.getUpdatedBy(),
dis.getUpdatedTime()));
//一千条执行一次
if(i >= 1000){
log.info(sb.toString());
entityManager.createNativeQuery(sb.toString()+";").executeUpdate();
i = 0;
sb = into();
}
}
if(i > 0){
entityManager.createNativeQuery(sb.toString()+";").executeUpdate();
}
return list;
}
//sql前半段方法提取
private StringBuilder into(){
StringBuilder sb = new StringBuilder();
sb.append("insert into lp_distribute_success_playbill("
+ "layoutId,"
+ "outChannelId,"
+ "contentId,"
+ "asset_id,"
+ "type,"
+ "playUrl,"
+ "name,"
+ "displayName,"
+ "playType,"
+ "startTime,"
+ "endTime,"
+ "duration,"
+ "insertMode,"
+ "distributeStatus,"
+ "createdBy,"
+ "createdTime,"
+ "updatedBy,"
+ "updatedTime) values");
return sb;
}
此方法再重新执行批量插入,5k+大约消耗3s,效率极大提高,完美解决。
在Repository层使用原生自定义sql,根据需要删除的条件组装sql语句,因为项目的事务加在impl层,所以此处不加事务注解,但是需要加==@Modifying==来标识。
@Modifying
@Query(value = "delete from lp_distribute_success_playbill where outChannelId=?1 and startTime>=?2",nativeQuery = true)
void deleteBatch(Long outChannelId,LocalDateTime startTime);
消耗时间过长问题完美解决~~~!!!!