多线程请求第三方接口
目录
一、需求
二、实现
三、分析
一、需求
Q:业务过程中遇到一个问题：
1.有一批数据，其中每个数据都需要请求第三方接口返回数据，
2.第三方没有批量请求接口，
3.请求返回的数据要返回给前端，对时效有要求。
以上，所以想用多线程批量调用对方接口，试着缩短接口响应时间。
二、实现
- 新建线程类实现callable接口
public class BatchCallThreadJob implements Callable {
private List> list;
private int threadCount;
private String faceUrl;
public BatchCallThreadJob(List> list, int threadCount,
String faceUrl) {
this.list = list;
this.threadCount = threadCount;
this.faceUrl = faceUrl;
}
@Override
public V call() throws Exception {
Integer count = 0;
Object buyPicture = null;
V map = new HashMap<>();
if(!CollectionUtils.isEmpty(list)) {
for(Map item : list) {
buyPicture = item.get("buyPicture");
if(buyPicture != null && !"".equals(buyPicture)) {
try {
//processPic方法为调用第三方接口逻辑
map.put(buyPicture +".png", processPic2(faceUrl,buyPicture.toString()));
}catch(Exception e){
e.printStackTrace();
continue;
}
}
}
}
try {
log.info("线程【" + Thread.currentThread().getName() + "】接口调用,result:{}", map.size());
} catch (Exception e) {
log.error("ThreeElementsThreadObj err", e);
continue;
}
count++;
log.info("调用总数量为:::::{}",count);
return map;
}
}
2.业务层调用线程类
@RequestMapping(value = "/download", method = RequestMethod.POST)
@ResponseBody
public Map download2(入参) throws Exception {
long startTime = System.currentTimeMillis();
Map resData = new HashMap<>();
List> list = 根据入参，查询数据库得到的list
long endTime = System.currentTimeMillis();
int perNum = 10;//每个线程分配的数据个数
int threadSum = list.size()/perNum;
ExecutorService e = Executors.newFixedThreadPool(5);//默认最多5个线程
V resultMap = new HashMap<>();
List>> futureList = new ArrayList<>();
int start = 0;
int end = 1;
//将list切割，分批调用起线程
for(int i = 1; i <= threadSum; i++){
end = i * perNum > list.size() ? list.size() : i * perNum;
Future f1 = e.submit(new BatchCallThreadJob(list.subList(start,end), i,faceUrl));
futureList.add(f1);
start = perNum * i;
//最后一段的sublist的起始、终止
if (i == threadSum && perNum * threadSum < list.size()) {
Future f2 = e.submit(new BatchCallThreadJob(list.subList(threadSum * perNum,list.size()), i,faceUrl));
futureList.add(f2);
}
}
//线程如有返回值，取值用Future
for(Future> f : futureList) {
try {
V item = f.get();
resultMap.putAll(item);
} catch (InterruptedException interruptedEx) {
interruptedEx.printStackTrace();
}catch (Exception ex) {
ex.printStackTrace();
}
}
e.shutdown();
long endTime2 = System.currentTimeMillis();
logger.info("调用第三方接口时间：[{}] ms", endTime2 - endTime );
/**其他数据处理步骤省略**/
logger.info("接口响应时间：[{}] ms", endTime3 - startTime );
return resData;
}
三、分析
1.遇到问题：
- 起初，把future.get()结果赋给了申明的局部变量，直接操作该变量，出现空指针异常
- preNum的大小对时间影响较大
2.针对preNum大小的不可控（或者是我理解不透彻），改用不切割list，全部扔给线程池，查看对比
public class BatchCallThreadJob implements Callable> {
private Map item;
private int threadCount;
private String faceUrl;
public BatchCallThreadJob(Map item, int threadCount,
String faceUrl) {
this.item = item;
this.threadCount = threadCount;
this.faceUrl = faceUrl;
}
@Override
public V call() throws Exception {
Object buyPicture = null;
V map = new HashMap<>();
buyPicture = item.get("buyPicture");
if(buyPicture != null && !"".equals(buyPicture)) {
map.put(buyPicture +".png", processPic2(faceUrl,buyPicture.toString()));
}
try {
log.info("线程【" + Thread.currentThread().getName() + "】接口调用,result:{}", map.size());
} catch (Exception e) {
log.error("ThreeElementsThreadObj err", e);
}
return map;
}
}
业务层逻辑：
//业务层修改，不切割list，改为遍历list
int i = 1;
for(V item : list) {
Future f = e.submit(new BatchCallThreadJob(item, i,faceUrl));
futureList.add(f);
i++;
}
总结分析：
1.preNum和线程池大小在一个量级时，两种方法时间相差不大，preNum量级过大时，切割list耗时也显著增加
2.对比单线程请求数据耗时，时间明显减少，原始单线程处理时间约为多线程的3倍
3.newFixedThreadPool 创建一个定长线程池，控制线程最大并发数，超出线程在队列中等待