java中设置带返回值的多线程爬虫

起初的目的是爬虫几个网站由于一个个运行感觉效率太低,想到多线程运行爬虫程序。
先上代码吧public class SearchDao {

public List getSearchData(String wd) throws URISyntaxException, InterruptedException, ExecutionException {
// //调用百度爬虫
// Baiduspilder bs=new Baiduspilder();
// List SList = new ArrayList();
// SList=bs.searchit(wd);
//
// //调用Bing爬虫
// Bingspilder bis =new Bingspilder();
// List SList1 = new ArrayList();
// SList1=bis.searchit(wd);
//
// //调用360爬虫
// Spilder360 s360 = new Spilder360();
// List SList2 = new ArrayList();
// SList2=s360.searchit(wd);
//
// //合并列表
// SList.addAll(SList1);
// SList.addAll(SList2);
//
// return SList;

ExecutorService pool = Executors.newCachedThreadPool();
//百度
Callable callable= new Baidu(wd);
Future SList1 = pool.submit(callable);
//Bing
Callable callable1= new Bing(wd);
Future SList2 = pool.submit(callable1);
//360
Callable callable2= new S360(wd);
Future SList3 = pool.submit(callable2);

List SList = new ArrayList<>();

SList.addAll(SList1.get());
SList.addAll(SList2.get());
SList.addAll(SList3.get());
return SList;
}

}
//线程部分
class Baidu implements Callable{
String wd=null;
public Baidu(String wd){
this.wd =wd;
}
@Override
public List call() throws Exception {
Baiduspilder bs = new Baiduspilder();
List SList = new ArrayList<>();
SList = bs.searchit(wd);
return SList;
}

}

class Bing implements Callable{
String wd=null;
public Bing(String wd){
this.wd =wd;
}
@Override
public List call() throws Exception {
Bingspilder bis = new Bingspilder();
List SList1 = new ArrayList<>();
SList1 = bis.searchit(wd);
return SList1;
}

}

class S360 implements Callable{
String wd=null;
public S360(String wd){
this.wd =wd;
}
@Override
public List call() throws Exception {
Spilder360 s360 = new Spilder360();
List SList2 = new ArrayList<>();
SList2 = s360.searchit(wd);
return SList2;
}

}

注释掉的是一开始分别运行的代码!
一开始编译一直不通过,后来发现是讲线程程序写进了同一个类中,分别将3个线程单独写出来之后编译通用了。这是一个带返回值的多线程,通过构造函数传入参数,实现Callable接口。然后返回爬取的结果放入list列表中,通过Future对象获取返回值。

你可能感兴趣的:(java中设置带返回值的多线程爬虫)