多线程处理文件集合,先拆分,在执行

try {
                    File file = new File(path);
                    File[] files = file.listFiles();
                    log.info("当前共有文件 "+files.length+"个");

                    List filesList = new ArrayList<>(Arrays.asList(files));
                    List> dividedLists = SplitListUtils.splitList(filesList, 10);
                    int numThreads = Math.min(10, dividedLists.size());  // 获取实际的任务数量
                    log.info("线程数:"+ numThreads);
                    // 创建线程池并执行任务
                    ExecutorService executor = Executors.newFixedThreadPool(numThreads);
                    for (List subFiles : dividedLists) {
                        executor.execute(() -> processFiles(subFiles));
                    }
                    executor.shutdown();
                    boolean finished = false;
                    try {
                        finished = executor.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS);
                    } catch (InterruptedException e) {
                        // 处理中断异常
                        e.printStackTrace();
                    }

// 在执行完所有任务后继续后面的操作处理 collect_task 结束
                    if (finished) {
                        // 所有任务已完成,可以执行后续操作
                        System.out.println("ok.....");
                    } else {
                        // 等待超时或者被中断,可以根据需要进行处理
                        System.out.println("执行中....");
                    }
                } catch (Exception e) {
                    e.printStackTrace();
                }
 public static  List> splitList(List list, int numLists) {
        List> dividedLists = new ArrayList<>();
        int size = list.size();
        int average = size / numLists;
        int remainder = size % numLists;
        int index = 0;

        for (int i = 0; i < numLists; i++) {
            int sublistSize = average + (i < remainder ? 1 : 0);
            List sublist = new ArrayList<>(list.subList(index, index + sublistSize));
            if(sublist.size() > 0){
                dividedLists.add(sublist);
            }

            index += sublistSize;
        }

        return dividedLists;
    }

上面代码的拆分, 最多集合是10个,如果大集合的总数小于10,那就是大集合的数量。要是大集合的数量大于10个,那就均分到10个小集合中。

你可能感兴趣的:(java)