分割大型文件

之前碰见一个同学的面试题目大概是分割一个大型文件，需要用上多线程。看到这个我的第一反应是使用fork/join框架。

1.我们先一步一步来，第一步先用最传统的方式来分割

public class 发文件分割 {

    private static final int spiltNum = 10*1024;

    /**
     * 1. 使用最传统的方式 直接分割堵塞io
     */
    public void test01(String pathName,String pathStr) throws IOException{
        File bigFile = new File(pathStr+"/"+pathName);
        FileInputStream in = new FileInputStream(bigFile);
        byte[] buf = new byte[spiltNum];
        int len;
        while ((len = in.read(buf)) != -1) {
            FileOutputStream outputStream = new FileOutputStream(new File(pathStr+"/"+pathName+"-"+new Date().getTime()));
            outputStream.write(buf, 0, len);
        }
    }

    /**
     * 测试
     */
    public static void main(String[] args) throws IOException {
        Path path = Paths.get("springbean/src/main/resources/io");
        String pathStr = path.toAbsolutePath()+"";
        发文件分割 spiltExample = new 发文件分割();
        spiltExample.test04("schoolpig.sql",pathStr);
    }
}

设置好路径和需要分割的文件名称。然后具体的test01方法进行分割操作。先获取文件流，根据分割的spiltNum设计多个输出流。

2.使用nio进行操作。nio是非阻塞的，但是操作文件的时候，还是阻塞的，但性能要比传统的好上一些

    /**
     * 2. nio处理
     */
    public void test02(String pathName,String pathStr) throws IOException{
        File bigFile = new File(pathStr+"/"+pathName);
        RandomAccessFile accessFile = new RandomAccessFile(bigFile,"rw");

        FileChannel channel = accessFile.getChannel();
        ByteBuffer buffer = ByteBuffer.allocate(spiltNum);
        int len ;
        while ( (len = channel.read(buffer)) != -1){
            buffer.flip();
            FileChannel channel1 = new FileOutputStream(new File(pathStr + "/" + pathName + "-" + new Date().getTime())).getChannel();
            channel1.write(buffer);
            buffer.clear();
        }
    }

思路和第一个几乎一样，也没有使用线程。下面例子来使用forkjoin进行尝试。

3.forkjoin在尝试的时候，一开始打算用nio进行读写，但是我实在找不到nio如何读取指定长度的buf，channel.read方法无法读取你想要的哪部分数据。所以没有采用

    /**
     * 3. nio处理 + forkjoin处理
     */
    public void test03(String pathName,String pathStr) throws IOException{
        File bigFile = new File(pathStr+"/"+pathName);
        long bigSize = bigFile.length();
        FileInputStream in = new FileInputStream(bigFile);

        ForkJoinPool forkJoinPool = new ForkJoinPool();
        forkjoinExample example = new forkjoinExample(pathName, pathStr, in,0,(int) bigSize);
        ForkJoinTask submit = forkJoinPool.submit(example);
    }

    class forkjoinExample extends RecursiveAction{

        private FileInputStream in;
        private String pathStr;
        private String pathName;
        private int start;
        private int end;

        public forkjoinExample(String pathName,String pathStr,FileInputStream in,int start,int end){
            this.pathName = pathName;
            this.pathStr = pathStr;
            this.in = in;
            this.start = start;
            this.end = end;
        }

        @Override
        protected void compute() {
            int len = end - start;
            System.out.println(start +"-"+end);
            byte[] buf = new byte[len];
            if (len <= spiltNum) {
                try {
                    FileOutputStream out = new FileOutputStream(new File(pathStr+"/"+pathName+"-"+start+"-"+end));
                    out.write(buf);
                    out.flush();
                } catch (IOException e) {
                    e.printStackTrace();
                }
            } else {
                int middle = (start + end) / 2;
                System.out.println(middle);
                forkjoinExample example = new forkjoinExample(pathName,pathStr,in,0,middle);
                forkjoinExample example2 = new forkjoinExample(pathName,pathStr,in,middle+1,end);

                example.fork();
                example2.fork();

                example.join();
                example2.join();
            }
        }
    }

很不幸的是 我无论怎么运行都无法运行这个方法，没有开始分割任务呢，就结束了。所以我无奈的只能选择固定长度的线程池进行分割。

4.线程池分割

    /*
    4.使用线程池来操作
     */
    public void test04(String pathName,String pathStr) throws IOException{
        File bigFile = new File(pathStr+"/"+pathName);
        long bigSize = bigFile.length();
        FileInputStream in = new FileInputStream(bigFile);
        ExecutorService pool = Executors.newFixedThreadPool(5);
        int start = 0;
        int end = (int) bigSize;
        while(end - start > spiltNum){
            pool.submit(new MyRunnable4(pathName,pathStr,start,end,in));
            start = start + spiltNum;
        }
        pool.submit(new MyRunnable4(pathName,pathStr,start,end,in));
        pool.shutdown();
    }

    class MyRunnable4 implements Runnable{

        private String pathName;
        private String pathStr;
        private int start;
        private int end;
        private FileInputStream in;

        public MyRunnable4(String pathName,String pathStr,int start,int end,FileInputStream in){
            this.pathName = pathName;
            this.pathStr = pathStr;
            this.start = start;
            this.end = end;
            this.in = in;
        }

        @Override
        public void run() {
            byte[] buf = new byte[spiltNum];
            try {
                int len = in.read(buf);
                FileOutputStream out = new FileOutputStream(new File(pathStr+"/"+pathName+"-"+start+"-"+len));
                out.write(buf,0,len);
                out.flush();
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
    }

完整文件在github上

不懈努力，慢慢前行。
之前喜欢在有道云上做笔记，因为只有自己看，所以做的偷懒不好。希望换到简述上可以认真一点。水平不高，如果问题请留言指出，一起探讨。