之前碰见一个同学的面试题目大概是分割一个大型文件，需要用上多线程。看到这个我的第一反应是使用fork/join框架。
1.我们先一步一步来，第一步先用最传统的方式来分割
public class 发文件分割 {
private static final int spiltNum = 10*1024;
/**
* 1. 使用最传统的方式 直接分割堵塞io
*/
public void test01(String pathName,String pathStr) throws IOException{
File bigFile = new File(pathStr+"/"+pathName);
FileInputStream in = new FileInputStream(bigFile);
byte[] buf = new byte[spiltNum];
int len;
while ((len = in.read(buf)) != -1) {
FileOutputStream outputStream = new FileOutputStream(new File(pathStr+"/"+pathName+"-"+new Date().getTime()));
outputStream.write(buf, 0, len);
}
}
/**
* 测试
*/
public static void main(String[] args) throws IOException {
Path path = Paths.get("springbean/src/main/resources/io");
String pathStr = path.toAbsolutePath()+"";
发文件分割 spiltExample = new 发文件分割();
spiltExample.test04("schoolpig.sql",pathStr);
}
}
设置好路径和需要分割的文件名称。然后具体的test01方法进行分割操作。先获取文件流，根据分割的spiltNum设计多个输出流。
2.使用nio进行操作。nio是非阻塞的，但是操作文件的时候，还是阻塞的，但性能要比传统的好上一些
/**
* 2. nio处理
*/
public void test02(String pathName,String pathStr) throws IOException{
File bigFile = new File(pathStr+"/"+pathName);
RandomAccessFile accessFile = new RandomAccessFile(bigFile,"rw");
FileChannel channel = accessFile.getChannel();
ByteBuffer buffer = ByteBuffer.allocate(spiltNum);
int len ;
while ( (len = channel.read(buffer)) != -1){
buffer.flip();
FileChannel channel1 = new FileOutputStream(new File(pathStr + "/" + pathName + "-" + new Date().getTime())).getChannel();
channel1.write(buffer);
buffer.clear();
}
}
思路和第一个几乎一样，也没有使用线程。下面例子来使用forkjoin进行尝试。
3.forkjoin在尝试的时候，一开始打算用nio进行读写，但是我实在找不到nio如何读取指定长度的buf，channel.read方法无法读取你想要的哪部分数据。所以没有采用
/**
* 3. nio处理 + forkjoin处理
*/
public void test03(String pathName,String pathStr) throws IOException{
File bigFile = new File(pathStr+"/"+pathName);
long bigSize = bigFile.length();
FileInputStream in = new FileInputStream(bigFile);
ForkJoinPool forkJoinPool = new ForkJoinPool();
forkjoinExample example = new forkjoinExample(pathName, pathStr, in,0,(int) bigSize);
ForkJoinTask submit = forkJoinPool.submit(example);
}
class forkjoinExample extends RecursiveAction{
private FileInputStream in;
private String pathStr;
private String pathName;
private int start;
private int end;
public forkjoinExample(String pathName,String pathStr,FileInputStream in,int start,int end){
this.pathName = pathName;
this.pathStr = pathStr;
this.in = in;
this.start = start;
this.end = end;
}
@Override
protected void compute() {
int len = end - start;
System.out.println(start +"-"+end);
byte[] buf = new byte[len];
if (len <= spiltNum) {
try {
FileOutputStream out = new FileOutputStream(new File(pathStr+"/"+pathName+"-"+start+"-"+end));
out.write(buf);
out.flush();
} catch (IOException e) {
e.printStackTrace();
}
} else {
int middle = (start + end) / 2;
System.out.println(middle);
forkjoinExample example = new forkjoinExample(pathName,pathStr,in,0,middle);
forkjoinExample example2 = new forkjoinExample(pathName,pathStr,in,middle+1,end);
example.fork();
example2.fork();
example.join();
example2.join();
}
}
}
很不幸的是 我无论怎么运行都无法运行这个方法，没有开始分割任务呢，就结束了。所以我无奈的只能选择固定长度的线程池进行分割。
4.线程池分割
/*
4.使用线程池来操作
*/
public void test04(String pathName,String pathStr) throws IOException{
File bigFile = new File(pathStr+"/"+pathName);
long bigSize = bigFile.length();
FileInputStream in = new FileInputStream(bigFile);
ExecutorService pool = Executors.newFixedThreadPool(5);
int start = 0;
int end = (int) bigSize;
while(end - start > spiltNum){
pool.submit(new MyRunnable4(pathName,pathStr,start,end,in));
start = start + spiltNum;
}
pool.submit(new MyRunnable4(pathName,pathStr,start,end,in));
pool.shutdown();
}
class MyRunnable4 implements Runnable{
private String pathName;
private String pathStr;
private int start;
private int end;
private FileInputStream in;
public MyRunnable4(String pathName,String pathStr,int start,int end,FileInputStream in){
this.pathName = pathName;
this.pathStr = pathStr;
this.start = start;
this.end = end;
this.in = in;
}
@Override
public void run() {
byte[] buf = new byte[spiltNum];
try {
int len = in.read(buf);
FileOutputStream out = new FileOutputStream(new File(pathStr+"/"+pathName+"-"+start+"-"+len));
out.write(buf,0,len);
out.flush();
} catch (IOException e) {
e.printStackTrace();
}
}
}
完整文件在github上
不懈努力，慢慢前行。
之前喜欢在有道云上做笔记，因为只有自己看，所以做的偷懒不好。希望换到简述上可以认真一点。水平不高，如果问题请留言指出，一起探讨。