解决springboot2.x整合mongodb下载50MB文件pending6秒
问题描述：从线上服务器下载50MB文件需要pending6秒，无法忍受
场景：LZ正欢快的敲着代码，同事过来说springboot1.x整合mongodb响应很快，大概50MB pending100ms左右，但是2.x之后就pending6秒
Springboot2.xmongodb下载文件简要代码
public void download(String address, String fileName, HttpServletResponse response) {
try {
GridFSFile pdfFile = fsTemplate.findOne(new Query(GridFsCriteria.whereFilename().is(address)));
GridFsResource convert = convert(pdfFile);
InputStream inputStream = convert.getInputStream();
byte[] buf = new byte[1024];
int len;
while ((len = inputStream.read(buf)) != -1) {
out.write(buf, 0, len);
}
contentType = one.getContentType();
} catch (Exception e) {
out.println("该文件没有contentType");
}
}
可以看到最终从mongodb里面拿出流，然后循环读流把文件写出去，很慢，但是springboot1.X时很快就可以响应
Spring boot1.x整合mongodb下载文件简要代码
private void sendFile(String id, String type, HttpServletResponse httpServletResponse) {
GridFSDBFile gridFSDBFile = MongoFileUtil.findFileById(id);
if (gridFSDBFile != null) {
try {
if (StringUtils.isNotEmpty(gridFSDBFile.getContentType())) {
httpServletResponse.setContentType(MediaType.parseMediaType(gridFSDBFile.getContentType()).toString());
} else {
httpServletResponse.setContentType(MediaType.APPLICATION_OCTET_STREAM_VALUE);
}
if (StringUtils.isEmpty(type) || !type.equals("show") || StringUtils.isEmpty(gridFSDBFile.getContentType())) {
httpServletResponse.setHeader("Connection", "close");
//强制下载配置
httpServletResponse.addHeader("Content-Disposition", "attachment;filename=" + URLEncoder.encode(gridFSDBFile.getFilename().trim(), "utf-8"));
}
if (StringUtils.isNotEmpty(type) && type.equals("show") && gridFSDBFile.getContentType().contains("image")) {
httpServletResponse.setHeader("Cache-Control", "max-age=600");
}
ServletOutputStream out = httpServletResponse.getOutputStream();
gridFSDBFile.writeTo(out);
out.flush();
out.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}可以看到是从mongodb拿到一个GridFSDBFile对象写出去为什么1.x很快就能响应但是升级2.x之后就那么慢呢？
我们看看2.x读数组所用的方法：
GridFSDownloadStreamImpl 的read方法
public int read(final byte[] b) {
return read(b, 0, b.length);
}
public int read(final byte[] b, final int off, final int len) {
checkClosed();
if (currentPosition == length) {
return -1;
//这就是为什么第一次很慢 以后就很快的原因 buffer不为null就表示之前已经把数组读到byte数组了
} else if (buffer == null) {
buffer = getBuffer(chunkIndex);
} else if (bufferOffset == buffer.length) {
chunkIndex += 1;
buffer = getBuffer(chunkIndex);
bufferOffset = 0;
}
int r = Math.min(len, buffer.length - bufferOffset);
//把需要的 off - r复制出去
System.arraycopy(buffer, bufferOffset, b, off, r);
bufferOffset += r;
currentPosition += r;
return r;
}可以看到大致流程是：从buffer数组里面coppy从开始到结束
System.arraycopy方式是native方法那么每读取一个byte数组是很快的，从debug看第一次循环的时候时间久，
以后每次都很快，而且第一次读之后接口就会有响应就是会有保存窗口弹出，说明服务器有响应了。那么问题出在哪里了呢？
从流程分析第一次读很慢之后很快，那么有可能第一次获取buffer数组的方法比较慢，
我们来研究一下buffer = getBuffer(chunkIndex);
private byte[] getBuffer(final int chunkIndexToFetch) {
return getBufferFromChunk(getChunk(chunkIndexToFetch), chunkIndexToFetch);
}private byte[] getBufferFromChunk(final Document chunk, final int expectedChunkIndex) {
if (chunk == null || chunk.getInteger("n") != expectedChunkIndex) {
throw new MongoGridFSException(format("Could not find file chunk for file_id: %s at chunk index %s.",
fileId, expectedChunkIndex));
}
if (!(chunk.get("data") instanceof Binary)) {
throw new MongoGridFSException("Unexpected data format for the chunk");
}
byte[] data = chunk.get("data", Binary.class).getData();
long expectedDataLength = 0;
boolean extraChunk = false;
if (expectedChunkIndex + 1 > numberOfChunks) {
extraChunk = true;
} else if (expectedChunkIndex + 1 == numberOfChunks) {
expectedDataLength = length - (expectedChunkIndex * (long) chunkSizeInBytes);
} else {
expectedDataLength = chunkSizeInBytes;
}
if (extraChunk && data.length > expectedDataLength) {
throw new MongoGridFSException(format("Extra chunk data for file_id: %s. Unexpected chunk at chunk index %s."
+ "The size was %s and it should be %s bytes.", fileId, expectedChunkIndex, data.length, expectedDataLength));
} else if (data.length != expectedDataLength) {
throw new MongoGridFSException(format("Chunk size data length is not the expected size. "
+ "The size was %s for file_id: %s chunk index %s it should be %s bytes.",
data.length, fileId, expectedChunkIndex, expectedDataLength));
}
return data;
}
public byte[] getData() {
return (byte[])this.data.clone();
}
private Document getChunk(final int startChunkIndex) {
if (cursor == null) {
cursor = getCursor(startChunkIndex);
}
Document chunk = null;
if (cursor.hasNext()) {
chunk = cursor.next();
if (batchSize == 1) {
discardCursor();
}
if (chunk.getInteger("n") != startChunkIndex) {
throw new MongoGridFSException(format("Could not find file chunk for file_id: %s at chunk index %s.",
fileId, startChunkIndex));
}
}
return chunk;
}
private MongoCursor getCursor(final int startChunkIndex) {
FindIterable findIterable;
Document filter = new Document("files_id", fileId).append("n", new Document("$gte", startChunkIndex));
if (clientSession != null) {
findIterable = chunksCollection.find(clientSession, filter);
} else {
findIterable = chunksCollection.find(filter);
}
return findIterable.batchSize(batchSize).sort(new Document("n", 1)).iterator();
} getBuffer/getBufferFromChunk方法没有什么东西 只是把byte数组拿出来，最多是数组进行了clone也是本地方法
下面继续看getCursor方法漏出来了
看到这里我们就需要大致了解一下mongodb怎么存储文件的了，分块(Chunk)，就是把文件弄成一块，一块的，至于每块的大小mongodb有默认值
private static final int DEFAULT_CHUNKSIZE_BYTES = 255 * 1024;getCursor可以大致看下把这个文件所有的‘块’组织成一个buffer数组(byte数组)。哦，所以第一次很慢，以后每次buffer不为null就直接coppy出来就ok了。
那么spring1.x为什么响应很快呢?
我们看writeTo方法
public long writeTo(final OutputStream out) throws IOException {
int nc = numChunks();
for (int i = 0; i < nc; i++) {
out.write(getChunk(i));
}
return length;
}private byte[] getChunk(final int chunkNumber) {
if (fs == null) {
throw new IllegalStateException("No GridFS instance defined!");
}
//每次把一块拿出来
DBObject chunk = fs.getChunksCollection().findOne(new BasicDBObject("files_id", id).append("n", chunkNumber));
if (chunk == null) {
throw new MongoException("Can't find a chunk! file id: " + id + " chunk: " + chunkNumber);
}
return (byte[]) chunk.get("data");
}每次把每一块的byte数组写出去就ok了，so响应嘎嘎的
那么springboot2.x之后就没有类似1.x这样的写法了吗？并不是，只是我们没有找打而已
@Autowired
private GridFS gridFS;
GridFSDBFile one = gridFS.findOne(new ObjectId(address));
one.writeTo(outputStream);
只是获取GridFS对象
@Configuration
public class MongoDbConfig {
@Resource
private MongoDbFactory mongoDbFactory;
@Bean
public GridFS getGridFS(){
DB db = mongoDbFactory.getLegacyDb();
return new GridFS(db);
}
}