Sprintboot 解压Zip文件,ZipEntry的zipEntry.getSize()为-1的问题

 

简介

这几天在做通过流下载zip文件以及上传zip文件不解压读取zip文件内容的功能,在读取zip文件内容的时候遇到了size为-1的情况,在此记录下遇到的情况、解决办法、以及未解决的问题。

示例

将上传和下载zip文件的功能做成了一个示例,放到了github上,链接:export-import-zip-use-stream,可以尝试运行下。

遇到的问题

通过流下载zip文件之后,再次导入该zip文件,不解压读取zip文件内容,发现ZipEntry的size()返回-1,如下图所示:

Sprintboot 解压Zip文件,ZipEntry的zipEntry.getSize()为-1的问题_第1张图片

 

但是尝试使用系统自带的压缩软件压缩了一个zip文件,并上传读取,发现一切正常,size不为-1。使用zipinfo命令查看两个文件作为对比,如下:

Sprintboot 解压Zip文件,ZipEntry的zipEntry.getSize()为-1的问题_第2张图片

 

可以看到上面文件是通过导出功能生成的,红框里缺少size。而下面的是系统压缩软件压缩的zip文件,红框里面带有size大小。故猜测可能是由于代码里生成ZipEntry的时候没有设置size,compressize,crc32等属性的原因。

读取zip文件时ZipEntry的size为-1解决办法

直接读取当前ZipEntry的流,直到为-1为止,代码如下:

@PostMapping("import")
@ResponseBody
public void importZip(@RequestParam("file")MultipartFile file) throws IOException {

    ZipInputStream zipInputStream = new ZipInputStream(file.getInputStream());

    ZipEntry zipEntry;
    while ((zipEntry = zipInputStream.getNextEntry()) != null) {
        if (zipEntry.isDirectory()) {
            // do nothing
        }else {
            String name = zipEntry.getName();
            long size = zipEntry.getSize();
            // unknown size
            // ZipEntry的size可能为-1,表示未知
            // 通过上面的几种方式下载,就会产生这种情况
            if (size == -1) {
                ByteArrayOutputStream baos = new ByteArrayOutputStream();
                while (true) {
                    int bytes = zipInputStream.read();
                    if (bytes == -1) break;
                    baos.write(bytes);
                }
                baos.close();
                System.out.println(String.format("Name:%s,Content:%s",name,new String(baos.toByteArray())));
            } else { // ZipEntry的size正常
                byte[] bytes = new byte[(int) zipEntry.getSize()];
                zipInputStream.read(bytes, 0, (int) zipEntry.getSize());
                System.out.println(String.format("Name:%s,Content:%s",name,new String(bytes)));
            }
        }

    }
    zipInputStream.closeEntry();
    zipInputStream.close();
}

 

此时可以正确读取文件内容。

下载文件时设置size的解决办法

需要设置ZipEntry的size,compresSize以及crc32等属性。

例如:

public static void main(String[] args) throws IOException {
    File infile  = new File("test_file.pdf");
    File outfile = new File("test.zip");
    try (FileInputStream  fis = new FileInputStream(infile);
         FileOutputStream fos = new FileOutputStream(outfile);
         ZipOutputStream  zos = new ZipOutputStream(fos) ) {

        byte[]  buffer = new byte[1024];
        ZipEntry entry = new ZipEntry("data");
        precalc(entry, fis.getChannel());
        zos.putNextEntry(entry);
        for(int bytesRead; (bytesRead = fis.read(buffer)) >= 0; )
            zos.write(buffer, 0, bytesRead);
        zos.closeEntry();
    }

    try(FileInputStream fin = new FileInputStream(outfile);
        ZipInputStream  zis = new ZipInputStream(fin) ) {

        ZipEntry entry = zis.getNextEntry();
        System.out.println("Entry size: " + entry.getSize());
        System.out.println("Compressed size: " + entry.getCompressedSize());
        System.out.println("CRC: " + entry.getCrc());
        zis.closeEntry();
    }
}

private static void precalc(ZipEntry entry, FileChannel fch) throws IOException {
    long uncompressed = fch.size();
    int method = entry.getMethod();
    CRC32 crc = new CRC32();
    Deflater def;
    byte[] drain;
    if(method != ZipEntry.STORED) {
        def   = new Deflater(Deflater.DEFAULT_COMPRESSION, true);
        drain = new byte[1024];
    }
    else {
        def   = null;
        drain = null;
    }
    ByteBuffer buf = ByteBuffer.allocate((int)Math.min(uncompressed, 4096));
    for(int bytesRead; (bytesRead = fch.read(buf)) != -1; buf.clear()) {
        crc.update(buf.array(), buf.arrayOffset(), bytesRead);
        if(def!=null) {
            def.setInput(buf.array(), buf.arrayOffset(), bytesRead);
            while(!def.needsInput()) def.deflate(drain, 0, drain.length);
        }
    }
    entry.setSize(uncompressed);
    if(def!=null) {
        def.finish();
        while(!def.finished()) def.deflate(drain, 0, drain.length);
        entry.setCompressedSize(def.getBytesWritten());
    }
    entry.setCrc(crc.getValue());
    fch.position(0);
}

 

另外一种解决方法,通过Enumeration 

一、分析下原因

(PS:参考大佬的博客地址:https://blog.csdn.net/zbj18314469395/article/details/84109499)

二、解决办法:

把判断条件由【(ze=zin.getNextEntry())!=null】换成【zipEnum.hasMoreElements ()】
使用了枚举, Enumeration 接口(枚举)。

通常用 Enumeration 中的以下两个方法打印向量中的所有元素:

(1) boolean hasMoreElements(); // 是否还有元素,如果返回 true ,则表示至少含有一个元素

(2) public Object nextElement(); // 如果 Enumeration 枚举对象还含有元素,该方法返回对象中的下一个元素。如果没有,则抛出NoSuchElementException 异常。

附上我的代码,已经修改过的,大家可以参考一下

public void processFile(String path) throws Exception {
        ZipFile zf = new ZipFile(path);
        //主要是这块
        java.util.Enumeration zipEnum = zf.entries();
        InputStream in = new BufferedInputStream(new FileInputStream(path));
        ZipInputStream zin = new ZipInputStream(in, Charset.forName("gbk"));
        ZipEntry ze;
        //重点是是这块的判断
        while (zipEnum.hasMoreElements()) {
            ze = (ZipEntry) zipEnum.nextElement();
            if (!ze.getName().contains("__MACOSX") && !ze.getName().contains(".DS_Store")) {
                if (!ze.isDirectory()) {
                    System.out.println("##name##" + ze.getName());
                    System.out.println("##size##" + ze.getSize());
                }
            }
        }
        zin.closeEntry();
    }

 

另外一个例子:

https://github.com/yrom/shrinker/blob/master/shrinker/src/main/java/net/yrom/tools/JarProcessor.java

package net.yrom.tools;
   import com.google.common.collect.ImmutableList;
        import org.apache.commons.io.IOUtils;
        import org.apache.commons.io.output.ByteArrayOutputStream;
        import java.io.ByteArrayInputStream;
        import java.io.EOFException;
        import java.io.IOException;
        import java.io.OutputStream;
        import java.io.UncheckedIOException;
        import java.nio.file.Files;
        import java.nio.file.Path;
        import java.util.List;
        import java.util.function.Function;
        import java.util.stream.Collectors;
        import java.util.zip.CRC32;
        import java.util.zip.ZipEntry;
        import java.util.zip.ZipInputStream;
        import java.util.zip.ZipOutputStream;
/**
 * @author yrom
 * @version 2017/11/29
 */
class JarProcessor extends ClassesProcessor {
    JarProcessor(Function classTransform, Path src, Path dst) {
        super(classTransform, src, dst);
    }
    @Override
    public void proceed() {
        try {
            List> entryList = readZipEntries(src)
                    .parallelStream()
                    .map(this::transformClassBlob)
                    .collect(Collectors.toList());
            if (entryList.isEmpty()) return;
            try (OutputStream fileOut = Files.newOutputStream(dst)) {
                ByteArrayOutputStream buffer = zipEntries(entryList);
                buffer.writeTo(fileOut);
            }
        } catch (IOException e) {
            throw new RuntimeException("Reading jar entries failure", e);
        }
    }
    private ByteArrayOutputStream zipEntries(List> entryList) throws IOException {
        ByteArrayOutputStream buffer = new ByteArrayOutputStream(8192);
        try (ZipOutputStream jar = new ZipOutputStream(buffer)) {
            jar.setMethod(ZipOutputStream.STORED);
            final CRC32 crc = new CRC32();
            for (Pair entry : entryList) {
                byte[] bytes = entry.second;
                final ZipEntry newEntry = new ZipEntry(entry.first);
                newEntry.setMethod(ZipEntry.STORED); // chose STORED method
                crc.reset();
                crc.update(entry.second);
                newEntry.setCrc(crc.getValue());
                newEntry.setSize(bytes.length);
                writeEntryToJar(newEntry, bytes, jar);
            }
            jar.flush();
        }
        return buffer;
    }
    private Pair transformClassBlob(Pair entry) {
        byte[] bytes = entry.second;
        entry.second = classTransform.apply(bytes);
        return entry;
    }
    private List> readZipEntries(Path src) throws IOException {
        ImmutableList.Builder> list = ImmutableList.builder();
        try (ZipInputStream zip = new ZipInputStream(new ByteArrayInputStream(Files.readAllBytes(src)))) {
            for (ZipEntry entry = zip.getNextEntry();
                 entry != null;
                 entry = zip.getNextEntry()) {
                String name = entry.getName();
                if (!name.endsWith(".class")) {
// skip
                    continue;
                }
                long entrySize = entry.getSize();
                if (entrySize >= Integer.MAX_VALUE) {
                    throw new OutOfMemoryError("Too large class file " + name + ", size is " + entrySize);
                }
                byte[] bytes = readByteArray(zip, (int) entrySize);
                list.add(Pair.of(name, bytes));
            }
        }
        return list.build();
    }
    private byte[] readByteArray(ZipInputStream zip, int expected) throws IOException {
        if (expected == -1) { // unknown size
            return IOUtils.toByteArray(zip);
        }
        final byte[] bytes = new byte[expected];
        int read = 0;
        do {
            int n = zip.read(bytes, read, expected - read);
            if (n <= 0) {
                break;
            }
            read += n;
        } while (read < expected);
        if (expected != bytes.length) {
            throw new EOFException("unexpected EOF");
        }
        return bytes;
    }
    private static void writeEntryToJar(ZipEntry entry, byte[] bytes, ZipOutputStream jar) {
        try {
            jar.putNextEntry(entry);
            jar.write(bytes);
            jar.closeEntry();
        } catch (IOException e) {
            throw new UncheckedIOException(e);
        }
    }
    // mutable pair
    static class Pair {
        F first;
        S second;
        Pair(F first, S second) {
            this.first = first;
            this.second = second;
        }
        static  Pair of(F first, S second) {
            return new Pair<>(first, second);
        }
    }
}

 

zip例子:

https://www.programcreek.com/java-api-examples/?code=FutureSonic/FutureSonic-Server/FutureSonic-Server-master/futuresonic-main/src/main/java/net/sourceforge/subsonic/controller/DownloadController.java

 

/**
     * Computes the CRC checksum for the given file.
     *
     * @param file The file to compute checksum for.
     * @return A CRC32 checksum.
     * @throws IOException If an I/O error occurs.
     */
    private static long computeCrc(File file) throws IOException {
        CRC32 crc = new CRC32();
        InputStream in = new FileInputStream(file);

        try {

            byte[] buf = new byte[8192];
            int n = in.read(buf);
            while (n != -1) {
                crc.update(buf, 0, n);
                n = in.read(buf);
            }

        } finally {
            in.close();
        }

        return crc.getValue();
    }


    private static void copyFileToStream(File file, OutputStream out) throws IOException {
        //LOG.info("Downloading '" + FileUtil.getShortPath(file) + "' to " + status.getPlayer());

        final int bufferSize = 16 * 1024; // 16 Kbit
        InputStream in = new BufferedInputStream(new FileInputStream(file), bufferSize);

        try {
            byte[] buf = new byte[bufferSize];
            long bitrateLimit = 0;
            long lastLimitCheck = 0;

            while (true) {
                long before = System.currentTimeMillis();
                int n = in.read(buf);
                if (n == -1) {
                    break;
                }
                out.write(buf, 0, n);
            }
        } finally {
            out.flush();
            in.close();
        }
    }


    private static void zip(ZipOutputStream out, String root, File file) throws IOException {

        // Exclude all hidden files starting with a "."
        if (file.getName().startsWith(".")) {
            return;
        }

        String zipName = file.getCanonicalPath().substring(root.length() + 1);
        if (file.isFile()) {
            ZipEntry zipEntry = new ZipEntry(zipName);
            zipEntry.setSize(file.length());
            zipEntry.setCompressedSize(file.length());
            zipEntry.setCrc(computeCrc(file));

            out.putNextEntry(zipEntry);
            copyFileToStream(file, out);
            out.closeEntry();
        } else {
            ZipEntry zipEntry = new ZipEntry(zipName + '/');
            zipEntry.setSize(0);
            zipEntry.setCompressedSize(0);
            zipEntry.setCrc(0);

            out.putNextEntry(zipEntry);
            out.closeEntry();

            File[] children = file.listFiles(); //FileUtil.listFiles(file);
            for (File child : children) {
                zip(out, root, child);
            }
        }
    }

    public static void zipDir(String zipFileName, String dir) {
        try {
            File zipFile = new File(zipFileName);
            FileOutputStream fos = new FileOutputStream(zipFile);

            //ZipOutputStream zos = new ZipOutputStream(new BufferedOutputStream(fos));
            CheckedOutputStream checksum = new CheckedOutputStream(fos, new CRC32());
            ZipOutputStream zos = new ZipOutputStream(checksum);

            File dirFile = new File(dir);
            String root = zipFile.getCanonicalPath().substring(0, zipFile.getCanonicalPath().length() - zipFile.getName().length() - 1);
            zip(zos, root, dirFile);
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

但是上面的zip方法,压缩zip后,用Zip4J解压会失败,文件也有问题。这个后面来研究。

 

转自:

http://cxis.me/2018/03/03/%E4%B8%8A%E4%BC%A0Zip%E6%96%87%E4%BB%B6%E4%B8%8D%E8%A7%A3%E5%8E%8B%E8%AF%BB%E5%8F%96%E6%96%87%E4%BB%B6%E5%86%85%E5%AE%B9%E6%97%B6ZipEntry%E7%9A%84size%E4%B8%BA-1%E7%9A%84%E9%97%AE%E9%A2%98/

 

你可能感兴趣的:(Java,SpringBoot)