几种压缩算法的压缩和速度比较 LZ4太快了

Quick Benchmark: Gzip vs Bzip2 vs LZMA vs XZ vs LZ4 vs LZO

EDIT: Add zstd

Selected archives

I have selected:

• Source of the kernel to test source compression
• Stream protocol with flush

Test conditions

Tests were run on a desktop:

• Intel Core i5 CPU 750 at 2.67GHz
• 8GB of DDR3 memory
• tmpfs as ram disk is used
• Linux kernel 3.3.2, gentoo amd64
• CFLAGS: -pipe -O2 -g -floop-block -floop-interchange -fgraphite
• bzip2-1.0.6-r3, xz-utils-5.0.3, gzip-1.4

Only normal mode will be tested firstly.

The file test results

Note: The first column with numbers 1..9 indicates the compression setting passed to gzip, bzip2 and lzmash (e.g. "gzip -9").

Tarball mode from linux-3.3, original size: 466083840 (445M)

Compressed file size in bytes

	gzip	bzip2	lzma	lzma -e	xz	xz -e	lz4	lzop
	gzip	bzip2	lzma	lzma -e	xz	xz -e	lz4	lzop
1	124875819 (120M)	93997047 (90M)	85618192 (82M)	72042179 (69M)	85630688 (82M)	72069084 (69M)	165844264 (159M)	168012430 (161M)
2	119040249 (114M)	87741348 (84M)	81480824 (78M)	70264395 (68M)	81492504 (78M)	70282944 (68M)	165844264 (159M)	166987891 (160M)
3	114931686 (110M)	84816957 (81MB)	79575087 (76MB)	69015118 (66MB)	79586568 (76MB)	69029204 (66MB)	165844264 (159M)	166987891 (160M)
5	102328357 (98M)	81837328 (79M)	69557610 (67M)	67879362 (65M)	69583428 (67M)	67875988 (65M)	-	166987891 (160M)
7	100128597 (96M)	80197758 (77M)	67276420 (65M)	66868212 (64M)	67294092 (65M)	66852780 (64M)	-	116205578 (111M)
9	99740486 (96M)	78963640 (76M)	65841213 (63M)	65362226 (63M)	65859432 (63M)	65372696 (63M)	-	114824102 (110M)

Compression ratio

	gzip	bzip2	lzma	lzma -e	xz	xz -e	lz4	lzop
	gzip	bzip2	lzma	lzma -e	xz	xz -e	lz4	lzop
1	26.8%	20.2%	18.4%	15.5%	18.4%	15.5%	35.6%	36.0%
2	25.5%	18.8%	17.5%	15.1%	17.5%	15.1%	35.6%	35.8%
3	24.7%	18.2%	17.1%	14.8%	17.1%	14.8%	35.6%	35.8%
5	22.0%	17.6%	14.9%	14.6%	14.9%	14.6%	-	35.8%
7	21.5%	17.2%	14.4%	14.3%	14.4%	14.3%	-	24.9%
9	21.4%	16.9%	14.1%	14.0%	14.1%	14.0%	-	24.6%

Compression time

	gzip	bzip2	lzma	lzma -e	xz	xz -e	lz4	lzop
	gzip	bzip2	lzma	lzma -e	xz	xz -e	lz4	lzop
1	8.1s	58.3s	31.7s	4m37s	32.2s	4m40s	1.3s	1.6s
2	8.5s	58.4s	40.7s	4m49s	41.9s	4m53s	1.4s	1.6s
3	9.6s	59.1s	1m2s	4m36s	1m1s	4m39s	1.3s	1.5s
5	14s	1m1s	3m5s	5m	3m6s	4m53s	-	1.5s
7	21s	1m2s	4m14s	5m52s	4m13s	5m57s	-	35s
9	33s	1m3s	4m48s	6m40s	4m51s	6m40s	-	1m5s

• gzip -1 vs lz4 -1 on x86: lz4 6.2x more fast
• gzip -1 vs lz4 -1 on ARM: lz4 3.6x more fast

Decompression time

	gzip	bzip2	lzma	lzma -e	xz	xz -e	lz4	lzop
	gzip	bzip2	lzma	lzma -e	xz	xz -e	lz4	lzop
1	3.5s	3.4s	6.7s	5.9s	7.2s	6.5s	0.4s	1.5s
2	3s	15.7	6.3s	5.6s	6.8s	6.3s	0.3s	1.4s
3	3.2s	15.9s	6s	5.6s	6.7s	6.2s	0.4s	1.4s
5	3.2s	16s	5.5s	5.4s	6.2s	6s	-	1.5s
7	3s	15s	5.3s	5.3s	5.9s	5.8s	-	1.3s
9	3s	15s	5s	5.1s	5.6s	5.6s	-	1.2s

Memory requirements on compression

	gzip	bzip2	lzma	lzma -e	xz	xz -e	lz4	lzop
	gzip	bzip2	lzma	lzma -e	xz	xz -e	lz4	lzop
1	0.4MB	1.1MB	8.3MB	12.6MB	8.3MB	12.5MB	12MB	0.7MB
2	0.4MB	1.9MB	15.8MB	24MB	15.8MB	24MB	12MB	0.7MB
3	0.4MB	2.7MB	30.7MB	46.9MB	30.8MB	47M	13.2MB	0.7MB
5	0.4MB	4.2MB	93MB	93MB	93MB	93MB	-	0.7MB
7	0.4MB	5.7MB	185MB	185MB	185MB	185MB	-	0.9MB
9	0.4MB	7.2MB	672MB	673MB	673MB	673MB	-	0.9MB

• Note: lz4 it's the program using this size, the code for internal lz4 use very less memory

Memory requirements on decompression

	gzip	bzip2	lzma	lzma -e	xz	xz -e	lz4	lzop
	gzip	bzip2	lzma	lzma -e	xz	xz -e	lz4	lzop
1	0.1MB	0.3MB	1MB	1MB	1MB	1MB	13MB	0.2MB
2	0.1MB	0.9MB	2.1MB	2.1MB	2.1MB	2.1MB	12MB	0.2MB
3	0.1MB	1.1MB	4.1MB	4.1MB	4.0MB	4.0MB	13MB	0.2MB
5	0.1MB	1.9MB	8.1MB	8.1MB	8.1MB	8.1MB	-	0.2MB
7	0.1MB	2.6MB	16MB	16MB	16MB	16MB	-	0.2MB
9	0.1MB	3.4MB	64MB	64MB	64MB	64MB	-	0.2MB

• Note: lz4 it's the program using this size, the code for internal lz4 use very less memory

Cli used for -1 compression:

• time gzip -1 linux-3.3.tar;du -ksh linux-3.3.tar*;ls -l linux-3.3.tar*;time gunzip linux-3.3.tar*
• time bzip2 -1 linux-3.3.tar;du -ksh linux-3.3.tar*;ls -l linux-3.3.tar*;time bunzip2 linux-3.3.tar.bz2
• time lzma -1 linux-3.3.tar;du -ksh linux-3.3.tar*;ls -l linux-3.3.tar*;read PAUSe;time unlzma linux-3.3.tar*
• time lzma -e -1 linux-3.3.tar;du -ksh linux-3.3.tar*;ls -l linux-3.3.tar*;read PAUSe;time unlzma linux-3.3.tar*
• time xz -1 linux-3.3.tar;du -ksh linux-3.3.tar*;ls -l linux-3.3.tar*;read PAUSe;time unxz linux-3.3.tar*
• time xz -e -1 linux-3.3.tar;du -ksh linux-3.3.tar*;ls -l linux-3.3.tar*;read PAUSe;time unxz linux-3.3.tar*
• time /home/user/lz4-read-only/lz4demo64 -c0 linux-3.3.tar linux-3.3.tar.lz4;du -ksh linux-3.3.tar*;ls -l linux-3.3.tar*;read PAUSe;time /home/user/lz4-read-only/lz4demo64 -d linux-3.3.tar.lz4 linux-3.3.tar;rm linux-3.3.tar.*
• time lzop -1 linux-3.3.tar;rm linux-3.3.tar;du -ksh linux-3.3.tar*;ls -l linux-3.3.tar*;read PAUSe;time lzop -d linux-3.3.tar*;rm linux-3.3.tar.*

Stream test

Time to compress

	gzip	zlib	lz4	lz4hc	none
	gzip	zlib	lz4	lz4hc	none
1	376	368	130	311	10
2	371	391
3	378	362
5	390	392
7	389	387
9	393	384

Time to decompress

	gzip	zlib	lz4	lz4hc	none
	gzip	zlib	lz4	lz4hc	none
1	1797	1778	809	796	9
2	1841	1831
3	1771	1780
5	1806	1960
7	1861	1851
9	1835	1806

Messured with:

timespec tstart,tstop;
clock_gettime(CLOCK_REALTIME, &tstart);
//compression/decompression
clock_gettime(CLOCK_REALTIME, &tstop);
compressionTime+=((quint64)tstop.tv_sec * 1000000LL + (quint64)tstop.tv_nsec / 1000LL)-((quint64)tstart.tv_sec * 1000000LL + (quint64)tstart.tv_nsec / 1000LL);

Memory used

gzip	zlib	lz4	lz4hc	none
gzip	zlib	lz4	lz4hc	none
0.5MB	0.5MB	0MB	0MB	0MB

The socket is always used as full duplex, then it's compression + decompression memory.

Compression ratio

It's transferred size/raw size (lower is better)

With the benchmark application

Uploaded at: https://github.com/alphaonex86/debug-devel/tree/master/QTcpCompressionBenchmark

gzip	zlib	lz4	lz4hc	none
gzip	zlib	lz4	lz4hc	none
0.99	0.97	1.59	1.59	1

The most over-head of small packet (3Bytes) is drop by high compression with zlib/gzip for the big packet.

Graphics

 

Quick benchmark on ARM64

A quick benchmark on ARM64 (odroid, Cortex A53), on kernel Image (12MB), use default compression level (-6) because no way to configure the compression level of btrfs

The speed is on compressed stream, mean the hdd.

	xz	gzip	lz4	zstd
Size	4.2M	5.5M	7.8M	5.8M
Compression speed	0.2 MB/s	1.8 MB/s	21.4 MB/s	2.8 MB/s
Decompression speed	4.8 MB/s	13.6 MB/s	48.4 MB/s	19.1 MB/s