SourceForge上发布dedup util 安装使用 ubuntu centos 版本 1.4.2
项目来自 刘爱贵博士的专栏 SourceForge上发布dedup util 点击打开链接
版本 1.4.2
由于安装中出现了些问题,现将解决方法说明如下:
[dedup util]
dedup util是一款开源的轻量级文件打包工具，它基于块级的重复数据删除技术，可以有效缩减数据容量，节省用户存储空间。目前已经在Sourceforge上创建项目，并且源码正在不断更新中。
更详细信息请参见 http://blog.csdn.net/liuben/archive/2010/01/09/5166538.aspx
dedup util目前处于pre-alpha开发阶段，支持文件打包、解包、追加文件、删除文件、罗列包中文件等功能。初步的测试结果显示，即使在不明确数据是否具备较高重复率的情况下，dedup技术也能较明显地减少数据包的数据量，产生的数据包比Tar工具的要小。
[源码]
项目URL: https://sourceforge.net/projects/deduputil
SVN代码库URL: https://deduputil.svn.sourceforge.net/svnroot/deduputil
[编译]
1.获取源码
svn co https://deduputil.svn.sourceforge.net/svnroot/deduputil deduputil
2.安装libz-dev
ubuntu: apt-get install libz-dev 如果不支持apt-get，请采用其他途径安装。
centos: yum install zlib-devel
3.编译安装
主要步骤:
./autogen.sh sudo ./configure sudo make sudo make install
演示如下:
[jz@localhost deduputil-code-83]$ sudo ./autogen.sh [sudo] password for jz: ./autogen.sh: line 3: aclocal: command not found ./autogen.sh: line 4: autoheader: command not found ./autogen.sh: line 5: autoconf: command not found ./autogen.sh: line 6: automake: command not found这种情况因缺编译环境:
ubuntu :
apt-get install automake
centos:
yum install automake 就可以了
[jz@localhost deduputil-code-83]$ ./autogen.sh [jz@localhost deduputil-code-83]$ sudo ./configure checking for a BSD-compatible install... /usr/bin/install -c checking whether build environment is sane... yes ...... config.status: creating config.h config.status: executing depfiles commands [jz@localhost deduputil-code-83]$ sudo make make all-recursive make[1]: Entering directory `/home/jz/deduputil-code-83' Making all in . make[2]: Entering directory `/home/jz/deduputil-code-83' make[2]: Leaving directory `/home/jz/deduputil-code-83' Making all in src ...... make[2]: Leaving directory `/home/jz/deduputil-code-83/src' make[1]: Leaving directory `/home/jz/deduputil-code-83' [jz@localhost deduputil-code-83]$ sudo make install Making install in . ...... make[1]: Leaving directory `/home/jz/deduputil-code-83/src'
[jz@localhost deduputil-code-83]$ dedup -h
Main options (must only one):
-c, --creat create a new archive
-x, --extract extract files from an archive
-a, --append append files to an archive
-r, --remove remove files from an archive
-t, --list list files in an archive
-s, --stat show information about an archive
Other options:
-C, --chunk chunk algorithms: FSP, CDC, SB, CDC as default
FSP - fixed-size partition
CDC - content-defined chunking
SB - sliding block
-f, --hashfunc set hash function for CDC file chunking, adler_hash as default
hash functions list as followed:
rabin_hash, RS_hash, JS_hash, PJW_hash, ELF_hash, AP_hash
simple_hash, BKDR_hash, JDBM_hash, DJB_hash, CRC_hash, adler_hash
-z, --compress filter the archive through zlib compression
-b, --block block size for deduplication, 4096 as default
-H, --hashtable hashtable bucket number, 10240 as default
-d, --directory change to directory, PWD as default
-v, --verbose print verbose messages
-V, --version display version number
-h, --help give this help list
[命令行]
Usage: dedup [OP TION...] [FILE]...
dedup tool packages files with deduplicaton technique.
Examples:
dedup -c foobar.ded foo bar # Create foobar.ded from files foo and bar.
dedup -a foobar.ded foo1 bar1 # Append files foo1 and bar1 into foobar.ded.
dedup -r foobar.ded foo1 bar1 # Remove files foo1 and bar1 from foobar.ded.
dedup -t foobar.ded # List all files in foobar.ded.
dedup -x foobar.ded # Extract all files from foobar.ded.
Options:
-c, --creat create a new archive
-x, --extract extrace files from an archive
-a, --append append files to an archive
-r, --remove remove files from an archive
-t, --list list files in an archive
-z, --compress filter the archive through zlib compression
-b, --block block size for deduplication, default is 4096
-H, --hashtable hashtable backet number, default is 10240
-d, --directory change to directory, default is PWD
-v, --verbose print verbose messages
-h, --help give this help list
[运行平台]
目前仅在Linux平台开发测试，其他平台未作评测。
[TODO]
1. 数据块碰撞问题
虽然MD5产生的碰撞的概率非常小，但仍然存在发生小概率事件的可能性。需要采用技术手段解决碰撞问题，这样才能保证数据安全，让用户放心使用。
2. 变长数据块
目前是定长数据块的实现，技术上较为简单，变长数据块可能会获得更高的数据压缩率。
3. 相似文件识别
如果两个文件只有很小的差别，比如在某处插入了若干字节，找出这些数据块并单独处理，可能会提高数据压缩率。
修改自:
http://blog.csdn.net/liuaigui/article/details/5641891