data deduplication (Intelligent compression or single-instance storage)

Data deduplication (often called "intelligent compression" or "single-instance storage") is amethod of reducing storage needs by eliminating redundant data. Only one unique instance of thedata is actually retained on storage media, such as disk or tape. Redundant data is replaced with apointer to the unique data copy. For example, a typical email system might contain 100 instances ofthe same one megabyte (MB) fileattachment. If the email platform is backed up or archived, all 100 instances are saved, requiring100 MB storage space. With data deduplication, only one instance of the attachment is actuallystored; each subsequent instance is just referenced back to the one saved copy. In this example, a100 MB storage demand could be reduced to only one MB.

Data deduplication offers other benefits. Lower storage space requirements will save money ondisk expenditures. The more efficient use of disk space also allows for longer disk retentionperiods, which provides better recovery time objectives (RTO) for a longertime and reduces the need for tape backups.Data deduplication also reduces the data that must be sent across a WAN forremotebackups,replication,anddisasterrecovery.

Data deduplication can generally operate at the file or block level. File deduplicationeliminates duplicate files (as in the example above), but this is not a very efficient means ofdeduplication. Block deduplication looks within a file and saves unique iterations of each block.Each chunk of data is processed using a hash algorithm such as MD5or SHA-1. This process generates a unique number for each piece which is then stored in an index.If a file is updated, only the changed data is saved. That is, if only a few bytes of a document orpresentation are changed, only the changed blocks are saved; the changes don't constitute anentirely new file. This behavior makes block deduplication far more efficient. However, blockdeduplication takes more processing power and uses a much larger index to track the individualpieces.

Hash collisions are a potential problem with deduplication. When a piece of data receives a hashnumber, that number is then compared with the index of other existing hash numbers. If that hashnumber is already in the index, the piece of data is considered a duplicate and does not need to bestored again. Otherwise the new hash number is added to the index and the new data is stored. Inrare cases, the hash algorithm may produce the same hash number for two different chunks of data.When a hash collision occurs, the system won't store the new data because it sees that its hashnumber already exists in the index.. This is called a false positive, and can result in data loss.Some vendors combine hash algorithms to reduce the possibility of a hash collision. Some vendorsare also examining metadata to identify data and prevent collisions.

In actual practice, data deduplication is often used in conjunction with other forms of datareduction such as conventionalcompressionand delta differencing. Taken together, these three techniques can be very effective at optimizingthe use of storage space.

原文出处：http://searchstorage.techtarget.com/definition/data-deduplication