Introduction
Reed-Solomon (RS) code is an error-correcting code first proposed
by Reed and Solomon in 1960, which is the most frequently applied digital error correction code around the word. The applications include data storage(hard drives, CD/DVD/Blue Ray), data transmission and common commercial activities(bar codes, QR codes) .
RS code has the advantage of high capability of correcting random or burst errors since it encodes groups of bits instead of one bit at a time. Redundant datas are generated so that the original data can be reconstructed with part of the stored or received data loss. People often back up important files which can be regarded as kind of data loss protection with redundant data. However, backup is just a copy of the original datas while the redundant data generated in the RS code is a fusion of the all the original data parts, which is more efficient for storage and error correction.
In this article, I have run through the procedure of the RS code and hope it is usefull for you to understand what is going on with this erasure code. Its application in the distributed strorage system are briefly introduced at the end. Part of implementations in pure Go are also provided whose source files can be found on the Githup website .
Galois Field
The finite field is also called Galois field which has finite elements and the property that arithmetic operations on field elements always have a result in the field. In the sequel, we illustrate two kind of representations of the finite elements and its arithmetic operations.
Proposition 1. For any prime and any natual number there exists a finite field with elements and vice versa.
With the proposition 1, the Galois Field is denoted as . In fact, the nature of RS encoding is mapping elments of into another elements of and . This Galois fields can be represented with the help of , the set of polynomials with coefficients in the field of two elements , namely the polynomial representation as
This representation can also be seen as a -bit digit or binary vector.
Proposition 2. has cyclic the multiplicative group , where is the primitive element.
Thus, has the exponential representation as
which is a better choice for the '' and '' operation. Even a binary matrix can be used to represente the elements. For example, we can establish a bijection between the vector representation and matrix representation over as follows
where the first column is the vector representation and the columns satisfy . This matrix representation transforms arithmetic over into arithmetic over which only has XOR, AND operations.
Galois Field Arithmetic
In this section, we discuss arithmetic in , whose element corresponds a byte data.
Addition and Subtraction
Addition and subtraction are operated under the polynomial representation (also a byte in ) and both are equivalent to XOR operation
func galAdd(a, b byte) byte {
return a ^ b
}
func galSub(a, b byte) byte {
return a ^ b
}
Multiplication and Division
The multiplication is operated with the exponential representation , before that we should establish a bijection (injection and surjection) between the two representations. Suppose
where . According to proposition 2, .
Proposition 3. has the irreducible polynomial which has no factors of smaller polynomials.
The irreducible polynomial is necessary to establish the bijection since some term are no longer in . Let , we have
or 0x1d . For convenience, we record the bijection with a table, namely called exponent table, whose indexs is the exponents of elements in exponential representation and value is elements in byte representation.
var expTable = [255]byte{0x1, 0x2, 0x4, 0x8, 0x10, 0x20, 0x40, 0x80, 0x1d, 0x3a, ..., 0x8e}
Use logTable[expTable[]]=, the logarithmic table is generated,
var logTable = []byte{
0, 0, 1, 25, 2, 50, 26, 198,
...
116, 214, 244, 234, 168, 80, 88, 175,
}
With these tables, the and functions in are easily defined,
func galMultiply(a, b byte) byte {
if a == 0 || b == 0 {
return 0
}
logA := int(logTable[a])
logB := int(logTable[b])
sum := logA+logB
if sum>254 {
sum -= 255
}
return expTable[sum]
}
func galDivide(a, b byte) byte {
if a == 0 {
return 0
}
if b == 0 {
panic("Argument 'divisor' is 0")
}
logA := int(logTable[a])
logB := int(logTable[b])
logResult := logA - logB
if logResult < 0 {
logResult += 255
}
return expTable[logResult]
}
Based on the result above, the power and inverse functions can be further obtained. In fact, the method to establish the bijection is not unique. The original paper of Reed and Solomon in 1960 provides another method with finite difference equation which has better computability.
RS Code
Coding Matrix Method
1. Orignal approach:
For arbitrary 8-bit symbols,, , , , we have the message polynomial
with this , codewords, i.e. ,,, are obtained and the the encoded messages, which will be transmitted or stored, are of the codewords (professionally called stripe). Using linear algebra, the stripe are denoted collectively as follows
Note that we only discuss one byte a shard (the messages are splited into multiple shards for encoding) here, in practice one input shard (data shard) contains thousands of bytes, in this case the output shard (code shard) contains same size of byte as input shard and the vectors in becomes matrice.
Coding procedure :
1.Split the whole message into same size input shards;
2. Build the Vandermonde matrix (coding matrix);
3. Multiplies the coding matrix by input shards to produce output shards.
func (r reedSolomon) Split(data []byte) ([][]byte, error) {
if len(data) == 0 {
return nil, ErrShortData
}
// Calculate number of bytes per data shard.
perShard := (len(data) + r.DataShards - 1) / r.DataShards
if cap(data) > len(data) {
data = data[:cap(data)]
}
// Only allocate memory if necessary
if len(data) < (r.Shards * perShard) {
// Pad data to r.Shards*perShard.
padding := make([]byte, (r.Shards*perShard)-len(data))
data = append(data, padding...)
}
// Split into equal-length shards.
dst := make([][]byte, r.Shards)
for i := range dst {
dst[i] = data[:perShard]
data = data[perShard:]
}
return dst, nil
}
func vandermonde(rows, cols int) (matrix, error) {
result, err := newMatrix(rows, cols)
if err != nil {
return nil, err
}
for r, row := range result {
for c := range row {
result[r][c] = galExp(byte(r), c)
}
}
return result, nil
}
// Multiplies a subset of rows from a coding matrix by a full set of
// input shards to produce some output shards.
// 'matrixRows' is The rows from the matrix to use.
// 'inputs' An array of byte arrays, each of which is one input shard.
// The number of inputs used is determined by the length of each matrix row.
// outputs Byte arrays where the computed shards are stored.
func (r reedSolomon) codeSomeShards(matrixRows, inputs, outputs [][]byte) {
for c := 0; c < r.DataShards; c++ {
in := inputs[c]
for iRow := 0; iRow < outputCount; iRow++ {
if c == 0 {
galMulSlice(matrixRows[iRow][c], in, outputs[iRow], r.o.useSSSE3, r.o.useAVX2)
} else {
galMulSliceXor(matrixRows[iRow][c], in, outputs[iRow], r.o.useSSSE3, r.o.useAVX2)
}
}
}
}
Since any rows of Vandermonde matrix are linearly independent, with arbitrary correct code shards, the original data shards can be reconstructed by multiplying the corresponding inverse matrix
func (r reedSolomon) reconstruct(shards [][]byte) error {
// Pull out an array holding just the shards that
// correspond to the rows of the submatrix. These shards
// will be the input to the decoding process that re-creates
// the missing data shards.
subShards := make([][]byte, r.DataShards)
validIndices := make([]int, r.DataShards)
subMatrixRow := 0
for matrixRow := 0; matrixRow < r.Shards && subMatrixRow < r.DataShards; matrixRow++ {
if len(shards[matrixRow]) != 0 {
subShards[subMatrixRow] = shards[matrixRow]
validIndices[subMatrixRow] = matrixRow
subMatrixRow++
}
// Pull out the rows of the matrix that correspond to the
// shards that we have and build a square matrix. This
// matrix could be used to generate the shards that we have
// from the original data.
subMatrix, _ := newMatrix(r.DataShards, r.DataShards)
for subMatrixRow, validIndex := range validIndices {
for c := 0; c < r.DataShards; c++ {
subMatrix[subMatrixRow][c] = r.m[validIndex][c]
}
// Invert the matrix
dataDecodeMatrix, err = subMatrix.Invert()
if err != nil {
return err
}
// Re-create any data shards that were missing.
//
// The input to the coding is all of the shards we actually
// have, and the output is the missing data shards. The computation
// is done using the special decode matrix we just built.
outputs := make([][]byte, r.ParityShards)
matrixRows := make([][]byte, r.ParityShards)
outputCount := 0
for iShard := 0; iShard < r.DataShards; iShard++ {
if len(shards[iShard]) == 0 {
if cap(shards[iShard]) >= shardSize {
shards[iShard] = shards[iShard][0:shardSize]
} else {
shards[iShard] = make([]byte, shardSize)
}
outputs[outputCount] = shards[iShard]
matrixRows[outputCount] = dataDecodeMatrix[iShard]
outputCount++
}
}
r.codeSomeShards(matrixRows, subShards, outputs[:outputCount], outputCount, shardSize)
return nil
}
However, in practice, we usually do not know which one is correct or corrupted for the received codewords. In this case, the plurality of votes method is necessary and
or
where is number of unkown errors, therefore always satisfies the in-equation. Another approach is to use the Berlekamp-Welsh Algorithm which avoids the heavy computation of votes:
Berlekamp-Welsh decoder:
1. Send , receive , and most of them such that ；
2. ,
3. Solve the coefficients of co, with
4. Derive .
In fact, ,.
The implicite principle: and agree on at least points. and both have at most zero points. Elimilate and , and are degree at least and agree on at least points, thus .
2. Systematic coding matrix:
In the original approach, all the code shards have been encoded. While in the systematic encoding, the original data shards become part of the code shards and only the parity shards should be encoded. In other words, the stripe contains the original datas and parity codewords together no longer codewords only. The coding matrix is composed of the top square identity matrix and the parity matrix. There are three methods of building the coding matrix in this systematic way:
- Elementary transform on the Vandermonde matrix as the procedure 1.
func buildMatrix(dataShards, totalShards int) (matrix, error) {
vm, err := vandermonde(totalShards, dataShards)
if err != nil {
return nil, err
}
top, err := vm.SubMatrix(0, 0, dataShards, dataShards)
if err != nil {
return nil, err
}
topInv, err := top.Invert()
if err != nil {
return nil, err
}
return vm.Multiply(topInv)
}
- Parity matrix is a transposed Vandermonde matrix.
func buildMatrixPAR1(dataShards, totalShards int) (matrix, error) {
result, err := newMatrix(totalShards, dataShards)
if err != nil {
return nil, err
}
for r, row := range result {
// The top portion of the matrix is the identity
// matrix, and the bottom is a transposed Vandermonde
// matrix starting at 1 instead of 0.
if r < dataShards {
result[r][r] = 1
} else {
for c := range row {
result[r][c] = galExp(byte(c+1), r-dataShards)
}
}
}
return result, nil
}
- Parity matrix is a transposed Cauchy matrix. Cauchy matrices are easier to invert than general matrices [8].
func buildMatrixCauchy(dataShards, totalShards int) (matrix, error) {
result, err := newMatrix(totalShards, dataShards)
if err != nil {
return nil, err
}
for r, row := range result {
// The top portion of the matrix is the identity
// matrix, and the bottom is a transposed Cauchy matrix.
if r < dataShards {
result[r][r] = 1
} else {
for c := range row {
result[r][c] = invTable[(byte(r ^ c))]
}
}
}
return result, nil
}
Generator Polynomial Method
Define the generator polynomial:
and the codeword polynomial can be directly computed as
For the systematic form, which is more often used in practice, we define
then the codeword polynomial becomes
where is the parity codeword polynomial. All the polynomial operation above is processed over . It is also observed that the correction of received message can be checked by testing its divisibility by g(x), and there is no need to decode for the systematic encoding if the answer is affirmative. Otherwise, we denote and suppose there are errors
Syndrome based decoder:
1.Calculate the Syndromes:
2. Solve
and use Chien search solve to derive the roots, denoted as,.
3. Use Forney algorithm to solve
4. The index of are determined by looking up the logarithmic table of .
5. Compute and .
It is worth mentioning that all the syndromes are zeros if , this can be used to check if the received message is corrupted or if the message was completely constructed. RS encoding is relatively straightforward for the generator approach, but decoding needs complicated algebraic computation, especially for the step 2. Because the real value of is unknown and the normal way has to use the trial value untill the matrix in is nonsingular. Other algebraic methods for the evaluation of this error location polynomial include Berlekamp–Massey algorithm and Extended Euclidean algorithm.
This syndrome based decoder can be implemented with different hardware unit such as matrix vector multiplication unit, remainder unit, and performs hard-decision decoding up to errors. Hard-decision decoding decides the bit according to the a threshold, where each bit is definitely one or zero. While soft-decision decoding requires additional reliability information to improve the decision, which has better coding gain for the white Guassian channel [2].
RS Code in Distributed Storage Systems
The RS code are stored in different disks in the distributed storage systems, and the performance of the arasure code in distributed storage systems involves disk I/O and repair bandwidth overhead.
Rotated Reed-Solomon code
In the conventional RS code, all the parity blocds are encoded with data blocks in the same strip, while in the rotated reed-solomon code, parity blocks may be generated with different stripes as in the following figure,
when the disk 5 in the figure fails, this method reduceS 3 operations of reading the data blocks than the conventional RS code [5].
Local Reconstruction Code (LRC)
LRC introduces local parity codes which requires slightly more storage space than conventional RS code, but significantly reduce the number of participating data discs for encoding, thus it is beneficial to the reduction of bandwidth and disc I/O overhead. The figure of pyramid code is shown as follows [6]
However, repair of the global redundancy still needs to access all data discs, another LRC approach in [7] further introduces parity code ( in the figure) for the global parity codes (,,,) to avoid this undesirable situation. By choosing the coefficients of , , , and , properly, the parity code of can be calculated by the existing parity codes and . Thus parity code does not have to occupy additional storage.
Observation: Copy is kind of LRC.
References
[1] I. Reed and G. Solomon, BPolynomial codes over certain finite fields,[ J. Soc. Ind. Appl. Math., vol. 8,
no. 2, pp. 300–304, Jun. 1960.
[2] Wicker and Bhargava, Reed-Solomon Codes and Their Applications, 1994.
[3] James S. Plank and Lihao Xu, Optimizing Cauchy Reed-Solomon Codes for Fault-Tolerant Network Storage Applications.
[4] Reed–Solomon codes for coders.
[5] Khan O, Burns R C, Plank J S, et al. Rethinking erasure codes for cloud file systems: minimizing I/O for recovery and degraded reads.
[6] Huang Cheng, Chen Minghua, Li Jin. Pyramid codes: flexible schemes to trade space for access efficiency in reliable data storage systems.
[7] Sathiamoorthy M, Asteris M, Papailiopoulos D, et al. Xoring elephants: novel erasure codes for big data.
[8] J. Blomer, M. Kalfane, R. Karp, M. Karpinski, M. Luby, and D. Zuckerman, An XOR-Based Erasure-Resilient Coding Scheme.