Unit Vector Compression

Recently, I compared a few methods for compressing generic unit vectors.

The method used in Cry Engine 3 for compressing normal vectors is irrelevant here, because it has a limitation that the z component cannot be -1. Since it's only designed for view space normal vectors, those vectors should not point away from the view. However, in our system, we need to handle compression of generic unit vectors, whose directions can be arbitrary.

 

Below are results generated from a unit test with 10,000,000 random unit vectors for each:

 

Quantized Spherical Coordinates in Jensen's Photon Map (16 bits):

Max. Error: 1.57047
Avg. Error: 0.711124

 

Optimized Spherical Coordinates (16 bits)

Max. Error: 0.562332
Avg. Error: 0.30557

 

Half Precision Spherical Coordinates (32 bits):

Max. Error: 0.118694
Avg. Error: 0.0324938

Optimized Spherical Coordinates (24 bits):

Max. Error: 0.0484566
Avg. Error: 0.0158454

 

In conclusion, the method described in "Encoding Normal Vectors using Optimized Spherical Coordinates" provides the best precision with the same number of bytes.