使用C++和OpenCV读取MNIST文件

 一.MNIST手写字体文件说明

      MNIST手写字体数据库下载地址http://yann.lecun.com/exdb/mnist/ 。

      MNIST手写字体的数据库说明在下载网站的下面也有，为了便于写程序，数据库文件说明如下：

      从上面的数据库说明可以看出来，MNIST手写字体数据库包含了是个文件，每个文件都是单纯的普通文件格式，因此，可以采用C++的文件流将其打开，每一个文件除了几个字节的文件头之外，就是剩下的要数据部分了。因此，可以先将文件的文件头读进来，然后利用magic number进行验证，验证所读的文件是否为MNIST文件。

      由于MNIST存储的格式是大端存储的，和大部分Intel处理器的存储方式不同，所以，直接将文件头的前面四个直接存储为int类型或者long类型的话，是无法获得正确的数值的，还需要进行从大端模式到小端模式的转换，而且，在不同的处理器上面，int类型和long类型存储的位数是不一样的，所以，用int类型或者long类型来读取文件头的前面四个字节不具有可移植性。

    在C++标准中，char类型的长度被定义为一个字节，这个在不同的处理器上面是不变的，因此，可以采用char类型的数组来存储文件头的部分，同时，使用char类型的数组来进行大端模式到小端模式的转换也是很容易的。

    大端模式和小端模式的定义的区别如下：

大端模式：高位字节放在内存低地址处，低位字节放在内存高地址处；

小端模式：低位字节放在内存低地址处，高位字节放在内存高地址处；Intel处理器一般为小端模式。

   

    下面是定义图片文件的文件头和标签的文件头

struct MNISTImageFileHeader
{
    unsigned char MagicNumber[4];
    unsigned char NumberOfImages[4];
    unsigned char NumberOfRows[4];
    unsigned char NumberOfColums[4];
};

struct MNISTLabelFileHeader
{
    unsigned char MagicNumber[4];
    unsigned char NumberOfLabels[4];
};

 

    由于大端模式是把高位字节放在内存的低位处，所以，char类型的数组的低字节表示的就是原来的数的高位，所以，在将char类型的数组转化为整数的时候，只要将数组的第一个元素左移24位，第二个元素左移16位，第三个元素左移8位，然后将这些元素相加起来就可以了，下面是用迭代的方法实现的代码。

    

int ConvertCharArrayToInt(unsigned char* array, int LengthOfArray)
{
    if (LengthOfArray < 0)
    {
        return -1;
    }
    int result = static_cast<signed int>(array[0]);
    for (int i = 1; i < LengthOfArray; i++)
    {
        result = (result << 8) + array[i];
    }
    return result;
}

   

  

    在有了上面的将大端模式存储的char类型数组转化为整数之后，我们就可以将MNIST的文件按照普通的文件格式读进来了，在读取了文件头之后，就可以用普通的读文件的方法将数据部分放在一个char数组中，然后就可以将其放在OpenCV的Mat对象中了。下面是全部读取MNIST文件的代码。

#ifndef MNIST_H
#define MNIST_H

#include <iostream>
#include <fstream>
#include <opencv2/opencv.hpp>

struct MNISTImageFileHeader
{
    unsigned char MagicNumber[4];
    unsigned char NumberOfImages[4];
    unsigned char NumberOfRows[4];
    unsigned char NumberOfColums[4];
};


struct MNISTLabelFileHeader
{
    unsigned char MagicNumber[4];
    unsigned char NumberOfLabels[4];
};

const int MAGICNUMBEROFIMAGE = 2051;
const int MAGICNUMBEROFLABEL = 2049;

int ConvertCharArrayToInt(unsigned char* array, int LengthOfArray);

bool IsImageDataFile(unsigned char* MagicNumber, int LengthOfArray);

bool IsLabelDataFile(unsigned char* MagicNumber, int LengthOfArray);

cv::Mat ReadData(std::fstream& DataFile, int NumberOfData, int DataSizeInBytes);

cv::Mat ReadImageData(std::fstream& ImageDataFile, int NumberOfImages);

cv::Mat ReadLabelData(std::fstream& LabelDataFile, int NumberOfLabel);

cv::Mat ReadImages(std::string& FileName);

cv::Mat ReadLabels(std::string& FileName);




#endif // MNIST_H

/**
 * @file ReadData.cpp The file contains the functions used to read image data
 *                    and label data from the origin mnist file
 * @author sheng
 * @version 1.0.0
 * @date  2014-04-09
 *
 * @function
 *
 * @histroy     <author>      <date>      <version>      <description>
 *               sheng      2014-04-09      1.0.0      build the module
 */

#include <MNIST.h>


/**
 * @brief IsImageDataFile  Check the input MagicNumber is equal to
 *                         MAGICNUMBEROFIMAGE
 * @param MagicNumber      The array of the magicnumber to be checked
 * @param LengthOfArray    The length of the array
 * @return true, if the magcinumber is mathed;
 *         false, otherwise.
 *
 * @author sheng
 * @version 1.0.0
 * @date  2014-04-08
 *
 * @histroy     <author>      <date>      <version>      <description>
 *               sheng      2014-04-08      1.0.0      build the function
 */
bool IsImageDataFile(unsigned char* MagicNumber, int LengthOfArray)
{
    int MagicNumberOfImage = ConvertCharArrayToInt(MagicNumber, LengthOfArray);
    if (MagicNumberOfImage == MAGICNUMBEROFIMAGE)
    {
        return true;
    }

    return false;
}




/**
 * @brief IsImageDataFile  Check the input MagicNumber is equal to
 *                         MAGICNUMBEROFLABEL
 * @param MagicNumber      The array of the magicnumber to be checked
 * @param LengthOfArray    The length of the array
 * @return true, if the magcinumber is mathed;
 *         false, otherwise.
 *
 * @author sheng
 * @version 1.0.0
 * @date  2014-04-08
 *
 * @histroy     <author>      <date>      <version>      <description>
 *               sheng      2014-04-08      1.0.0      build the function
 */
bool IsLabelDataFile(unsigned char *MagicNumber, int LengthOfArray)
{
    int MagicNumberOfLabel = ConvertCharArrayToInt(MagicNumber, LengthOfArray);
    if (MagicNumberOfLabel == MAGICNUMBEROFLABEL)
    {
        return true;
    }

    return false;
}




/**
 * @brief ReadData  Read the data in a opened file
 * @param DataFile  The file which the data is read from.
 * @param NumberOfData  The number of the data
 * @param DataSizeInBytes  The size fo the every data
 * @return The Mat which rows is a data,
 *         Return a empty Mat if the file is not opened or the some flag was
 *                 seted when reading the  data.
 *
 * @author sheng
 * @version 1.0.0
 * @date  2014-04-08
 *
 * @histroy     <author>      <date>      <version>      <description>
 *               sheng      2014-04-08      1.0.0      build the function
 */

cv::Mat ReadData(std::fstream& DataFile, int NumberOfData, int DataSizeInBytes)
{
    cv::Mat DataMat;


    // read the data if the file is opened.
    if (DataFile.is_open())
    {


        int AllDataSizeInBytes = DataSizeInBytes * NumberOfData;
        unsigned char* TmpData = new unsigned char[AllDataSizeInBytes];
        DataFile.read((char *)TmpData, AllDataSizeInBytes);

        //        // If the state is good, convert the array to a mat.
        //        if (!DataFile.fail())
        //        {
        //            DataMat = cv::Mat(NumberOfData, DataSizeInBytes, CV_8UC1,
        //                              TmpData).clone();
        //        }

        DataMat = cv::Mat(NumberOfData, DataSizeInBytes, CV_8UC1,
                          TmpData).clone();
        delete [] TmpData;
        DataFile.close();

    }

    return DataMat;
}




/**
 * @brief ReadImageData  Read the Image data from the MNIST file.
 * @param ImageDataFile  The file which contains the Images.
 * @param NumberOfImages The number of the images.
 * @return The mat contains the image and each row of the mat is a image.
 *         Return empty mat is the file is closed or the data is not matching
 *                the number.
 *
 * @author sheng
 * @version 1.0.0
 * @date  2014-04-08
 *
 * @histroy     <author>      <date>      <version>      <description>
 *               sheng      2014-04-08      1.0.0      build the function
 */
cv::Mat ReadImageData(std::fstream& ImageDataFile, int NumberOfImages)
{
    int ImageSizeInBytes = 28 * 28;

    return ReadData(ImageDataFile, NumberOfImages, ImageSizeInBytes);
}



/**
 * @brief ReadLabelData Read the label data from the MNIST file.
 * @param LabelDataFile The file contained the labels.
 * @param NumberOfLabel The number of the labels.
 * @return The mat contains the labels and each row of the mat is a label.
 *         Return empty mat is the file is closed or the data is not matching
 *                the number.
 *
 * @author sheng
 * @version 1.0.0
 * @date  2014-04-08
 *
 * @histroy     <author>      <date>      <version>      <description>
 *               sheng      2014-04-08      1.0.0      build the function
 */
cv::Mat ReadLabelData(std::fstream& LabelDataFile, int NumberOfLabel)
{
    int LabelSizeInBytes = 1;

    return ReadData(LabelDataFile, NumberOfLabel, LabelSizeInBytes);
}




/**
 * @brief ReadImages Read the Training images.
 * @param FileName  The name of the file.
 * @return The mat contains the image and each row of the mat is a image.
 *         Return empty mat is the file is closed or the data is not matched.
 *
 * @author sheng
 * @version 1.0.0
 * @date  2014-04-08
 *
 * @histroy     <author>      <date>      <version>      <description>
 *               sheng      2014-04-08      1.0.0      build the function
 */
cv::Mat ReadImages(std::string& FileName)
{
    std::fstream File(FileName.c_str(), std::ios_base::in | std::ios_base::binary);

    if (!File.is_open())
    {
        return cv::Mat();
    }

    MNISTImageFileHeader FileHeader;
    File.read((char *)(&FileHeader), sizeof(FileHeader));

    if (!IsImageDataFile(FileHeader.MagicNumber, 4))
    {
        return cv::Mat();
    }

    int NumberOfImage = ConvertCharArrayToInt(FileHeader.NumberOfImages, 4);

    return ReadImageData(File, NumberOfImage);
}




/**
 * @brief ReadLabels  Read the label from the MNIST file.
 * @param FileName  The name of the file.
 * @return The mat contains the image and each row of the mat is a image.
 *         Return empty mat is the file is closed or the data is not matched.
 *
 * @author sheng
 * @version 1.0.0
 * @date  2014-04-08
 *
 * @histroy     <author>      <date>      <version>      <description>
 *               sheng      2014-04-08      1.0.0      build the function
 */
cv::Mat ReadLabels(std::string& FileName)
{
    std::fstream File(FileName.c_str(), std::ios_base::in | std::ios_base::binary);

    if (!File.is_open())
    {
        return cv::Mat();
    }

    MNISTLabelFileHeader FileHeader;
    File.read((char *)(&FileHeader), sizeof(FileHeader));

    if (!IsLabelDataFile(FileHeader.MagicNumber, 4))
    {
        return cv::Mat();
    }

    int NumberOfImage = ConvertCharArrayToInt(FileHeader.NumberOfLabels, 4);

    return ReadLabelData(File, NumberOfImage);
}