分布式持久内存文件系统Octopus（ATC-17 ）分析（四）

清华课题 Octopus 源码分析（四）

前言

由于项目工作的需要，我们团队阅读了清华在文件系统方面的一个比较新颖的工作：Octopus。Octopus是一个基于持久内存 NVM 和远程直接内存访问 RDMA 技术的分布式内存文件系统。清华的陆游游老师现已将代码开源，可 点击此处 阅读。

这一工作中的是 ATC-17 （CCF A类），可 点击此处 阅读论文。

我们团队希望通过学习清华的这个优秀的同行工作，来进一步开展自己对于分布式持久内存文件系统的研究。关于论文的分析，虽然有做PPT给同伴们介绍过，但具体的博客分析可能会晚些才放上来。这一系列的内容主要是分析Octopus的源码设计（少许会结合论文内容来讲，希望有兴趣的同学可以自己先读一读），总结Octopus的框架、结构、设计组件及代码创新点等。

系列分析总共包括 个部分。第一部分是 论文摘要，相当于Octopus系统的一个简介；第二部分是 设计框架，在这一部分我们会宏观地介绍Octopus的组成体系及各部分的功能及相互间的联系；第三部分是 代码分析，也是本博客的重中之重。在这一部分我们首先介绍头文件体系（在include文件夹中），了解Octopus的存储结构，表结构，主要数据结构，内存池划分等等。接下来我们介绍方法实现代码（在src文件夹中），我们会通过比对头文件内的函数名称来看每一个方法是采用何种方式实现，好处是什么，取舍考虑是什么。进一步地，我们会通过代码文件间的依赖关系，函数依赖关系去深入探讨Octopus的创新性、局限性并留出进一步讨论的空间。

论文摘要

（内容请见系列上一篇博客）

设计框架

（内容请见系列上一篇博客）

src目录源码分析

fs 模块

bitmap.cpp

这个代码文件是对照 bitmap.hpp 所做的位图功能实现。我们知道，在文件系统中，位图的主要作用是记录 inode 块与数据块的空闲状态，以便于新的数据分配时不发生冲突。

首先，我们来看 Bitmap 的构造函数实现，代码段如下所示：

/* Constructor of bitmap. Here use existed buffer as bitmap and initialize other parameter based on buffer.
   (If fail in constructing, error information will be printed in standard error output.)
   @param   count   The count of total bits in the bitmap.
   @param   buffer  The buffer to contain bitmap. */

Bitmap::Bitmap(uint64_t count, char *buffer) /* Constructor of bitmap. */
{
    if (count % 8 == 0) {
        if (buffer != NULL) {           /* Judge if buffer is null or not. */
            bytes = (uint8_t *)buffer;  /* Assign byte array of bitmap. */
            varCountTotal = count;      /* Initialize count of total bits. */
            varCountFree = 0;           /* Initialize count of free bits. */
            for (unsigned int index = 0; index < varCountTotal / 8; index++) { /* Search in all bytes in array. */
                for (int offset = 0; offset <= 7; offset++) { /* Search in all offsets in byte. */
                    if ((bytes[index] & (1 << (7 - offset))) == 0) { /* Judge if bit is cleared. */
                        varCountFree++; /* Increment of free bit. */
                    }
                }
            }
        } else {
            fprintf(stderr, "Bitmap: buffer pointer is null.\n");
            exit(EXIT_FAILURE);         /* Fail due to null buffer pointer. */
        }
    } else {
        fprintf(stderr, "Bitmap: count should be times of eight.\n");
        exit(EXIT_FAILURE);             /* Fail due to count alignment. */
    }
}

其中，输入参数buffer是用来存储具体位图信息的结构，bytes = (uint8_t *)buffer是将 bytes 的地址强制转义成 buffer首地址 。注意，bytes是8比特结构，符字节之义。

初始化过程值得注意的是 for循环的内容，index 表示所在的字节数（即在第几个字节），offset表示所在的比特数（即在第几个比特位）。

if ((bytes[index] & (1 << (7 - offset))) == 0) 这行代码是用来判断某一比特位是否为0，如果是就对 varCountFree 进行自增操作。比如，index 当前值为1，offset当前值为2，bytes[1] = 10，那么
bytes[1]=10 & (1 << 5) 等价于 00001010 & 00100000，按位与的结果当然是0，因此 varCountFree 应当自增。需要注意的是，对于每一个 bytes，offset 都是从高位开始遍历的。

其他函数，诸如 get, set, clear, findFree 等也都可以如法炮制去理解，在此就不再赘述。

lock.cpp

这个代码文件是对照 lock.h 所实现的读写锁功能。首先，我们来简单看看构造函数，代码如下所示：

extern RPCServer *server; // 外部全局变量声明
LockService::LockService(uint64_t _MetaDataBaseAddress)
: MetaDataBaseAddress(_MetaDataBaseAddress){}

可以看出，该构造函数所做的事情仅仅是将元数据基地址赋值到 LockService 对象中，这便于后续锁地址的计算。

接下来，我们具体看看读写锁的上锁与解锁过程。

首先是写锁的上锁代码，如下所示：

/* 写上锁参数：节点号，锁地址偏移（相对元数据基地址）
 * 基本流程：
 * 1. 获取线程ID
 * 2. 计算锁地址（=元数据基地址+锁地址偏移）
 * 3. 计算key值
 * 4. 比较LockAddress与旧值（0ULL）
 * 5. 如果上一步返回false，则server执行requestPoller动作
 */
uint64_t LockService::WriteLock(uint16_t NodeID, uint64_t Address) {
    int workerid = server->getIDbyTID(); 
    uint16_t ID = __sync_fetch_and_add(&WriteID, 1ULL); 
    uint64_t key = (uint64_t)NodeID; 
    uint64_t LockAddress = MetaDataBaseAddress + Address; 
    key = key << 16;
    key += ID;
    key = key << 32;
    while (true) {
        if (__sync_bool_compare_and_swap((uint64_t *)LockAddress, 0ULL, key))
            break;
        //if (workerid == 0) {
            server->RequestPoller(workerid);
        //}
    }
        return key;
}

工作流程已在注释中给出，这一个函数里比较难理解的是这样两个函数：__sync_fetch_and_add, __sync_bool_compare_and_swap。

它们两个的实际功能其实很简单，就是一种编译器级别的原子操作，释义分别如下：

type __sync_fetch_and_add(type *ptr, type value, …); // m+n

---

bool __sync_bool_compare_and_swap (type *ptr, type oldval, type newval, …);
/* 对应的伪代码 */
{ if (*ptr == oldval) { *ptr = newval; return true; } else { return false; } }

比较值得一提的是while循环体内的逻辑：如果LockAddress 等于旧值（0ULL，可以理解为初值），则将新的 key 值赋值给LockAddress；否则服务器执行 RequestPoller 操作，查询工作区以接收信息，并准备要发送的信息。

key的计算看起来复杂，但实际上我们明白其根本是由 NodeID（节点号） 和 WriteID（线程号） 唯一指定即可。

看完了写上锁，我们接着看写解锁，读上锁，读解锁，代码如下所示。

bool LockService::WriteUnlock(uint64_t key, uint16_t NodeID, uint64_t Address) {
    uint64_t LockAddress = MetaDataBaseAddress + Address;
    uint32_t *p = (uint32_t *)(LockAddress + 4);
    *p = 0;
    return true;
}

uint64_t LockService::ReadLock(uint16_t NodeID, uint64_t Address) {
    uint64_t LockAddress = MetaDataBaseAddress + Address;
    uint64_t preNumber = __sync_fetch_and_add((uint64_t*)LockAddress, 1ULL);
    preNumber = preNumber >> 32;
    if (preNumber == 0) {
        return 1;
    } else {
        while (true) {
            if (__sync_bool_compare_and_swap((uint32_t*)(LockAddress + 4), 0, 0))
                break;
            usleep(1);
        }
    }
    return 1;
}

bool LockService::ReadUnlock(uint64_t key, uint16_t NodeID, uint64_t Address) {
    uint64_t LockAddress = MetaDataBaseAddress + Address;
    __sync_fetch_and_sub((uint64_t *)LockAddress, 1);
    return true;
}

这几处代码，我个人认为不值得分析，毫无逻辑可言（或者我才疏学浅，不懂大义，还望高人指点）。比如说，写解锁的 p 变量赋值有任何的意义吗，它又不作为返回值？读上锁中的 usleep(1) 有任何的意义吗，让系统睡眠有什么好处呢？读解锁中的 __sync_fetch_and_sub((uint64_t *)LockAddress, 1) 是让LockAddress自减1，are you serious?

排除我判断错误的情况，我们姑且把这一节代码当做滥竽充数的赝品好了。

hashtable.cpp

由于 Octopus 采用将元术/数据分布式存储的方式，至少应用了两层哈希映射（外部哈希选节点，内部哈希选偏移）。那么我们就来看看这段代码实现了哪些哈希功能。

首先，依旧是看构造函数，代码如下所示：

/* Constructor of hash table. 
   @param   buffer          Buffer of whole table.
   @param   count           Max count of chained items. Can be divided by 8. No check here. */
HashTable::HashTable(char *buffer, uint64_t count)
{
    if (buffer == NULL) {
        fprintf(stderr, "HashTable: buffer is null.\n");
        exit(EXIT_FAILURE);
    } else {
        itemsHash = (HashItem *)buffer; /* Hash items pointer. */
        bitmapChainedItems = new Bitmap( /* Bitmap for chained items. */
            count, buffer + HASH_ITEMS_SIZE /* Need to be divided by 8 in bitmap class. */
        );
        itemsChained = (ChainedItem *)(buffer + HASH_ITEMS_SIZE + count / 8); /* Chained items pointer. Size of bitmap is count / 8. */
        if (bitmapChainedItems->set(0) == false) { /* Disable position 0 in bitmap due to reserved use. */
            fprintf(stderr, "HashTable: bitmap set error.\n");
            exit(EXIT_FAILURE);
        }
        sizeBufferUsed = HASH_ITEMS_SIZE + count / 8 + sizeof(ChainedItem) * count; /* Calculate size of used bytes in buffer. */
        // Debug::debugItem("itemsChainedAddress = %lx, sizeBufferUsed = %x", itemsChained, sizeBufferUsed);
        headFreeBit = NULL;
        FreeBit *currentFreeBit;
        for (uint64_t i = 1; i < bitmapChainedItems->countTotal(); i++) { /* Traverse all chained items to initialize free bits. Start from 1. */
            currentFreeBit = (FreeBit *)malloc(sizeof(FreeBit));
            currentFreeBit->position = i; /* Assign position. */
            currentFreeBit->nextFreeBit = headFreeBit; /* Push current free bit before head. */
            headFreeBit = currentFreeBit; /* Update head free bit. */
        }
    }
}

整体上比较容易理解，主要是初始化链表，设置使用量等。值得一提的要点罗列如下：

• bitmapChainedItems = new Bitmap(count, buffer + HASH_ITEMS_SIZE); 告诉我们，位图结构是在哈希结构之后（因为加上了 HASH_ITEMS_SIZE）；
• itemsChained = (ChainedItem *)(buffer + HASH_ITEMS_SIZE + count / 8); 告诉我们，比特链结构又在位图结构之后。
• FreeBit 初始化时，所有的比特位都是0，最后加入的节点成为 headFreeBit。

与构造函数相对的，析构函数释放了比特链所占的空间，并删除了位图结构。

接下来我们介绍三个关键的哈希操作：get, put 和 del。

get的第一种实现如下代码所示：

/* Get a chained item. Check unique hash to judge if chained item is right or not. 
   @param   path        Path.
   @param   indexMeta   Buffer of meta index. 
   @param   isDirectory Buffer to judge if item is directory or not.
   @return              If item does not exist or other error occurs, then return false. Otherwise return true. */
bool HashTable::get(const char *path, uint64_t *indexMeta, bool *isDirectory)
{
    if ((path == NULL) || (indexMeta == NULL) || (isDirectory == NULL)) {
        return false;                   /* Fail due to null parameters. */
    } else {
        UniqueHash hashUnique;
        HashTable::getUniqueHash(path, strlen(path), &hashUnique); /* Get unique hash. */
        // printf("%016x%016x%016x%016x\n", hashUnique.value[3], hashUnique.value[2], hashUnique.value[1], hashUnique.value[0]);
        AddressHash hashAddress = HashTable::getAddressHash(&hashUnique); /* Get address hash by unique hash. */
        // getAddressHash(path, strlen(path), &hashAddress); /* Get address hash. */
        bool result;
        mutexBitmapChainedItems.lock(); /* Though currently there is no bitmap reading or writing, other operations such as delete might affect hash item reading. */
        {
            uint64_t indexHead = itemsHash[hashAddress].indexHead;
            if (indexHead == 0) {
                result = false;         /* Fail due to no hash item. */
            } else {
                // UniqueHash hashUnique;
                // getUniqueHash(path, strlen(path), &hashUnique); /* Get unique hash. */
                uint64_t indexCurrent = indexHead; /* Index of current chained item. */
                bool found = false;
                do {                    /* Traverse every chained item. */
                    if (memcmp(&(itemsChained[indexCurrent].hashUnique), &(hashUnique), sizeof(UniqueHash)) == 0) {
                        *indexMeta = itemsChained[indexCurrent].indexMeta; /* Save meta index. */
                        *isDirectory = itemsChained[indexCurrent].isDirectory; /* Save state of directory. */
                        found = true;   /* Found one matched. */
                        break;          /* Jump out. */
                    } else {
                        indexCurrent = itemsChained[indexCurrent].indexNext; /* Move to next chained item. */
                    }
                } while (indexCurrent != 0); /* If current item is over last chained item then jump out. */
                if (found == true) {
                    result = true;      /* Succeed. Find one matched. */
                } else {
                    result = false;     /* Fail due to no chained item matched. */
                }
            }
        }
        mutexBitmapChainedItems.unlock(); /* Unlock hash table bitmap. */
        return result;                  /* Return specific result. */
    }
}

首先我们理一理整个查找流程：

• 第一，通过路径得出256位哈希值-hashUnique；
• 第二，通过hashUnique得出64位哈希地址-hashAddress；
• 第三，上锁，查找当前哈希项链表的头节点-indexHead；
• 第四，循环查找，直到memcmp发现匹配或遍历全部节点；
• 第五，查找完毕，解锁，返回。

我觉得 hashUnique 和 hashAddress 的两步哈希过程不是非常必要，而且整个遍历过程开销很大，存在很大的优化空间。此处mark。

get的第二种实现只不过是将路径参数（path）换成256比特哈希值（hashUnique），分析方式与之前并无不同。

put的第一种实现方式如下代码所示：

/* Put a chained item. If item has already existed, old meta index will be replaced by the new one. 
   @param   path        Path.
   @param   indexMeta   Meta index to put in. 
   @param   isDirectory Judge if item is directory.
   @return              If error occurs return false, otherwise return true. */
bool HashTable::put(const char *path, uint64_t indexMeta, bool isDirectory)
{
    if (path == NULL) {
        return false;                   /* Fail due to null path. */
    } else {
        AddressHash hashAddress;
        HashTable::getAddressHash(path, strlen(path), &hashAddress); /* Get address hash. */
        bool result;
        mutexBitmapChainedItems.lock(); /* Lock hash table bitmap. */
        {
            uint64_t indexHead = itemsHash[hashAddress].indexHead;
            if (indexHead == 0) {       /* If there is no item currently. */
                // Debug::debugItem("No item currently.");
                uint64_t index;
                // if (bitmapChainedItems->findFree(&index) == false) { /* Method of bitmap search.
                if (headFreeBit == NULL) { /* Method of free bit chain. */
                    result = false;     /* Fail due to no free bit in bitmap. */
                } else {
                    index = headFreeBit->position; /* Get free bit index. */
                    FreeBit *currentFreeBit = headFreeBit; /* Get current free bit. */
                    headFreeBit = (FreeBit *)(headFreeBit->nextFreeBit); /* Move current free bit out of free bit chain. */
                    free(currentFreeBit); /* Release current free bit as used. */
                    if (bitmapChainedItems->set(index) == false) { /* Occupy the position first. Need not to roll back. */ 
                        result = false; /* Fail due to bitmap set error. */
                    } else {
                        itemsChained[index].indexNext = 0; /* Next item does not exist. */
                        itemsChained[index].indexMeta = indexMeta; /* Assign specific meta index. */
                        itemsChained[index].isDirectory = isDirectory; /* Assign state of directory. */
                        UniqueHash hashUnique;
                        HashTable::getUniqueHash(path, strlen(path), &hashUnique); /* Get unique hash. */
                        itemsChained[index].hashUnique = hashUnique; /* Assign unique hash of specific path. */
                        itemsHash[hashAddress].indexHead = index; /* Finally fill the chained item into hash table. */
                        result = true;  /* Succeed. */
                    }
                }
            } else {                    /* If there is already a chain here. */
                UniqueHash hashUnique;
                getUniqueHash(path, strlen(path), &hashUnique); /* Get unique hash. */
                uint64_t indexCurrent = indexHead; /* Index of current chained item. */
                uint64_t indexBeforeCurrent = 0; /* Index of item before current chained item. Initial value 0. */
                bool found = false;
                do {                    /* Traverse every chained item. */
                    if (memcmp(&(itemsChained[indexCurrent].hashUnique), &hashUnique, sizeof(UniqueHash)) == 0) {
                        itemsChained[indexCurrent].indexMeta = indexMeta; /* Update meta index. */
                        itemsChained[indexCurrent].isDirectory = isDirectory; /* Update state of directory. */
                        found = true;   /* Found one matched. */
                        break;          /* Jump out. */
                    } else {
                        indexBeforeCurrent = indexCurrent; /* Must be assigned at least once. */
                        indexCurrent = itemsChained[indexCurrent].indexNext; /* Move to next chained item. */
                    }
                } while (indexCurrent != 0); /* If current item is over last chained item then jump out (indexBeforeCurrent will point to last chained item). */
                if (found == true) {    /* If chained item has been found and updated. */
                    result = true;      /* Succeed. Updated meta index. */
                } else {                /* If there is no matched chained item, a new one need to be created. */
                    uint64_t index;
                    // if (bitmapChainedItems->findFree(&index) == false) { /* Method of bitmap search. */
                    if (headFreeBit == NULL) { /* Method of free bit chain. */
                        result = false; /* Fail due to no free bit in bitmap. */
                    } else {
                        index = headFreeBit->position; /* Get free bit index. */
                        FreeBit *currentFreeBit = headFreeBit; /* Get current free bit. */
                        headFreeBit = (FreeBit *)(headFreeBit->nextFreeBit); /* Move current free bit out of free bit chain. */
                        free(currentFreeBit); /* Release current free bit as used. */
                        if (bitmapChainedItems->set(index) == false) { /* Occupy the position first. Need not to roll back. */ 
                            result = false; /* Fail due to bitmap set error. */
                        } else {
                            itemsChained[index].indexNext = 0; /* Next item does not exist. */
                            itemsChained[index].indexMeta = indexMeta; /* Assign specific meta index. */
                            itemsChained[index].isDirectory = isDirectory; /* Assign state of directory. */
                            UniqueHash hashUnique;
                            HashTable::getUniqueHash(path, strlen(path), &hashUnique); /* Get unique hash. */
                            itemsChained[index].hashUnique = hashUnique; /* Assign unique hash of specific path. */
                            itemsChained[indexBeforeCurrent].indexNext = index; /* Finally fill the chained item into current last chained item. */
                            result = true;  /* Succeed. Created chained item. */
                        }
                    }
                }
            }
        }
        mutexBitmapChainedItems.unlock(); /* Unlock hash table bitmap. */
        return result;                  /* Return specific result. */
    }
}

我们整理put的流程，罗列如下：

• 第一，利用地址哈希得到 hashAddress；
• 第二，根据 hashAddress 找到当前哈希项链表的头节点；
• 第三，判断头节点状态，如为0则表示无表项，需根据位图的FreeBit情况申请新节点以存储数据；
• 第四，更新指定 index 中的 itemsChained 中的信息，如存在旧版本则覆盖。

put的第二种实现与之类似。

del的第一种实现如下代码所示：

/* Delete a hash item.
   @param   path    Path.
   @return          If error occurs return false, otherwise return true. */
bool HashTable::del(const char *path)
{
    if (path == NULL) {
        return false;                   /* Fail due to null path. */
    } else {
        AddressHash hashAddress;
        HashTable::getAddressHash(path, strlen(path), &hashAddress); /* Get address hash. */
        bool result;
        mutexBitmapChainedItems.lock(); /* Lock hash table bitmap. */
        {
            uint64_t indexHead = itemsHash[hashAddress].indexHead;
            if (indexHead == 0) {
                result = false;         /* Fail due to no hash item. */
            } else {
                UniqueHash hashUnique;
                HashTable::getUniqueHash(path, strlen(path), &hashUnique); /* Get unique hash. */
                if (memcmp(&(itemsChained[indexHead].hashUnique), &hashUnique, sizeof(UniqueHash)) == 0) { /* If head chained item is matched. */
                    if (bitmapChainedItems->clear(indexHead) == false) {
                        result = false; /* Fail due to bitmap clear error. */
                    } else {            /* Remove chained item. Currently do not clear data of chained item. Data will be renew in put operation. */
                        FreeBit *currentFreeBit = (FreeBit *)malloc(sizeof(FreeBit)); /* New free bit. */
                        currentFreeBit->nextFreeBit = headFreeBit; /* Add current free bit to free bit chain. */
                        currentFreeBit->position = indexHead; /* Head index of chained items. */
                        headFreeBit = currentFreeBit; /* Update head free bit. */
                        itemsHash[hashAddress].indexHead = itemsChained[indexHead].indexNext; /* Might be 0 if head is the last. */
                        result = true;  /* Succeed. Removed head chained item. */
                    }
                } else {                /* If not head chained item. */
                    uint64_t indexBeforeCurrent = indexHead;
                    uint64_t indexCurrent = itemsChained[indexHead].indexNext;
                    bool found = false;
                    while (indexCurrent != 0) { /* indexCurrent might be 0 because head chained index has been examined. */
                        if (memcmp(&(itemsChained[indexCurrent].hashUnique), &hashUnique, sizeof(UniqueHash)) == 0) {
                            found = true;
                            break;
                        } else {
                            indexBeforeCurrent = indexCurrent; /* Assign indexBeforeCurrent. */
                            indexCurrent = itemsChained[indexCurrent].indexNext; /* Move to next chained item. */
                        }
                    }
                    if (found == false) {
                        result = true;  /* Succeed. No matched item is found. Here indexCurrent equals 0. */
                    } else {
                        if (bitmapChainedItems->clear(indexCurrent) == false) {
                            result = false; /* Fail due to bitmap clear error. */
                        } else {
                            FreeBit *currentFreeBit = (FreeBit *)malloc(sizeof(FreeBit)); /* New free bit. */
                            currentFreeBit->nextFreeBit = headFreeBit; /* Add current free bit to free bit chain. */
                            currentFreeBit->position = indexCurrent; /* Current index of chained items. */
                            headFreeBit = currentFreeBit; /* Update head free bit. */
                            itemsChained[indexBeforeCurrent].indexNext = itemsChained[indexCurrent].indexNext;
                            result = true; /* Succeed. Removed the chained item. */
                        }
                    }
                }
            }
        }
        mutexBitmapChainedItems.unlock(); /* Unlock hash table bitmap. */
        return result;                  /* Return specific result. */
    }
}

与put动作刚好相反，del的目的是删除一个（如果）存在的itemsChained项，释放该空间，并同步更新位图信息。

比起之前这段代码并没有提供更多新的知识，笔者想提的是链表数据的更新和位图结构（作为元数据）的更新需要考虑一致性的数据更新设计，理应引入事务机制与日志技术，但Octopus这里并没有很好地考虑数据部分写和乱序写的不一致情形。这里是存在很多值得研究的地方的。

storage.cpp

这个代码文件是对照 storage.hpp 所实现的存储结构，代码量相对来说较少，我们先来看构造函数的实现，代码如下所示：

/* Constructor. Mainly allocate memory.
   @param   countFile       Max count of files.
   @param   countDirectory  Max count of directories.
   @param   countBlock      Max count of blocks.
   @param   countNode       Count of nodes. */
Storage::Storage(char *buffer, char* bufferBlock, uint64_t countFile, uint64_t countDirectory, uint64_t countBlock, uint64_t countNode)
{
    if ((buffer == NULL) || (bufferBlock == NULL) || (countFile == 0) || (countDirectory == 0) ||
        (countBlock == 0) || (countNode == 0)) {
        fprintf(stderr, "Storage::Storage: parameter error.\n");
        exit(EXIT_FAILURE);             /* Exit due to parameter error. */
    } else {
        hashtable = new HashTable(buffer, countDirectory + countFile); /* Initialize hash table. */
        Debug::notifyInfo("sizeof Hash Table = %d MB", hashtable->sizeBufferUsed / 1024 / 1024);

        tableFileMeta = new Table<FileMeta>(buffer + hashtable->sizeBufferUsed, countFile); /* Initialize file meta table. */
        Debug::notifyInfo("sizeof File Meta Size = %d MB", tableFileMeta->sizeBufferUsed / 1024 / 1024);

        tableDirectoryMeta = new Table<DirectoryMeta>(buffer + hashtable->sizeBufferUsed + tableFileMeta->sizeBufferUsed, countDirectory); /* Initialize directory meta table. */
        Debug::notifyInfo("sizeof Directory Meta Size = %d MB", tableDirectoryMeta->sizeBufferUsed / 1024 / 1024);

        tableBlock = new Table<Block>(bufferBlock, countBlock); /* Initialize block table. */

        this->countNode = countNode;    /* Assign count of nodes. */
        sizeBufferUsed = hashtable->sizeBufferUsed + tableFileMeta->sizeBufferUsed + tableDirectoryMeta->sizeBufferUsed + tableBlock->sizeBufferUsed; /* Size of used bytes in buffer. */
    }
}

提取该构造函数中的要点罗列如下：

• hashtable = new HashTable(buffer, countDirectory + countFile); 新建了一个哈希表，注意哈希项目数量=文件数+目录数；
• 针对文件元数据、目录元数据以及数据块，该哈希表（依照Table模板）为某一种结构建立了一个Table，Table的结构如下所示：

template class Table
{
private:
    Bitmap *bitmapItems;                /* Bitmap for items. */
    std::mutex mutexBitmapItems;        /* Mutex for bitmap for items. */
    T *items;                           /* Items of table. */
    FreeBit *headFreeBit;               /* Head free bit in the chain. */
    FreeBit *tailFreeBit;               /* Tail free bit in the chain. */

public:
    uint64_t sizeBufferUsed;            /* Size of used bytes in buffer. */
    bool create(uint64_t *index, T *item); /* Create an item. */
    bool create(uint64_t *index);       /* Create an item. No item will be assigned. */
    bool get(uint64_t index, T *item);  /* Get an item. */
    bool get(uint64_t index, T *item, uint64_t *address);
    bool put(uint64_t index, T *item);  /* Put an item. */
    bool put(uint64_t index, T *item, uint64_t *address);
    bool remove(uint64_t index);        /* Remove an item. */
    uint64_t countSavedItems();         /* Saved items count. */
    uint64_t countTotalItems();         /* Total items count. */
    Table(char *buffer, uint64_t count); /* Constructor of table. */
    ~Table();                           /* Destructor of table. */
};

我们再来看一个哈希函数，代码如下所示：

NodeHash Storage::getNodeHash(UniqueHash *hashUnique)
{
    if (hashUnique == NULL) {
        fprintf(stderr, "Storage::getNodeHash: buffer is null.\n");
        exit(EXIT_FAILURE);             /* Exit due to null unique hash. */
    } else {
        return ((hashUnique->value[3] % countNode) + 1); /* Return node hash. From 1 to countNode. */
    }
}

这是一个节点哈希（外部哈希），用来指定某一个服务器节点处理该文件服务请求。

TxManager.cpp

（内容请见系列下一章博客）

filesystem.cpp

（内容请见系列下一章博客）