php7 HashTable实现及对比php5.X版本内存消耗优势

The new zval implementation

Before getting to the actual hashtable, I’d like to take a quick look at the new zval structure and highlight how it differs from the old one. The zval struct is defined as follows:

struct _zval_struct {
	zend_value value;
	union {
		struct {
			ZEND_ENDIAN_LOHI_4(
				zend_uchar type,
				zend_uchar type_flags,
				zend_uchar const_flags,
				zend_uchar reserved)
		} v;
		uint32_t type_info;
	} u1;
	union {
		uint32_t var_flags;
		uint32_t next;       /* hash collision chain */
		uint32_t cache_slot; /* literal cache slot */
		uint32_t lineno;     /* line number (for ast nodes) */
	} u2;
};

 

The new hashtable implementation

With all the preliminaries behind us, we can finally look at the new hashtable implementation used by PHP 7. Lets start by looking at the bucket structure:

typedef struct _Bucket {
	zend_ulong        h;
	zend_string      *key;
	zval              val;
} Bucket;

The main hashtable structure is more interesting:

typedef struct _HashTable {
	uint32_t          nTableSize;
	uint32_t          nTableMask;
	uint32_t          nNumUsed;
	uint32_t          nNumOfElements;
	zend_long         nNextFreeElement;
	Bucket           *arData;
	uint32_t         *arHash;
	dtor_func_t       pDestructor;
	uint32_t          nInternalPointer;
	union {
		struct {
			ZEND_ENDIAN_LOHI_3(
				zend_uchar    flags,
				zend_uchar    nApplyCount,
				uint16_t      reserve)
		} v;
		uint32_t flags;
	} u;
} HashTable;

 

Order of elements

The arData array stores the elements in order of insertion. So the first array element will be stored in arData[0], the second in arData[1] etc. This does not in any way depend on the used key, only the order of insertion matters here.

So if you store five elements in the hashtable, slots arData[0] to arData[4] will be used and the next free slot is arData[5]. We remember this number in nNumUsed. You may wonder: Why do we store this separately, isn’t it the same as nNumOfElements?

It is, but only as long as only insertion operations are performed. If an element is deleted from a hashtable, we obviously don’t want to move all elements in arData that occur after the deleted element in order to have a continuous array again. Instead we simply mark the deleted value with an IS_UNDEF zval type.

As an example, consider the following code:

$array = [
	'foo' => 0,
	'bar' => 1,
	0     => 2,
	'xyz' => 3,
	2     => 4
];
unset($array[0]);
unset($array['xyz']);

This will result in the following arData structure:

nTableSize     = 8
nNumOfElements = 3
nNumUsed       = 5

[0]: key="foo", val=int(0)
[1]: key="bar", val=int(1)
[2]: val=UNDEF
[3]: val=UNDEF
[4]: h=2, val=int(4)
[5]: NOT INITIALIZED
[6]: NOT INITIALIZED
[7]: NOT INITIALIZED

nTableSize 初始为8

nNumOfElements 当前有效的元素（nNumUsed减去了UNDEF标记的元素）

 nNumUsed 已存在的元素(因为unset key为0 和 xyz的元素，作了UNDEF 标记))

As you can see the first five arData elements have been used, but elements at position 2 (key 0) and 3 (key 'xyz') have been replaced with an IS_UNDEF tombstone, because they were unset. These elements will just remain wasted memory for now. However, once nNumUsed reaches nTableSize PHP will try compact the arData array, by dropping any UNDEF entries that have been added along the way. Only if all buckets really contain a value the arData will be reallocated to twice the size.

The new way of maintaining array order has several advantages over the doubly linked list that was used in PHP 5.x. One obvious advantage is that we save two pointers per bucket, which corresponds to 8/16 bytes. Additionally it means that iterating an array looks roughly as follows:

uint32_t i;
for (i = 0; i < ht->nNumUsed; ++i) {
	Bucket *b = &ht->arData[i];
	if (Z_ISUNDEF(b->val)) continue;

	// do stuff with bucket
}

注意此处判断： 循环中，if( Z_ISUNDEF(b -> val))  continue;了 检查zval中是否是UNDEF，是则跳过接着循环

 

Hashtable lookup

Until now we have only discussed how PHP arrays represent order. The actual hashtable lookup uses the second arHash array, which consists of uint32_t values. The arHash array has the same size (nTableSize) as arData and both are actually allocated as one chunk of memory.

The hash returned from the hashing function (DJBX33A for string keys) is a 32-bit or 64-bit unsigned integer, which is too large to directly use as an index into the hash array. We first need to adjust it to the table size using a modulus operation. Instead of hash % ht->nTableSize we use hash & (ht->nTableSize - 1), which is the same if the size is a power of two, but doesn’t require expensive integer division. The value ht->nTableSize - 1 is stored in ht->nTableMask.

Next, we look up the index idx = ht->arHash[hash & ht->nTableMask] in the hash array. This index corresponds to the head of the collision resolution list. So ht->arData[idx] is the first entry we have to examine. If the key stored there matches the one we’re looking for, we’re done.

Otherwise we must continue to the next element in the collision resolution list. The index to this element is stored in bucket->val.u2.next, which are the normally unused last four bytes of the zval structure that get a special meaning in this context. We continue traversing this linked list (which uses indexes instead of pointers) until we either find the right bucket or hit an INVALID_IDX- which means that an element with the given key does not exist.

In code, the lookup mechanism looks like this:

zend_ulong h = zend_string_hash_val(key);
uint32_t idx = ht->arHash[h & ht->nTableMask];
while (idx != INVALID_IDX) {
	Bucket *b = &ht->arData[idx];
	if (b->h == h && zend_string_equals(b->key, key)) {
		return b;
	}
	idx = Z_NEXT(b->val); // b->val.u2.next
}
return NULL;

 

Empty hashtables

Empty hashtables get a bit of special treating both in PHP 5.x and PHP 7. If you create an empty array [] chances are pretty good that you won’t actually insert any elements into it. As such the arData/arHash arrays will only be allocated when the first element is inserted into the hashtable.

To avoid checking for this special case in many places, a small trick is used: While the nTableSizeis set to either the hinted size or the default value of 8, the nTableMask (which is usually nTableSize - 1) is set to zero. This means that hash & ht->nTableMask will always result in the value zero as well.

So the arHash array for this case only needs to have one element (with index zero) that contains an INVALID_IDX value (this special array is called uninitialized_bucket and is allocated statically). When a lookup is performed, we always find the INVALID_IDX value, which means that the key has not been found (which is exactly what you want for an empty table).

Memory utilization

This should cover the most important aspects of the PHP 7 hashtable implementation. First lets summarize why the new implementation uses less memory. I’ll only use the numbers for 64bit systems here and only look at the per-element size, ignoring the main HashTable structure (which is less significant asymptotically).

In PHP 5.x a whopping 144 bytes per element were required. In PHP 7 the value is down to 36 bytes, or 32 bytes for the packed case. Here’s where the difference comes from:

• Zvals are not individually allocated, so we save 16 bytes allocation overhead.
• Buckets are not individually allocated, so we save another 16 bytes of allocation overhead.
• Zvals are 16 bytes smaller for simple values.
• Keeping order no longer needs 16 bytes for a doubly linked list, instead the order is implicit.
• The collision list is now singly linked, which saves 8 bytes. Furthermore it’s now an index list and the index is embedded into the zval, so effectively we save another 8 bytes.
• As the zval is embedded into the bucket, we no longer need to store a pointer to it. Due to details of the previous implementation we actually save two pointers, so that’s another 16 bytes.
• The length of the key is no longer stored in the bucket, which is another 8 bytes. However, if the key is actually a string and not an integer, the length still has to be stored in the zend_string structure. The exact memory impact in this case is hard to quantify, because zend_string structures are shared, whereas previously hashtables had to copy the string if it wasn’t interned.
• The array containing the collision list heads is now index based, so saves 4 bytes per element. For packed arrays it is not necessary at all, in which case we save another 4 bytes.

However it should be clearly said that this summary is making things look better than they really are in several respects. First of all, the new hashtable implementation uses a lot more embedded (as opposed to separately allocated) structures. How can this negatively affect things?

If you look at the actually measured numbers at the start of this article, you’ll find that on 64bit PHP 7 an array with 100000 elements took 4.00 MiB of memory. In this case we’re dealing with a packed array, so we would actually expect 32 * 100000 = 3.05 MiB memory utilization. The reason behind this is that we allocate everything in powers of two. The nTableSize will be 2^17 = 131072 in this case, so we’ll allocate 32 * 131072 bytes of memory (which is 4.00 MiB).

Of course the previous hashtable implementation also used power of two allocations. However it only allocated an array with bucket pointers in this way (where each pointer is 8 bytes). Everything else was allocated on demand. So in PHP 7 we loose 32 * 31072 (0.95 MiB) in unused memory, while in PHP 5.x we only waste 8 * 31072 (0.24 MiB).

 

原文地址：https://nikic.github.io/2014/12/22/PHPs-new-hashtable-implementation.html