python 读写h5py文件（转载）

原文链接，大家还是看原文吧。（仅供个人资料保存，不喜勿喷）

 

1. Creating HDF5 files

      We first load the numpy and h5py modules

      

import numpy as np
import h5py

Now mock up some simple dummy data to save to our file.

d1 = np.random.random(size = (1000,20))
d2 = np.random.random(size = (1000,200))
print d1.shape, d2.shape

output:(1000, 20) (1000, 200)

The first step to creating a HDF5 file is to initialise it. It uses a very similar syntax to initialising a typical text file in numpy. The first argument provides the filename and location, the second the mode. We’re writing the file, so we provide a w for write access.

hf = h5py.File('data.h5', 'w')

This creates a file object, hf, which has a bunch of associated methods. One is create_dataset, which does what it says on the tin. Just provide a name for the dataset, and the numpy array.

hf.create_dataset('dataset_1', data=d1)
hf.create_dataset('dataset_2', data=d2)

All we need to do now is close the file, which will write all of our work to disk.

hf.close()

2.  Reading HDF5 files

     To open and read data we use the same File method in read mode, r.

      

hf = h5py.File('data.h5', 'r')

   To see what data is in this file, we can call the keys() method on the file object.

hf.keys()

[u'group1']

 We can then grab each dataset we created above using the get method, specifying the name.

n1 = hf.get('dataset_1')
n1

This returns a HDF5 dataset object. To convert this to an array, just call numpy’s array method.

n1 = np.array(n1)
n1.shape

(1000, 20)

hf.close()

3. Groups

Groups are the basic container mechanism in a HDF5 file, allowing hierarchical organisation of the data. Groups are created similarly to datasets, and datsets are then added using the group object.

d1 = np.random.random(size = (100,33))
d2 = np.random.random(size = (100,333))
d3 = np.random.random(size = (100,3333))

hf = h5py.File('data.h5', 'w')

g1 = hf.create_group('group1')

g1.create_dataset('data1',data=d1)
g1.create_dataset('data2',data=d1)

We can also create subfolders. Just specify the group name as a directory format.

g2 = hf.create_group('group2/subfolder')

g2.create_dataset('data3',data=d3)

As before, to read data in irectories and subdirectories use the get method with the full subdirectory path.

group2 = hf.get('group2/subfolder')

group2.items()

[(u'data3', )]

group1 = hf.get('group1')

group1.items()

[(u'data1', ),
 (u'data2', )]

n1 = group1.get('data1')
np.array(n1).shape

(100, 33)

hf.close()

4.  Compression

To save on disk space, while sacrificing read speed, you can compress the data. Just add the compression argument, which can be either gzip, lzf or szip. gzip is the most portable, as it’s available with every HDF5 install, lzf is the fastest but doesn’t compress as effectively as gzip, and szip is a NASA format that is patented up; if you don’t know about it, chances are your organisation doesn’t have the patent, so avoid.

For gzip you can also specify the additional compression_opts argument, which sets the compression level. The default is 4, but it can be an integer between 0 and 9.

hf = h5py.File('data.h5', 'w')

hf.create_dataset('dataset_1', data=d1, compression="gzip", compression_opts=9)
hf.create_dataset('dataset_2', data=d2, compression="gzip", compression_opts=9)

hf.close()