Caffe2 创建你的专属数据集（Create Your Own Dataset）[9]

这一节尝试把你的数据转换成caffe2能够使用的形式。这个教程使用Iris的数据集。你可以点击这里查看Ipython Notebook教程。

DB数据格式

Caffe2使用二进制的DB格式来保存数据。Caffe2 DB其实是键-值存储方式的一个美名而已。在键-值（key-value）存储方式里，键是随机生成的，所以batches是独立同分布的。而值（Value）则是真正的数据，他们包含着训练过程中真正用到的数据。所以，DB中保存的数据格式就像下面这样:

key1 value1 key2 value2 key3 value3 ...

在DB中，他把keys和values看成strings。你可以用TensorProtos protobuf来将你要保存的东西保存成DB数据结构。一个TensorProtos protobuf封装了Tensor（多维矩阵），和它的数据类型，形状信息。然后，你可以通过TensorProtosDBInput操作来载入数据到SGD训练过程中。

准备自己的数据

这里，我们向你展示如何创建自己的数据集。为此，我们将会使用UCI Iris数据集。这是一个非常受欢迎的经典的用于分类鸢尾花的数据集。它包含4个代表花的外形特征的实数。这个数据集包含3种鸢尾花。你可以从这里下载数据集。

%matplotlib inline
import urllib2 # 用于从网上下载数据集
import numpy as np
from matplotlib import pyplot
from StringIO import StringIO
from caffe2.python import core, utils, workspace
from caffe2.proto import caffe2_pb2

WARNING:root:This caffe2 python run does not have GPU support. Will run in CPU only mode.
WARNING:root:Debug message: No module named caffe2_pybind11_state_gpu
#如果你在Mac OS下使用homebrew，你可能会遇到一个错误： malloc_zone_unregister() 函数失败.这不是Caffe2的问题，而是因为 homebrew leveldb 的内存分配不兼容. 但这不影响使用。

f = urllib2.urlopen('https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data')
raw_data = f.read()
print('Raw data looks like this:')
print(raw_data[:100] + '...')

输出：

Raw data looks like this:
5.1,3.5,1.4,0.2,Iris-setosa
4.9,3.0,1.4,0.2,Iris-setosa
4.7,3.2,1.3,0.2,Iris-setosa
4.6,3.1,1.5,0.2,...

#将特征保存到一个特征矩阵
features = np.loadtxt(StringIO(raw_data), dtype=np.float32, delimiter=',', usecols=(0, 1, 2, 3))
#把label存到一个特征矩阵中
label_converter = lambda s : {'Iris-setosa':0, 'Iris-versicolor':1, 'Iris-virginica':2}[s]
labels = np.loadtxt(StringIO(raw_data), dtype=np.int, delimiter=',', usecols=(4,), converters={4: label_converter})

在我们开始训练之前，最好将数据集分成训练集和测试集。在这个例子中，让我们随机打乱数据，用前100个数据做训练集，剩余50个数据做测试。当然你也可以用更加复杂的方式，例如使用交叉校验的方式将数据集分成多个训练集和测试集。关于交叉校验的更多信息，请看这里。

random_index = np.random.permutation(150)
features = features[random_index]
labels = labels[random_index]
train_features = features[:100]
train_labels = labels[:100]
test_features = features[100:]
test_labels = labels[100:]

legend = ['rx', 'b+', 'go']
pyplot.title("Training data distribution, feature 0 and 1")
for i in range(3):
    pyplot.plot(train_features[train_labels==i, 0], train_features[train_labels==i, 1], legend[i])
pyplot.figure()
pyplot.title("Testing data distribution, feature 0 and 1")
for i in range(3):
    pyplot.plot(test_features[test_labels==i, 0], test_features[test_labels==i, 1], legend[i])

现在，把数据放进Caffe2的DB中去。在这个DB中，我们将会使用train_xxx作为key，并对于每一个点使用一个TensorProtos对象去储存，一个TensorProtos包含两个tensor：一个是特征，一个是label。我们使用Caffe2的Python DB接口。

# 构建一个TensorProtos protobuf 
feature_and_label = caffe2_pb2.TensorProtos()
feature_and_label.protos.extend([
    utils.NumpyArrayToCaffe2Tensor(features[0]),
    utils.NumpyArrayToCaffe2Tensor(labels[0])])
print('This is what the tensor proto looks like for a feature and its label:')
print(str(feature_and_label))
print('This is the compact string that gets written into the db:')
print(feature_and_label.SerializeToString())

This is what the tensor proto looks like for a feature and its label:
protos {
  dims: 4
  data_type: FLOAT
  float_data: 5.40000009537
  float_data: 3.0
  float_data: 4.5
  float_data: 1.5
}
protos {
  data_type: INT32
  int32_data: 1
}
This is the compact string that gets written into the db:
��������̬@@@�@�?
���"��

现在真正写入DB中去

def write_db(db_type, db_name, features, labels):
    db = core.C.create_db(db_type, db_name, core.C.Mode.write)
    transaction = db.new_transaction()
    for i in range(features.shape[0]):
        feature_and_label = caffe2_pb2.TensorProtos()
        feature_and_label.protos.extend([
            utils.NumpyArrayToCaffe2Tensor(features[i]),
            utils.NumpyArrayToCaffe2Tensor(labels[i])])
        transaction.put(
            'train_%03d'.format(i),
            feature_and_label.SerializeToString())
    # Close the transaction, and then close the db.
    del transaction
    del db

write_db("minidb", "iris_train.minidb", train_features, train_labels)
write_db("minidb", "iris_test.minidb", test_features, test_labels)

现在让我恩创建一个简单的网络，这个网络只包含一个简单的TensorProtosDBInput 操作，用来展示我们如何从创建好的DB中读入数据。

net_proto = core.Net("example_reader")
dbreader = net_proto.CreateDB([], "dbreader", db="iris_train.minidb", db_type="minidb")
net_proto.TensorProtosDBInput([dbreader], ["X", "Y"], batch_size=16)

print("The net looks like this:")
print(str(net_proto.Proto()))

The net looks like this:
name: "example_reader"
op {
  output: "dbreader"
  name: ""
  type: "CreateDB"
  arg {
    name: "db_type"
    s: "minidb"
  }
  arg {
    name: "db"
    s: "iris_train.minidb"
  }
}
op {
  input: "dbreader"
  output: "X"
  output: "Y"
  name: ""
  type: "TensorProtosDBInput"
  arg {
    name: "batch_size"
    i: 16
  }
}

创建网络

workspace.CreateNet(net_proto)

# 先跑一次，然后获取里面的数据
workspace.RunNet(net_proto.Proto().name)
print("The first batch of feature is:")
print(workspace.FetchBlob("X"))
print("The first batch of label is:")
print(workspace.FetchBlob("Y"))

# 再跑一次
workspace.RunNet(net_proto.Proto().name)
print("The second batch of feature is:")
print(workspace.FetchBlob("X"))
print("The second batch of label is:")
print(workspace.FetchBlob("Y"))

The first batch of feature is:
[[ 5.19999981  4.0999999   1.5         0.1       ]
 [ 5.0999999   3.79999995  1.5         0.30000001]
 [ 6.9000001   3.0999999   4.9000001   1.5       ]
 [ 7.69999981  2.79999995  6.69999981  2.        ]
 [ 6.5999999   2.9000001   4.5999999   1.29999995]
 [ 6.30000019  2.79999995  5.0999999   1.5       ]
 [ 7.30000019  2.9000001   6.30000019  1.79999995]
 [ 5.5999999   2.9000001   3.5999999   1.29999995]
 [ 6.5         3.          5.19999981  2.        ]
 [ 5.          3.4000001   1.5         0.2       ]
 [ 6.9000001   3.0999999   5.4000001   2.0999999 ]
 [ 6.          3.4000001   4.5         1.60000002]
 [ 5.4000001   3.4000001   1.70000005  0.2       ]
 [ 6.30000019  2.70000005  4.9000001   1.79999995]
 [ 5.19999981  2.70000005  3.9000001   1.39999998]
 [ 6.19999981  2.9000001   4.30000019  1.29999995]]
The first batch of label is:
[0 0 1 2 1 2 2 1 2 0 2 1 0 2 1 1]
The second batch of feature is:
[[ 5.69999981  2.79999995  4.0999999   1.29999995]
 [ 5.0999999   2.5         3.          1.10000002]
 [ 4.4000001   2.9000001   1.39999998  0.2       ]
 [ 7.          3.20000005  4.69999981  1.39999998]
 [ 5.69999981  2.9000001   4.19999981  1.29999995]
 [ 5.          3.5999999   1.39999998  0.2       ]
 [ 5.19999981  3.5         1.5         0.2       ]
 [ 6.69999981  3.          5.19999981  2.29999995]
 [ 6.19999981  3.4000001   5.4000001   2.29999995]
 [ 6.4000001   2.70000005  5.30000019  1.89999998]
 [ 6.5         3.20000005  5.0999999   2.        ]
 [ 6.0999999   3.          4.9000001   1.79999995]
 [ 5.4000001   3.4000001   1.5         0.40000001]
 [ 4.9000001   3.0999999   1.5         0.1       ]
 [ 5.5         3.5         1.29999995  0.2       ]
 [ 6.69999981  3.          5.          1.70000005]]
The second batch of label is:
[1 1 0 1 1 0 0 2 2 2 2 2 0 0 0 1]

至此，本节教程结束。
转载请注明出处：http://www.jianshu.com/c/cf07b31bb5f2