图像描述show and tell代码im2txt阅读（TensorFlow官方实现）杂七杂八记录（草稿一：数据预处理部分）
数据预处理部分代码
0 概述
代码位于im2txt/data/build_mscoco_data.py，数据预处理的步骤如下：
- 读取MS COCO训练集和验证集caption文件，提取图像id号、路径（依据图像文件名）、具体描述（利用nltk进行tokenize）
- 按照比例划分具体的训练集/验证集/测试集
- 利用划分后的训练集中的具体描述信息，创建词汇表（按照词频降序），存入文件（格式：单词，词频）；同时创建词汇字典，即word_to_id，便于利用单词获取id号
- 利用划分后的数据集（包含图像id、路径和描述）和词汇字典，将数据保存到TFRecord中
MS COCO图像数据存储在data/mscoco/raw-data/train2014和data/mscoco/raw-data/train2014中，描述文件存储在data/mscoco/raw-data/annotations/captions_train2014.json和data/mscoco/raw-data/annotations/captions_val2014.json中。
数据预处理命令如下：
python im2txt/data/build_mscoco_data.py \
# MS COCO训练集图像存储路径
--train_image_dir="data/mscoco/raw-data/train2014" \
# MS COCO验证集图像存储路径
--val_image_dir="data/mscoco/raw-data/val2014" \
# MS COCO训练集图像描述存储路径
--train_captions_file="data/mscoco/raw-data/annotations/captions_train2014.json" \
# MS COCO验证集图像存储路径
--val_captions_file="data/mscoco/raw-data/annotations/captions_val2014.json" \
# 数据预处理之后保存TFRecord的路径
--output_dir="data/mscoco" \
# 单词词汇表保存位置，格式（单词，词频）
--word_counts_output_file="data/mscoco/word_counts.txt" \
代码的main函数如下：
def main(unused_argv):
def _is_valid_num_shards(num_shards):
"""Returns True if num_shards is compatible with FLAGS.num_threads."""
return num_shards < FLAGS.num_threads or not num_shards % FLAGS.num_threads
assert _is_valid_num_shards(FLAGS.train_shards), (
"Please make the FLAGS.num_threads commensurate with FLAGS.train_shards")
assert _is_valid_num_shards(FLAGS.val_shards), (
"Please make the FLAGS.num_threads commensurate with FLAGS.val_shards")
assert _is_valid_num_shards(FLAGS.test_shards), (
"Please make the FLAGS.num_threads commensurate with FLAGS.test_shards")
if not tf.gfile.IsDirectory(FLAGS.output_dir):
tf.gfile.MakeDirs(FLAGS.output_dir)
# Load image metadata from caption files.
# 打开MS COCO训练集图像描述文件，提取（图像id，图像具体路径，图像参考描述）信息，组织成namedtuple格式
mscoco_train_dataset = _load_and_process_metadata(FLAGS.train_captions_file,
FLAGS.train_image_dir)
# 打开MS COCO验证集图像描述文件，提取（图像id，图像具体路径，图像参考描述）信息，组织成namedtuple格式
mscoco_val_dataset = _load_and_process_metadata(FLAGS.val_captions_file,
FLAGS.val_image_dir)
# Redistribute the MSCOCO data as follows:
# train_dataset = 100% of mscoco_train_dataset + 85% of mscoco_val_dataset.
# val_dataset = 5% of mscoco_val_dataset (for validation during training).
# test_dataset = 10% of mscoco_val_dataset (for final evaluation).
# 按照MS COCO训练集和验证集重新划分具体使用的训练集/验证集/测试集
train_cutoff = int(0.85 * len(mscoco_val_dataset))
val_cutoff = int(0.90 * len(mscoco_val_dataset))
train_dataset = mscoco_train_dataset + mscoco_val_dataset[0:train_cutoff]
val_dataset = mscoco_val_dataset[train_cutoff:val_cutoff]
test_dataset = mscoco_val_dataset[val_cutoff:]
# Create vocabulary from the training captions.
# 提取划分之后训练集图像的所有参考描述
train_captions = [c for image in train_dataset for c in image.captions]
# 根据参考描述创建词汇字典
vocab = _create_vocab(train_captions)
# 处理数据集图像，存储到TFRecord中
_process_dataset("train", train_dataset, vocab, FLAGS.train_shards)
_process_dataset("val", val_dataset, vocab, FLAGS.val_shards)
_process_dataset("test", test_dataset, vocab, FLAGS.test_shards)
1 读取训练集、验证集图像描述文件
利用_load_and_process_metadata(captions_file, image_dir)函数进行处理，把该部分数据整理为[图像id，图像路径，图像参考的描述]格式（A list of ImageMetadata），代码如下：
# 构造一个namedtuple，用于存储该部分信息，可以利用定义的命名（image_id,filename,captions）对数据进行访问
ImageMetadata = namedtuple("ImageMetadata", ["image_id", "filename", "captions"])
def _load_and_process_metadata(captions_file, image_dir):
"""Loads image metadata from a JSON file and processes the captions.
Args:
captions_file: JSON file containing caption annotations. 图像描述文件
image_dir: Directory containing the image files. 图像存储路径
Returns:
A list of ImageMetadata. [图像id，图像路径，图像参考的描述]
"""
with tf.gfile.FastGFile(captions_file, "r") as f:
caption_data = json.load(f)
# Extract the filenames.
id_to_filename = [(x["id"], x["file_name"]) for x in caption_data["images"]]
# Extract the captions. Each image_id is associated with multiple captions.
# 提取每张图片及其对应的参考描述
id_to_captions = {}
for annotation in caption_data["annotations"]:
# 图像id
image_id = annotation["image_id"]
# 图像对应的描述，大多数图片包含5条描述，还有少量包含6条描述、7条描述
caption = annotation["caption"]
id_to_captions.setdefault(image_id, [])
id_to_captions[image_id].append(caption)
assert len(id_to_filename) == len(id_to_captions)
assert set([x[0] for x in id_to_filename]) == set(id_to_captions.keys())
print("Loaded caption metadata for %d images from %s" %
(len(id_to_filename), captions_file))
# Process the captions and combine the data into a list of ImageMetadata.
print("Processing captions.")
image_metadata = []
num_captions = 0
for image_id, base_filename in id_to_filename:
# 构造图像的具体路径
filename = os.path.join(image_dir, base_filename)
# 利用nltk对图像描述进行处理，每条描述添加了和标记
captions = [_process_caption(c) for c in id_to_captions[image_id]]
# 构造数据[图像id，图像路径，图像参考的描述]
image_metadata.append(ImageMetadata(image_id, filename, captions))
num_captions += len(captions)
print("Finished processing %d captions for %d images in %s" %
(num_captions, len(id_to_filename), captions_file))
return image_metadata
def _process_caption(caption):
"""Processes a caption string into a list of tonenized words.
Args:
caption: A string caption.
Returns:
A list of strings; the tokenized caption.
"""
tokenized_caption = [FLAGS.start_word]
tokenized_caption.extend(nltk.tokenize.word_tokenize(caption.lower()))
tokenized_caption.append(FLAGS.end_word)
return tokenized_caption
2 数据集划分
这部分没啥，就是一个再划分。
3 词汇表及词汇字典的构建
上一步，划分得到最终我们使用的训练集之后，我们需要利用训练集中所有图像的参考描述来构建词汇表（根据词频）并建立word_to_id词汇字典（方便模型通过单词获取单词的id编号，然后根据id编号获取该单词的词嵌入向量）。通过_create_vocab(captions)函数实现（在这之前，把训练集所有图像的参考描述组织存储到captions中），代码如下：
def _create_vocab(captions):
"""Creates the vocabulary of word to word_id.
The vocabulary is saved to disk in a text file of word counts. The id of each
word in the file is its corresponding 0-based line number.
Args:
captions: A list of lists of strings. 包含训练集所有描述
Returns:
A Vocabulary object.
"""
print("Creating vocabulary.")
# 利用Counter对描述中单词进行词频统计
counter = Counter()
for c in captions:
counter.update(c)
print("Total words:", len(counter))
# Filter uncommon words and sort by descending count.
# 仅保留词频大于FLAGS.min_word_count的单词，格式保存为（单词，词频）
word_counts = [x for x in counter.items() if x[1] >= FLAGS.min_word_count]
# 按照词频从大到小进行排序
word_counts.sort(key=lambda x: x[1], reverse=True)
print("Words in vocabulary:", len(word_counts))
# Write out the word counts file.
# 把统计得到的单词及其词频信息保存到文件中
with tf.gfile.FastGFile(FLAGS.word_counts_output_file, "w") as f:
f.write("\n".join(["%s %d" % (w, c) for w, c in word_counts]))
print("Wrote vocabulary file:", FLAGS.word_counts_output_file)
# Create the vocabulary dictionary.
# 构建word_to_id词汇字典
# 只保留排序之后的单词，丢弃词频
reverse_vocab = [x[0] for x in word_counts]
unk_id = len(reverse_vocab)
# 构建word_to_id字典，x为单词，y为单词在排序之后词汇表中的位置
vocab_dict = dict([(x, y) for (y, x) in enumerate(reverse_vocab)])
# 其余未被收纳的单词都被定义为4 将训练集/验证集/测试集存储到TFRecord中
利用函数_process_dataset(name, images, vocab, num_shards)进行处理（该函数使用了多线程进行数据处理和存储），其中images为第2步划分之后包含（图像id，图像具体路径，参考描述）的数据，vocab为第3步构建的word_to_id字典，num_shards为存储TFRecord的文件个数（train：256，val：4；test：8）。
相关代码如下：
def _process_dataset(name, images, vocab, num_shards):
"""Processes a complete data set and saves it as a TFRecord.
Args:
name: Unique identifier specifying the dataset.
images: List of ImageMetadata.
vocab: A Vocabulary object.
num_shards: Integer number of shards for the output files.
"""
# Break up each image into a separate entity for each caption.
# 之前是一条记录包含一张图像和该图像的多条描述
# 这里把每一条描述分割，一条记录包含一张图像和该图像的一条描述（因此图像数据会重复）
images = [ImageMetadata(image.image_id, image.filename, [caption])
for image in images for caption in image.captions]
# Shuffle the ordering of images. Make the randomization repeatable.
random.seed(12345)
# 对数据进行随机排序，打乱顺序
random.shuffle(images)
# Break the images into num_threads batches. Batch i is defined as
# images[ranges[i][0]:ranges[i][1]].
# 线程数
num_threads = min(num_shards, FLAGS.num_threads)
# 依据线程数，计算划分到每个线程中处理数据的起始索引
spacing = np.linspace(0, len(images), num_threads + 1).astype(np.int)
ranges = []
threads = []
for i in xrange(len(spacing) - 1):
ranges.append([spacing[i], spacing[i + 1]])
# Create a mechanism for monitoring when all threads are finished.
# Tensorflow线程管理器
coord = tf.train.Coordinator()
# Create a utility for decoding JPEG images to run sanity checks.
# ImageDecoder类，把图像从string数据串转换为RGB数据
# 参考前面博客，图像用tf.gfile.FastGFile().read()打开，用tf.image.decode_jpeg(image, channels=3)进行解码
decoder = ImageDecoder()
# Launch a thread for each batch.
# 创建线程对图像数据进行处理
print("Launching %d threads for spacings: %s" % (num_threads, ranges))
for thread_index in xrange(len(ranges)):
# 处理数据所需参数：线程号，该线程需处理数据索引*，名称，所有数据*，图像解码类*，word_to_id字典*，存储为TFRecord个数*
# *为必须的重要数据
args = (thread_index, ranges, name, images, decoder, vocab, num_shards)
t = threading.Thread(target=_process_image_files, args=args)
t.start()
threads.append(t)
# Wait for all the threads to terminate.
# 把线程加入到线程管理器中
coord.join(threads)
print("%s: Finished processing all %d image-caption pairs in data set '%s'." %
(datetime.now(), len(images), name))
# 线程目标函数
def _process_image_files(thread_index, ranges, name, images, decoder, vocab,
num_shards):
"""Processes and saves a subset of images as TFRecord files in one thread.
Args:
thread_index: Integer thread identifier within [0, len(ranges)].
ranges: A list of pairs of integers specifying the ranges of the dataset to
process in parallel.
name: Unique identifier specifying the dataset.
images: List of ImageMetadata.
decoder: An ImageDecoder object.
vocab: A Vocabulary object.
num_shards: Integer number of shards for the output files.
"""
# Each thread produces N shards where N = num_shards / num_threads. For
# instance, if num_shards = 128, and num_threads = 2, then the first thread
# would produce shards [0, 64).
# 总共需要存储为n_shards个TFRecord文件，一共有num_threads个线程，
# 因此每个线程处理num_shards / num_threads个TFRecord文件（需要验证能整除 ）。
num_threads = len(ranges)
assert not num_shards % num_threads
num_shards_per_batch = int(num_shards / num_threads)
# 该线程一共需要处理ranges[thread_index][0]到ranges[thread_index][1]的图像数据
# 其次，每个线程需要存储num_shards_per_batch个TFRecord文件
# 因此需要对数据进行划分，获取存储到各个TFRecord文件中数据的索引
shard_ranges = np.linspace(ranges[thread_index][0], ranges[thread_index][1],
num_shards_per_batch + 1).astype(int)
# 该线程中需要处理的图像数据量
num_images_in_thread = ranges[thread_index][1] - ranges[thread_index][0]
counter = 0
# 每个TFRecord文件的处理和存储
for s in xrange(num_shards_per_batch):
# Generate a sharded version of the file name, e.g. 'train-00002-of-00010'
# 生成TFRecord文件名
shard = thread_index * num_shards_per_batch + s
output_filename = "%s-%.5d-of-%.5d" % (name, shard, num_shards)
output_file = os.path.join(FLAGS.output_dir, output_filename)
# TFRecord存储器
writer = tf.python_io.TFRecordWriter(output_file)
shard_counter = 0
# 该TFRecord文件需要处理的数据的索引
images_in_shard = np.arange(shard_ranges[s], shard_ranges[s + 1], dtype=int)
for i in images_in_shard:
# 索引i的数据，包括图像id，图像路径和一条参考描述
image = images[i]
# 利用_to_sequence_example函数处理图像数据和其描述
sequence_example = _to_sequence_example(image, decoder, vocab)
if sequence_example is not None:
# 将数据写入TFRecord文件中，
# shard_counter为该TFRecord文件处理的数据量，counter为该线程处理的数据量
writer.write(sequence_example.SerializeToString())
shard_counter += 1
counter += 1
if not counter % 1000:
print("%s [thread %d]: Processed %d of %d items in thread batch." %
(datetime.now(), thread_index, counter, num_images_in_thread))
sys.stdout.flush()
writer.close()
print("%s [thread %d]: Wrote %d image-caption pairs to %s" %
(datetime.now(), thread_index, shard_counter, output_file))
sys.stdout.flush()
shard_counter = 0
print("%s [thread %d]: Wrote %d image-caption pairs to %d shards." %
(datetime.now(), thread_index, counter, num_shards_per_batch))
sys.stdout.flush()
# 处理图像数据和描述数据
def _to_sequence_example(image, decoder, vocab):
"""Builds a SequenceExample proto for an image-caption pair.
Args:
image: An ImageMetadata object.
decoder: An ImageDecoder object.
vocab: A Vocabulary object.
Returns:
A SequenceExample proto.
"""
# 根据图像具体路径，打开图像，格式为string，因此还需要保证能将其解码为RGB数据
with tf.gfile.FastGFile(image.filename, "r") as f:
encoded_image = f.read()
# （所以此处只是验证，没有进行解码，在模型的输入构造部分包括了将输入解码为RGB数据部分）
try:
decoder.decode_jpeg(encoded_image)
except (tf.errors.InvalidArgumentError, AssertionError):
print("Skipping file with invalid JPEG data: %s" % image.filename)
return
# 1、组织图像id和图像像素数据
context = tf.train.Features(feature={
"image/image_id": _int64_feature(image.image_id),
"image/data": _bytes_feature(encoded_image),
})
assert len(image.captions) == 1
caption = image.captions[0]
# 2、组织描述数据，利用传入的word_to_id词汇字典将描述的每个单词转为其id（即在词汇表中的索引）
caption_ids = [vocab.word_to_id(word) for word in caption]
feature_lists = tf.train.FeatureLists(feature_list={
"image/caption": _bytes_feature_list(caption),
"image/caption_ids": _int64_feature_list(caption_ids)
})
sequence_example = tf.train.SequenceExample(
context=context, feature_lists=feature_lists)
return sequence_example
其他记录
- MS COCO数据集并不是每一张图片正好包含5个描述语句，训练集中就有126张图片有6个描述，1张图片有7个描述。（我之前一直以为规规整整每张图像就5个参考描述）
namedtuple有点类似与C语言、C++中的结构体？可以按照其命名访问元素。
DemoData = namedtuple('Student', ['name','age'])
Students = []
Students.append(DemoData('xiaoming', 20))
Students.append(DemoData('xiaofa', 19))
for msg in Students:
print msg.name, msg.age
- 在把数据存储到TFRecord中时，会对图像描述进行划分，保证每一条记录包含一个图像和对应的一条描述（因此图像数据会被多次复制）。