mat转ckpt

SiameseFC-TensorFlow 代码详细注解（一）：预训练模型下载转换测试以及结果可视化

2018年05月27日 15:52:26 StayFoolishAndHappy 阅读数：1428 标签： SiameseFC孪生网络结构tensorflow目标跟踪 更多

个人分类： 目标跟踪SiameseFCPython

版权声明：本文为博主原创文章，未经博主允许不得转载。 https://blog.csdn.net/StayFoolish_Fan/article/details/80467907

说明：该系列博客源码链接为：https://github.com/bilylee/SiamFC-TensorFlow，是实验室同小组的师兄用TensorFlow实现SiameseFC算法的最终公开版本，经过了长时间的打磨，各个模块功能明确，整体可读性和可移植性极好，我相信这对做Tracking的小伙伴来说，是个入门SiameseFC Tracker的特别好的选择。哈哈，觉得代码很棒的小伙伴们可以点个Star哦，也欢迎交流学习和指教。

这篇博客主要的目的就是简单地跑一下实验，让下载的代码能用预训练的模型去测试单个视频，并对结果可视化，从视觉上感受一下这个跟踪算法的效果，至于如果要自己训练自己的模型该如何准备训练数据，如何设计自己的模型，如何训练自己的模型，以及如何评估自己的模型等，这些问题都将在后面的系列博客中慢慢道来。

1： SiameseFC-TensorFlow环境配置

可参考源码中的说明，这里将截图放在这里，大家自行准备可运行的环境。

2：预训练模型下载转换和测试

可参考源代码中的说明，这里也给个截图，然后主要对一些文件做一些详细一点的注解。内容主要有预训练模型和测试视频的下载，模型转换以及转换前后的模型对比检验，视频的测试和结果的可视化。

2.1  预训练模型和测试视频下载

核心文件：scripts/download_assets.py

核心功能：下载SiameseFC的matlab版本预训练模型，同时下载用于测试的单个视频。

1. #! /usr/bin/env python

2. # -*- coding: utf-8 -*-

3. #

4. # Copyright © 2017 bily Huazhong University of Science and Technology

5. #

6. # Distributed under terms of the MIT license.

7.  

8. import os.path as osp

9. import sys

10. import zipfile

11.  

12. import six.moves.urllib as urllib # 数据下载相关库 ， urllib

13.  

14. CURRENT_DIR = osp.dirname(__file__) # 返回当前.py脚本文件的路径

15. ROOT_DIR = osp.join(CURRENT_DIR, '..')

16. sys.path.append(ROOT_DIR) # 增加模块的搜索路径

17.  

18. from utils.misc_utils import mkdir_p # 自己编写的makir_p函数，生成路径

19.  
20.  

21. def download_or_skip(download_url, save_path): # 数据下载函数

22. if not osp.exists(save_path): # 判断数据是否已经下载，避免重复下载

23. print('Downloading: {}'.format(download_url))

24. opener = urllib.request.URLopener()

25. opener.retrieve(download_url, save_path)

26. else:

27. print('File {} exists, skip downloading.'.format(save_path))

28.  
29.  

30. if __name__ == '__main__': # 数据下载准备 主函数

31. assets_dir = osp.join(ROOT_DIR, 'assets') # 添加数据资源保存的路径assets_dir

32.  

33. # Make assets directory

34. mkdir_p(assets_dir) # 生成路径，存储数据资源

35.  

36. # Download the pretrained color model # 下载SiameseFC-color pretrained 模型

37. download_base = 'https://www.robots.ox.ac.uk/~luca/stuff/siam-fc_nets/'

38. model_name = '2016-08-17.net.mat'

39. download_or_skip(download_base + model_name, osp.join(assets_dir, model_name))

40.  

41. # Download the pretrained gray model # 下载SiameseFC-color-gray pretrained 模型

42. download_base = 'https://www.robots.ox.ac.uk/~luca/stuff/siam-fc_nets/'

43. model_name = '2016-08-17_gray025.net.mat'

44. download_or_skip(download_base + model_name, osp.join(assets_dir, model_name))

45.  

46. # Download one test sequence # 下载一个测试视频，供测试和显示

47. download_base = "http://cvlab.hanyang.ac.kr/tracker_benchmark/seq_new/"

48. seq_name = 'KiteSurf.zip'

49. download_or_skip(download_base + seq_name, osp.join(assets_dir, seq_name))

50.  

51. # Unzip the test sequence # 将下载的视频.zip文件解压缩

52. with zipfile.ZipFile(osp.join(assets_dir, seq_name), 'r') as zip_ref:

53. zip_ref.extractall(assets_dir)

2.2  模型转换

核心文件：experiments/SiamFC-3s-color-pretrained.py、SiamFC-3s-gray-pretrained.py

相关文件：scripts/convert_pretrained_model.py,utils/train_utils.py

核心功能：将siameseFC的color和gray模型由matlab格式转换成tensorflow方便读取的格式。

就从最外边的实验封装文件慢慢往里看吧：experiments/SiamFC-3s-color-pretrained.py

1. #! /usr/bin/env python

2. # -*- coding: utf-8 -*-

3. #

4. # Copyright © 2017 bily Huazhong University of Science and Technology

5. #

6. # Distributed under terms of the MIT license.

7.  

8. """Load pretrained color model in the SiamFC paper and save it in the TensorFlow format"""

9. from __future__ import absolute_import

10. from __future__ import division

11. from __future__ import print_function

12.  

13. import os.path as osp

14. import sys

15.  

16. CURRENT_DIR = osp.dirname(__file__)

17. sys.path.append(osp.join(CURRENT_DIR, '..'))

18. print( osp.join(CURRENT_DIR, '..')) # 添加搜索路径

19.  

20. from configuration import LOG_DIR # 从配置文件中导入log存储的路径

21. from scripts.convert_pretrained_model import ex # 导入对应的experiment

22. # 这里只是实验的表皮，具体实验还得看scripts.convert_pretrained_model详细内容

23. if __name__ == '__main__':

24. RUN_NAME = 'SiamFC-3s-color-pretrained'

25. ex.run(config_updates={'model_config': {'embed_config': {'embedding_checkpoint_file': '/workspace/czx/Projects/SiamFC-TensorFlow/assets/2016-08-17.net.mat',

26. 'train_embedding': False, },

27. },

28. 'train_config': {'train_dir': osp.join(LOG_DIR, 'track_model_checkpoints', RUN_NAME), },

29. 'track_config': {'log_dir': osp.join(LOG_DIR, 'track_model_inference', RUN_NAME), }

30. }, # 实验运行管理，这里的话就是根据需要更新一些配置文件中的参数

31. options={'--name': RUN_NAME,

32. '--force': True,

33. '--enforce_clean': False,

34. })

模型转换实验文件：scripts/convert_pretrained_model.py

1. #! /usr/bin/env python

2. # -*- coding: utf-8 -*-

3. #

4. # Copyright © 2017 bily Huazhong University of Science and Technology

5. #

6. # Distributed under terms of the MIT license.

7.  

8. """Convert the matlab-pretrained model into TensorFlow format"""

9.  

10. from __future__ import absolute_import

11. from __future__ import division

12. from __future__ import print_function

13.  

14. import logging

15. import os

16. import os.path as osp

17. import sys

18.  

19. import numpy as np

20. import tensorflow as tf

21.  

22. CURRENT_DIR = osp.dirname(__file__)

23. sys.path.append(osp.join(CURRENT_DIR, '..')) # 添加搜索路径

24.  

25. import configuration

26. import siamese_model

27. from utils.misc_utils import auto_select_gpu, save_cfgs

28.  

29. # Set GPU

30. os.environ['CUDA_VISIBLE_DEVICES'] = auto_select_gpu() # 自动选择GPU

31. tf.logging.set_verbosity(tf.logging.DEBUG)

32.  

33. from sacred import Experiment # 更好地进行实验管理

34.  

35. ex = Experiment(configuration.RUN_NAME)

36.  
37.  

38. @ex.config # 加载参数配置

39. def configurations():

40. # Add configurations for current script, for more details please see the documentation of `sacred`.

41. model_config = configuration.MODEL_CONFIG

42. train_config = configuration.TRAIN_CONFIG

43. track_config = configuration.TRACK_CONFIG

44.  
45.  

46. @ex.automain # 主函数

47. def main(model_config, train_config, track_config):

48. # Create training directory

49. train_dir = train_config['train_dir'] # 创建训练路径

50. if not tf.gfile.IsDirectory(train_dir):

51. tf.logging.info('Creating training directory: %s', train_dir)

52. tf.gfile.MakeDirs(train_dir)

53.  

54. # Build the Tensorflow graph

55. g = tf.Graph()

56. with g.as_default(): # 默认graph

57. # Set fixed seed

58. np.random.seed(train_config['seed'])

59. tf.set_random_seed(train_config['seed'])

60. # 实际上单纯地转换这样的一个预训练模型格式是不需要在这里调用siameseFC构建的，但是整份代码，将这样的

61. # 一种预训练模型加载看成是模型训练的一个初始化，而模型转换根据配置参数进行的一个初始化方式。

62. model = siamese_model.SiameseModel(model_config, train_config, mode='inference')

63. model.build()

64.  

65. # Save configurations for future reference

66. save_cfgs(train_dir, model_config, train_config, track_config)

67.  

68. saver = tf.train.Saver(tf.global_variables(),

69. max_to_keep=train_config['max_checkpoints_to_keep'])

70.  

71. # Dynamically allocate GPU memory

72. gpu_options = tf.GPUOptions(allow_growth=True)

73. sess_config = tf.ConfigProto(gpu_options=gpu_options)

74.  

75. sess = tf.Session(config=sess_config)

76. model_path = tf.train.latest_checkpoint(train_config['train_dir'])

77.  

78. if not model_path:

79. # Initialize all variables

80. sess.run(tf.global_variables_initializer())

81. sess.run(tf.local_variables_initializer())

82. start_step = 0

83. # 因为在这里我们转换预训练模型的格式的话只需要设置if后面的参数即可

84. # Load pretrained embedding model if needed

85. if model_config['embed_config']['embedding_checkpoint_file']:

86. model.init_fn(sess) # 这是模型初始化的一个方法，后面的给出具体调用了啥函数

87.  

88. else:

89. logging.info('Restore from last checkpoint: {}'.format(model_path))

90. sess.run(tf.local_variables_initializer())

91. saver.restore(sess, model_path)

92. start_step = tf.train.global_step(sess, model.global_step.name) + 1

93.  

94. checkpoint_path = osp.join(train_config['train_dir'], 'model.ckpt')

95. saver.save(sess, checkpoint_path, global_step=start_step) # 保存为.ckpt

这里贴出model.init_fn的内容吧，且先不要深究siamese_model.py里面的其他内容

1. def setup_embedding_initializer(self):

2. """Sets up the function to restore embedding variables from checkpoint."""

3. embed_config = self.model_config['embed_config']

4. if embed_config['embedding_checkpoint_file']: # 上面说过模型转换的时候是有设置matlab文件路径的

5. # Restore Siamese FC models from .mat model files # 这才是加载.mat model的函数

6. initialize = load_mat_model(embed_config['embedding_checkpoint_file'],

7. 'convolutional_alexnet/', 'detection/')

8.  

9. def restore_fn(sess): # 初始化方式，下面赋值给self.init_fn了

10. tf.logging.info("Restoring embedding variables from checkpoint file %s",

11. embed_config['embedding_checkpoint_file'])

12. sess.run([initialize])

13.  

14. self.init_fn = restore_fn

所以到这里，我们就真正进入到如何转换.mat为.ckpt文件了：

load_mat_model函数在utils/train_utils.py文件里，这里还是贴一下这个文件里的内容吧。

1. #! /usr/bin/env python

2. # -*- coding: utf-8 -*-

3. #

4. # Copyright © 2017 bily Huazhong University of Science and Technology

5. #

6. # Distributed under terms of the MIT license.

7.  

8. """Utilities for model construction"""

9. from __future__ import absolute_import

10. from __future__ import division

11. from __future__ import print_function

12.  

13. import re

14.  

15. import numpy as np

16. import tensorflow as tf

17. from scipy import io as sio

18.  

19. from utils.misc_utils import get_center

20.  

21. # 在训练过程中构建groudtruth用到，用返回的结果和siamese网络得到的score map计算loss

22. def construct_gt_score_maps(response_size, batch_size, stride, gt_config=None):

23. """Construct a batch of groundtruth score maps

24.  

25. Args:

26. response_size: A list or tuple with two elements [ho, wo]

27. batch_size: An integer e.g., 16

28. stride: Embedding stride e.g., 8

29. gt_config: Configurations for groundtruth generation

30.  

31. Return:

32. A float tensor of shape [batch_size] + response_size

33. """

34. with tf.name_scope('construct_gt'):

35. ho = response_size[0]

36. wo = response_size[1]

37. y = tf.cast(tf.range(0, ho), dtype=tf.float32) - get_center(ho)

38. x = tf.cast(tf.range(0, wo), dtype=tf.float32) - get_center(wo)

39. [Y, X] = tf.meshgrid(y, x)

40.  

41. def _logistic_label(X, Y, rPos, rNeg): # 构建一个高斯的二值groundtruth响应图

42. # dist_to_center = tf.sqrt(tf.square(X) + tf.square(Y)) # L2 metric

43. dist_to_center = tf.abs(X) + tf.abs(Y) # Block metric

44. Z = tf.where(dist_to_center <= rPos,

45. tf.ones_like(X),

46. tf.where(dist_to_center < rNeg,

47. 0.5 * tf.ones_like(X),

48. tf.zeros_like(X)))

49. return Z

50. # 且先留意这里构建的groundtruth和siamese网络的stride有关

51. rPos = gt_config['rPos'] / stride

52. rNeg = gt_config['rNeg'] / stride

53. gt = _logistic_label(X, Y, rPos, rNeg)

54.  

55. # Duplicate a batch of maps

56. gt_expand = tf.reshape(gt, [1] + response_size)

57. gt = tf.tile(gt_expand, [batch_size, 1, 1])

58. return gt

59. # 从matlab模型文件存储路径读取模型参数

60. def get_params_from_mat(matpath):

61. """Get parameter from .mat file into parms(dict)"""

62.  

63. def squeeze(vars_):

64. # Matlab save some params with shape (*, 1)

65. # However, we don't need the trailing dimension in TensorFlow.

66. if isinstance(vars_, (list, tuple)):

67. return [np.squeeze(v, 1) for v in vars_]

68. else:

69. return np.squeeze(vars_, 1)

70.  

71. netparams = sio.loadmat(matpath)["net"]["params"][0][0]

72. params = dict() # 将模型数据以dict形式存储起来

73. # 既然看到了这里，自己也不防单独将matlab模型加载起来，看看里面都是什么样的形式

74. for i in range(netparams.size):

75. param = netparams[0][i]

76. name = param["name"][0]

77. value = param["value"]

78. value_size = param["value"].shape[0]

79.  

80. match = re.match(r"([a-z]+)([0-9]+)([a-z]+)", name, re.I)

81. if match:

82. items = match.groups()

83. elif name == 'adjust_f':

84. params['detection/weights'] = squeeze(value)

85. continue

86. elif name == 'adjust_b':

87. params['detection/biases'] = squeeze(value)

88. continue

89. else:

90. raise Exception('unrecognized layer params')

91.  

92. op, layer, types = items

93. layer = int(layer)

94. if layer in [1, 3]:

95. if op == 'conv': # convolution

96. if types == 'f':

97. params['conv%d/weights' % layer] = value

98. elif types == 'b':

99. value = squeeze(value)

100. params['conv%d/biases' % layer] = value

101. elif op == 'bn': # batch normalization

102. if types == 'x':

103. m, v = squeeze(np.split(value, 2, 1))

104. params['conv%d/BatchNorm/moving_mean' % layer] = m

105. params['conv%d/BatchNorm/moving_variance' % layer] = np.square(v)

106. elif types == 'm':

107. value = squeeze(value)

108. params['conv%d/BatchNorm/gamma' % layer] = value

109. elif types == 'b':

110. value = squeeze(value)

111. params['conv%d/BatchNorm/beta' % layer] = value

112. else:

113. raise Exception

114. elif layer in [2, 4]:

115. if op == 'conv' and types == 'f':

116. b1, b2 = np.split(value, 2, 3)

117. else:

118. b1, b2 = np.split(value, 2, 0)

119. if op == 'conv':

120. if types == 'f':

121. params['conv%d/b1/weights' % layer] = b1

122. params['conv%d/b2/weights' % layer] = b2

123. elif types == 'b':

124. b1, b2 = squeeze(np.split(value, 2, 0))

125. params['conv%d/b1/biases' % layer] = b1

126. params['conv%d/b2/biases' % layer] = b2

127. elif op == 'bn':

128. if types == 'x':

129. m1, v1 = squeeze(np.split(b1, 2, 1))

130. m2, v2 = squeeze(np.split(b2, 2, 1))

131. params['conv%d/b1/BatchNorm/moving_mean' % layer] = m1

132. params['conv%d/b2/BatchNorm/moving_mean' % layer] = m2

133. params['conv%d/b1/BatchNorm/moving_variance' % layer] = np.square(v1)

134. params['conv%d/b2/BatchNorm/moving_variance' % layer] = np.square(v2)

135. elif types == 'm':

136. params['conv%d/b1/BatchNorm/gamma' % layer] = squeeze(b1)

137. params['conv%d/b2/BatchNorm/gamma' % layer] = squeeze(b2)

138. elif types == 'b':

139. params['conv%d/b1/BatchNorm/beta' % layer] = squeeze(b1)

140. params['conv%d/b2/BatchNorm/beta' % layer] = squeeze(b2)

141. else:

142. raise Exception

143.  

144. elif layer in [5]:

145. if op == 'conv' and types == 'f':

146. b1, b2 = np.split(value, 2, 3)

147. else:

148. b1, b2 = squeeze(np.split(value, 2, 0))

149. assert op == 'conv', 'layer5 contains only convolution'

150. if types == 'f':

151. params['conv%d/b1/weights' % layer] = b1

152. params['conv%d/b2/weights' % layer] = b2

153. elif types == 'b':

154. params['conv%d/b1/biases' % layer] = b1

155. params['conv%d/b2/biases' % layer] = b2

156.  

157. return params

158. # .mat模型数据加载转换为.ckpt格式进行存储

159. def load_mat_model(matpath, embed_scope, detection_scope=None):

160. """Restore SiameseFC models from .mat model files"""

161. params = get_params_from_mat(matpath)

162.  

163. assign_ops = []

164.  

165. def _assign(ref_name, params, scope=embed_scope):

166. var_in_model = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES,

167. scope + ref_name)[0]

168. var_in_mat = params[ref_name]

169. op = tf.assign(var_in_model, var_in_mat)

170. assign_ops.append(op)

171.  

172. for l in range(1, 6):

173. if l in [1, 3]:

174. _assign('conv%d/weights' % l, params)

175. # _assign('conv%d/biases' % l, params)

176. _assign('conv%d/BatchNorm/beta' % l, params)

177. _assign('conv%d/BatchNorm/gamma' % l, params)

178. _assign('conv%d/BatchNorm/moving_mean' % l, params)

179. _assign('conv%d/BatchNorm/moving_variance' % l, params)

180. elif l in [2, 4]:

181. # Branch 1

182. _assign('conv%d/b1/weights' % l, params)

183. # _assign('conv%d/b1/biases' % l, params)

184. _assign('conv%d/b1/BatchNorm/beta' % l, params)

185. _assign('conv%d/b1/BatchNorm/gamma' % l, params)

186. _assign('conv%d/b1/BatchNorm/moving_mean' % l, params)

187. _assign('conv%d/b1/BatchNorm/moving_variance' % l, params)

188. # Branch 2

189. _assign('conv%d/b2/weights' % l, params)

190. # _assign('conv%d/b2/biases' % l, params)

191. _assign('conv%d/b2/BatchNorm/beta' % l, params)

192. _assign('conv%d/b2/BatchNorm/gamma' % l, params)

193. _assign('conv%d/b2/BatchNorm/moving_mean' % l, params)

194. _assign('conv%d/b2/BatchNorm/moving_variance' % l, params)

195. elif l in [5]:

196. # Branch 1

197. _assign('conv%d/b1/weights' % l, params)

198. _assign('conv%d/b1/biases' % l, params)

199. # Branch 2

200. _assign('conv%d/b2/weights' % l, params)

201. _assign('conv%d/b2/biases' % l, params)

202. else:

203. raise Exception('layer number must below 5')

204.  

205. if detection_scope:

206. _assign(detection_scope + 'biases', params, scope='')

207.  

208. initialize = tf.group(*assign_ops)

209. return initialize

上面代码中的一些语句的具体含义可以自己利用.mat数据慢慢体会，琢磨一会应该就ok了。

到此为止，我们的模型算是已经转换为.ckpt格式的了，按理说就可以用着个模型直接进行测试评估了，但在此之前，我们往往还需要做的一件事就是确定我们转换得到的.ckpt是没有问题的，进入下一节验证.ckpt没有问题之前把.mat格式的模型截个图看看长什么样，更具体的细节这里就不展开了。

2.3  模型转换验证

核心文件：tests/test_converted_model.py 

核心功能：输入同一张图片01.jpg，SiameseFC-TensorFlow代码加载.mat和转换后的.ckpt模型时运算得到01.jpg的feature map均和原作者matlab版本代码计算得到的feature map相同（差异及其的微小），如此一来既验证了网络设计没问题，同时也验证了.mat模型转换为.ckpt模型时没有问题。

1. #! /usr/bin/env python

2. # -*- coding: utf-8 -*-

3. #

4. # Copyright © 2017 bily Huazhong University of Science and Technology

5. #

6. # Distributed under terms of the MIT license.

7.  

8. """Tests for track model"""

9.  

10. from __future__ import absolute_import

11. from __future__ import division

12. from __future__ import print_function

13.  

14. import os.path as osp

15. import sys

16.  

17. import numpy as np

18. import scipy.io as sio

19. import tensorflow as tf

20. from scipy.misc import imread # Only pillow 2.x is compatible with matlab 2016R

21.  

22. CURRENT_DIR = osp.dirname(__file__)

23. PARENT_DIR = osp.join(CURRENT_DIR, '..')

24. sys.path.append(PARENT_DIR)

25.  

26. import siamese_model

27. import configuration

28. from utils.misc_utils import load_cfgs

29. # 直接通过model.init_fn加载.mat格式的模型数据计算01.jpg的feature map

30. # 用这个feature map和作者matlab的结果

31. def test_load_embedding_from_mat():

32. """Test if the embedding model loaded from .mat

33. produces the same features as the original MATLAB implementation"""

34. matpath = osp.join(PARENT_DIR, 'assets/2016-08-17.net.mat')

35. test_im = osp.join(CURRENT_DIR, '01.jpg')

36. gt_feat = osp.join(CURRENT_DIR, 'result.mat')

37.  

38. model_config = configuration.MODEL_CONFIG

39. model_config['embed_config']['embedding_name'] = 'convolutional_alexnet'

40. model_config['embed_config']['embedding_checkpoint_file'] = matpath # For SiameseFC

41. model_config['embed_config']['train_embedding'] = False

42.  

43. g = tf.Graph()

44. with g.as_default():

45. model = siamese_model.SiameseModel(model_config, configuration.TRAIN_CONFIG, mode='inference')

46. model.build() # 模型的构建，这里不用深究，后续博客会细说

47. with tf.Session() as sess:

48. # Initialize models

49. init = tf.global_variables_initializer()

50. sess.run(init)

51.  

52. # Load model here

53. model.init_fn(sess)

54.  

55. # Load image

56. im = imread(test_im)

57. im_batch = np.expand_dims(im, 0)

58.  

59. # Feed image

60. feature = sess.run([model.exemplar_embeds], feed_dict={model.examplar_feed: im_batch})

61.  

62. # Compare with features computed from original source code

63. ideal_feature = sio.loadmat(gt_feat)['r']['z_features'][0][0]

64. diff = feature - ideal_feature

65. diff = np.sqrt(np.mean(np.square(diff)))

66. print('Feature computation difference: {}'.format(diff))

67. print('You should get something like: 0.00892720464617')

68.  

69. def test_load_embedding_from_converted_TF_model():

70. """Test if the embedding model loaded from converted TensorFlow checkpoint

71. produces the same features as the original implementation"""

72. checkpoint = osp.join(PARENT_DIR, 'Logs/SiamFC/track_model_checkpoints/SiamFC-3s-color-pretrained')

73. test_im = osp.join(CURRENT_DIR, '01.jpg')

74. gt_feat = osp.join(CURRENT_DIR, 'result.mat')

75.  

76. if not osp.exists(checkpoint):

77. raise Exception('SiamFC-3s-color-pretrained is not generated yet.')

78. model_config, train_config, track_config = load_cfgs(checkpoint)

79.  

80. # Build the model

81. g = tf.Graph()

82. with g.as_default():

83. model = siamese_model.SiameseModel(model_config, train_config, mode='inference')

84. model.build()

85.  

86. with tf.Session() as sess:

87. # Load model here

88. saver = tf.train.Saver(tf.global_variables())

89. if osp.isdir(checkpoint):

90. model_path = tf.train.latest_checkpoint(checkpoint)

91. else:

92. model_path = checkpoint

93.  

94. saver.restore(sess, model_path)

95.  

96. # Load image

97. im = imread(test_im)

98. im_batch = np.expand_dims(im, 0)

99.  

100. # Feed image

101. feature = sess.run([model.exemplar_embeds], feed_dict={model.examplar_feed: im_batch})

102.  

103. # Compare with features computed from original source code

104. ideal_feature = sio.loadmat(gt_feat)['r']['z_features'][0][0]

105. diff = feature - ideal_feature

106. diff = np.sqrt(np.mean(np.square(diff)))

107. print('Feature computation difference: {}'.format(diff))

108. print('You should get something like: 0.00892720464617')

109.  
110.  

111. def test():

112. test_load_embedding_from_mat()

113. test_load_embedding_from_converted_TF_model()

114.  
115.  

116. if __name__ == '__main__':

117. test()

 

上面验证代码大体逻辑很清晰，至于模型构建的内容在此可以先跳过，后面再详细讲讲，下面是运行该测试脚本的结果截图，上面是.mat结果,下面是.ckpt结果，可以看到这两个结果是一样的：

2.4  用预训练模型在视频上进行测试

核心文件：scripts/run_tracking.py

核心功能：用预训练模型在新的视频上进行测试，视频是上面下载过测KiteSurf。

这份代码涉及到后面的视频测试，这里先只需要会用该脚本就好，你只要会到对应的log 路径下找到你对应视频测试的一些保存结果就好，因为这些结果都是后面可视化的数据，至于保存的数据是什么，保存数据的这个代码在哪个文件里后面自然会涉及到。

1. #! /usr/bin/env python

2. # -*- coding: utf-8 -*-

3. #

4. # Copyright © 2017 bily Huazhong University of Science and Technology

5. #

6. # Distributed under terms of the MIT license.

7.  

8. r"""Generate tracking results for videos using Siamese Model"""

9.  

10. from __future__ import absolute_import

11. from __future__ import division

12. from __future__ import print_function

13.  

14. import logging

15. import os

16. import os.path as osp

17. import sys

18. from glob import glob

19.  

20. import tensorflow as tf

21. from sacred import Experiment

22.  

23. CURRENT_DIR = osp.dirname(__file__)

24. sys.path.append(osp.join(CURRENT_DIR, '..'))

25.  

26. from inference import inference_wrapper

27. from inference.tracker import Tracker

28. from utils.infer_utils import Rectangle

29. from utils.misc_utils import auto_select_gpu, mkdir_p, sort_nicely, load_cfgs

30.  

31. ex = Experiment()

32.  
33.  

34. @ex.config # 模型和测试视频

35. def configs():

36. checkpoint = '/workspace/czx/Projects/SiamFC-TensorFlow/Logs/SiamFC/track_model_checkpoints/SiamFC-3s-color-pretrained'

37. input_files = '/workspace/czx/Projects/SiamFC-TensorFlow/assets/KiteSurf'

38.  
39.  

40. @ex.automain

41. def main(checkpoint, input_files):

42. os.environ['CUDA_VISIBLE_DEVICES'] = auto_select_gpu()

43.  

44. model_config, _, track_config = load_cfgs(checkpoint)

45. track_config['log_level'] = 1

46.  

47. g = tf.Graph()

48. with g.as_default(): # 模型测试时候需要构建的，且先不要深究

49. model = inference_wrapper.InferenceWrapper()

50. restore_fn = model.build_graph_from_config(model_config, track_config, checkpoint)

51. g.finalize()

52. # 这一块就是你存储测试视频结果数据的路径，记得去看看都有哪些数据。

53. if not osp.isdir(track_config['log_dir']):

54. logging.info('Creating inference directory: %s', track_config['log_dir'])

55. mkdir_p(track_config['log_dir'])

56.  

57. video_dirs = []

58. for file_pattern in input_files.split(","):

59. video_dirs.extend(glob(file_pattern))

60. logging.info("Running tracking on %d videos matching %s", len(video_dirs), input_files)

61.  

62. gpu_options = tf.GPUOptions(allow_growth=True)

63. sess_config = tf.ConfigProto(gpu_options=gpu_options)

64.  

65. with tf.Session(graph=g, config=sess_config) as sess:

66. restore_fn(sess)

67.  

68. tracker = Tracker(model, model_config=model_config, track_config=track_config)

69.  

70. for video_dir in video_dirs:

71. if not osp.isdir(video_dir):

72. logging.warning('{} is not a directory, skipping...'.format(video_dir))

73. continue

74.  

75. video_name = osp.basename(video_dir)

76. video_log_dir = osp.join(track_config['log_dir'], video_name)

77. mkdir_p(video_log_dir)

78.  

79. filenames = sort_nicely(glob(video_dir + '/img/*.jpg'))

80. first_line = open(video_dir + '/groundtruth_rect.txt').readline()

81. bb = [int(v) for v in first_line.strip().split(',')]

82. init_bb = Rectangle(bb[0] - 1, bb[1] - 1, bb[2], bb[3]) # 0-index in python

83. # 返回的跟踪结果

84. trajectory = tracker.track(sess, init_bb, filenames, video_log_dir)

85. with open(osp.join(video_log_dir, 'track_rect.txt'), 'w') as f:

86. for region in trajectory:

87. rect_str = '{},{},{},{}\n'.format(region.x + 1, region.y + 1,

88. region.width, region.height)

89. f.write(rect_str)

好了，你现在已经可以用预训练模型测试你想测试的视频了，接下来为了能更好地分析问题，将你测试的一些结果都进行可视化，在此之前还是瞧瞧咱们测试前后都有哪些数据输入和输出：

输入：解压缩之后的KiteSurf，包括了存在img文件夹中的图像数据和配置文件.cfg和groudtruth_rect.txt文件。

输出：每张图片跟踪到的目标的bbox和scale，以及每张图片的respones响应，还有crop出来的图片，最后呢就是像groundth_rect.txt一样的一个跟踪结果文件track_rect.txt。

2.5  视频测试结果可视化 

核心文件：scripts/show_tracking.py

核心功能：模型测试跟踪，这里也先只贴出文件内容，且先不深究。

1. #! /usr/bin/env python

2. # -*- coding: utf-8 -*-

3. #

4. # Copyright © 2017 bily Huazhong University of Science and Technology

5. #

6. # Distributed under terms of the MIT license.

7.  

8. import os.path as osp

9. import sys

10.  

11. from sacred import Experiment

12.  

13. ex = Experiment()

14.  

15. import numpy as np

16. from matplotlib.pyplot import imread, Rectangle

17.  

18. CURRENT_DIR = osp.abspath(osp.dirname(__file__))

19. sys.path.append(osp.join(CURRENT_DIR, ".."))

20.  

21. from utils.videofig import videofig

22.  
23.  

24. def readbbox(file):

25. with open(file, 'r') as f:

26. lines = f.readlines()

27. bboxs = [[float(val) for val in line.strip().replace(' ', ',').replace('\t', ',').split(',')] for line in lines]

28. return bboxs

29.  
30.  

31. def create_bbox(bbox, color):

32. return Rectangle((bbox[0], bbox[1]), bbox[2], bbox[3],

33. fill=False, # remove background\n",

34. edgecolor=color)

35.  
36.  

37. def set_bbox(artist, bbox):

38. artist.set_xy((bbox[0], bbox[1]))

39. artist.set_width(bbox[2])

40. artist.set_height(bbox[3])

41.  
42.  

43. @ex.config

44. def configs():

45. videoname = 'KiteSurf'

46. runname = 'SiamFC-3s-color-pretrained'

47. data_dir = '/workspace/czx/Projects/SiamFC-TensorFlow/assets/'

48. track_log_dir = '/workspace/czx/Projects/SiamFC-TensorFlow/Logs/SiamFC/track_model_inference/{}/{}'.format(runname, videoname)

49.  
50.  

51. @ex.automain

52. def main(videoname, data_dir, track_log_dir):

53. track_log_dir = osp.join(track_log_dir)

54. video_data_dir = osp.join(data_dir, videoname)

55. te_bboxs = readbbox(osp.join(track_log_dir, 'track_rect.txt'))

56. gt_bboxs = readbbox(osp.join(video_data_dir, 'groundtruth_rect.txt'))

57. num_frames = len(gt_bboxs)

58.  

59. def redraw_fn(ind, axes):

60. ind += 1

61. input_ = imread(osp.join(track_log_dir, 'image_cropped{}.jpg'.format(ind)))

62. response = np.load(osp.join(track_log_dir, 'response{}.npy'.format(ind)))

63. org_img = imread(osp.join(data_dir, videoname, 'img', '{:04d}.jpg'.format(ind + 1)))

64. gt_bbox = gt_bboxs[ind]

65. te_bbox = te_bboxs[ind]

66.  

67. bbox = np.load(osp.join(track_log_dir, 'bbox{}.npy'.format(ind)))

68.  

69. if not redraw_fn.initialized:

70. ax1, ax2, ax3 = axes

71. redraw_fn.im1 = ax1.imshow(input_)

72. redraw_fn.im2 = ax2.imshow(response)

73. redraw_fn.im3 = ax3.imshow(org_img)

74.  

75. redraw_fn.bb1 = create_bbox(bbox, color='red')

76. redraw_fn.bb2 = create_bbox(gt_bbox, color='green')

77. redraw_fn.bb3 = create_bbox(te_bbox, color='red')

78.  

79. ax1.add_patch(redraw_fn.bb1)

80. ax3.add_patch(redraw_fn.bb2)

81. ax3.add_patch(redraw_fn.bb3)

82.  

83. redraw_fn.text = ax3.text(0.03, 0.97, 'F:{}'.format(ind), fontdict={'size': 10, },

84. ha='left', va='top',

85. bbox={'facecolor': 'red', 'alpha': 0.7},

86. transform=ax3.transAxes)

87.  

88. redraw_fn.initialized = True

89. else:

90. redraw_fn.im1.set_array(input_)

91. redraw_fn.im2.set_array(response)

92. redraw_fn.im3.set_array(org_img)

93. set_bbox(redraw_fn.bb1, bbox)

94. set_bbox(redraw_fn.bb2, gt_bbox)

95. set_bbox(redraw_fn.bb3, te_bbox)

96. redraw_fn.text.set_text('F: {}'.format(ind))

97.  

98. redraw_fn.initialized = False

99.  

100. videofig(int(num_frames) - 1, redraw_fn,

101. grid_specs={'nrows': 2, 'ncols': 2, 'wspace': 0, 'hspace': 0},

102. layout_specs=['[0, 0]', '[0, 1]', '[1, :]'])

运行show_tracking.py脚本你应该就能得到这么一个视频，下面是两个截图，左上角是crop出来的图片，右上角是对应的响应图，下面是视频帧中原始的图片。

有了上面的一些可视化结果你就很方便进行下一步的分析了，比如上面两个图片中第一个中的响应比第二个要小很多，从crop的图片来看，很容易分析出人的侧脸且低头并有一定的运动模糊导致最后的响应偏低，对于其他视频也是可以如此分析的。如果你想测试其他的视频分析跟踪效果，可以去下载OTB100，VOT等数据集上的视频进行评估。

3：小结

1：这篇博客看到这里，你可以了解的有：

（1）预训练模型有哪些，如何下载；

（2）如何将.mat格式模型转换为.ckpt格式模型并大致知道模型里面都有哪些参数，如何验证转换是否正确；

（3）在不需要深入了解模型设计、训练、测试和评估的基础上 能够利用已有脚本对单个视频进行测试和可视化分析。

2：这篇博客只谈到了预训练模型的测试问题，那么接下来还会进一步介绍的有：

（1）数据预处理：如果要自己重新训练模型，用什么数据，又该如何进行预处理？

（2）模型设计：SiameseFC模型具体是什么样的？比如说网络结构是怎样的，Loss怎么构建等。

（3）模型训练：如何训练自己的模型？比如说数据怎么加载，怎么设置参数训练等。

（4）模型评估：单个视频测试的具体实现以及如何在OTB数据集上进行评估等。

3：好了，到这了呢基本上第一篇就写完了，代码贴的可能有点多，当然可能有很多疏漏的地方，欢迎指教和交流，我是认为这份代码是相当棒的，尤其是对于做跟踪的小伙伴而言，大家要是也这么觉得，那就点个Star吧，哈哈。