ValueError: Variable lstm_cell/rnn/multi_rnn_cell/cell_0/basic_lstm_cell/kernel already exists

写在前面

最近在学习使用tensorflow构建language model，遇到关于模型重用的问题，就将其归纳总结记录下来，供大家参考，学习交流。

错误提示

48 Traceback (most recent call last): 
49 File "anna_writer.py", line 274, in <module> 
50 samp = generate_samples(checkpoint, 20000, prime="The ") 
51 File "anna_writer.py", line 234, in generate_samples 
52 conf.lstm_size, conf.keep_prob, conf.grad_clip, False) 
53 File "anna_writer.py", line 74, in __init__ 
54 self.add_lstm_cell() 
55 File "anna_writer.py", line 110, in add_lstm_cell 
56 initial_state=self.initial_state) 
57 File "/home/tf1.0/local/lib/python2.7/site-packages/tensorflow/python/ops/rnn.py", line 574, ii n dynamic_rnn 
58 dtype=dtype) 
59 File "/home/tf1.0/local/lib/python2.7/site-packages/tensorflow/python/ops/rnn.py", line 737, ii n _dynamic_rnn_loop 
60 swap_memory=swap_memory) 
61 File "/home/tf1.0/local/lib/python2.7/site-packages/tensorflow/python/ops/control_flow_ops.py"" , line 2770, in while_loop 
62 result = context.BuildLoop(cond, body, loop_vars, shape_invariants) 
63 File "/home/tf1.0/local/lib/python2.7/site-packages/tensorflow/python/ops/control_flow_ops.py"" , line 2599, in BuildLoop 
64 pred, body, original_loop_vars, loop_vars, shape_invariants) 
65 File "/home/tf1.0/local/lib/python2.7/site-packages/tensorflow/python/ops/control_flow_ops.py"" , line 2549, in _BuildLoop 
66 body_result = body(*packed_vars_for_body) 
67 File "/home/tf1.0/local/lib/python2.7/site-packages/tensorflow/python/ops/rnn.py", line 722, ii n _time_step 
68 (output, new_state) = call_cell() 
69 File "/home/tf1.0/local/lib/python2.7/site-packages/tensorflow/python/ops/rnn.py", line 708, ii n  
70 call_cell = lambda: cell(input_t, state) 
71 File "/home/tf1.0/local/lib/python2.7/site-packages/tensorflow/python/ops/rnn_cell_impl.py", ll ine 180, in __call__ 
72 return super(RNNCell, self).__call__(inputs, state) 
73 File "/home/tf1.0/local/lib/python2.7/site-packages/tensorflow/python/layers/base.py", line 444 1, in __call__ 
74 outputs = self.call(inputs, *args, **kwargs) 
75 File "/home/tf1.0/local/lib/python2.7/site-packages/tensorflow/python/ops/rnn_cell_impl.py", l ine 916, in call 
76 cur_inp, new_state = cell(cur_inp, cur_state) 
77 File "/home/tf1.0/local/lib/python2.7/site-packages/tensorflow/python/ops/rnn_cell_impl.py", l ine 752, in __call__ 
78 output, new_state = self._cell(inputs, state, scope) 
79 File "/home/tf1.0/local/lib/python2.7/site-packages/tensorflow/python/ops/rnn_cell_impl.py", l ine 180, in __call__ 
80 return super(RNNCell, self).__call__(inputs, state) 
81 File "/home/tf1.0/local/lib/python2.7/site-packages/tensorflow/python/layers/base.py", line 44 1, in __call__ 
82 outputs = self.call(inputs, *args, **kwargs) 
83 File "/home/tf1.0/local/lib/python2.7/site-packages/tensorflow/python/ops/rnn_cell_impl.py", l ine 383, in call 
84 concat = _linear([inputs, h], 4 * self._num_units, True) 
85 File "/home/tf1.0/local/lib/python2.7/site-packages/tensorflow/python/ops/rnn_cell_impl.py", l ine 1017, in _linear 
86 initializer=kernel_initializer) 
87 File "/home/tf1.0/local/lib/python2.7/site-packages/tensorflow/python/ops/variable_scope.py", line 1065, in get_variable 
88 use_resource=use_resource, custom_getter=custom_getter) 
89 File "/home/tf1.0/local/lib/python2.7/site-packages/tensorflow/python/ops/variable_scope.py", line 962, in get_variable 
90 use_resource=use_resource, custom_getter=custom_getter) 
91 File "/home/tf1.0/local/lib/python2.7/site-packages/tensorflow/python/ops/variable_scope.py", line 360, in get_variable 
92 validate_shape=validate_shape, use_resource=use_resource) 
93 File "/home/tf1.0/local/lib/python2.7/site-packages/tensorflow/python/ops/variable_scope.py", line 1405, in wrapped_custom_getter 
94 *args, **kwargs) 
95 File "/home/tf1.0/local/lib/python2.7/site-packages/tensorflow/python/ops/rnn_cell_impl.py", l ine 183, in _rnn_get_variable 
96 variable = getter(*args, **kwargs) 
97 File "/home/tf1.0/local/lib/python2.7/site-packages/tensorflow/python/ops/rnn_cell_impl.py", l ine 183, in _rnn_get_variable 
98 variable = getter(*args, **kwargs) 
99 File "/home/tf1.0/local/lib/python2.7/site-packages/tensorflow/python/ops/variable_scope.py", line 352, in _true_getter 
100 use_resource=use_resource) 
101 File "/home/tf1.0/local/lib/python2.7/site-packages/tensorflow/python/ops/variable_scope.py", line 664, in _get_single_variable 
102 name, "".join(traceback.format_list(tb)))) 
103 ValueError: Variable lstm_cell/rnn/multi_rnn_cell/cell_0/basic_lstm_cell/kernel already exists, disallowed. Did you mean to set reuse=True in VarScope? Originally defined at:
104 105 File "anna_writer.py", line 110, in add_lstm_cell 
106 initial_state=self.initial_state) 
107 File "anna_writer.py", line 74, in __init__ 
108 self.add_lstm_cell() 
109 File "anna_writer.py", line 172, in train 110 conf.grad_clip, is_training=True)

解决方法

这个问题困扰了我两天，始终找不到解决方案，当我的训练模型和预测模型分开运行时程序没有报错，但是两个程序放在一起运行时就会出现问题，网上搜索的结果大都是关于共享权重的问题，错误提示是ValueError: Variable hello/rnn/basic_lstm_cell/weights already exists,disallowed.，这个error跟我的错误还是有一定的区别的。这些问题主要原因在于使用多层lstm_cell或者双向lstm的时候忽略了定义变量的variable_scope，导致lstm_cell的作用域不一样，但是程序加载的时候并不知道，所以当声明的cell不是同一个的时候，需要用

with tf.variable_scope(name):

来定义不同的作用范围就可以了，具体还要根据实际情况。

而我的问题好像网上还没有这样的解释，我仔细看错误的提示，分析我的代码，当train和predict放在一起的时候，会调用两次class language_model：这时候就会出现系统里应该存在两个不同的lstm_cell模型，但是系统无法辨别出来，所以会提示kernel already exists，而不是weights already exists。
而tensorflow有一个reset_default_graph()函数，我对python多线程不是很清楚，贴一下t f.reset_default_graph()的源码。

def reset_default_graph():
  """Clears the default graph stack and resets the global default graph.

  NOTE: The default graph is a property of the current thread. This
  function applies only to the current thread.  Calling this function while
  a `tf.Session` or `tf.InteractiveSession` is active will result in undefined
  behavior. Using any previously created `tf.Operation` or `tf.Tensor` objects
  after calling this function will result in undefined behavior.
  """

然后在我定义的language_model类中添加这个函数之后之前的问题就解决了。

class language_model:
    def __init__(self, num_classes, batch_size=100, seq_length=50, learning_rate=0.01, num_layers=5, hidden_units=128,
                 keep_prob=0.8, grad_clip=5, is_training=True):
        # 模型的训练和预测放在同一个文件下时如果没有这个函数会报错。
        tf.reset_default_graph()  
        self.learning_rate = learning_rate
        self.num_layers = num_layers
        self.hidden_units = hidden_units
        self.is_training = is_training
        self.keep_prob = keep_prob
        self.grad_clip = grad_clip
        self.num_classes = num_classes

        if self.is_training:
            self.batch_size = batch_size
            self.seq_length = seq_length
        else:
            self.batch_size = 1
            self.seq_length = 1

        with tf.name_scope('add_input_layer'):
            self.add_input_layer()
        with tf.variable_scope('lstm_cell'):
            self.add_multi_cells()
        with tf.name_scope('build_output'):
            self.build_output()
        with tf.name_scope('cost'):
            self.compute_cost()
        with tf.name_scope('train_op'):
            self.train_op()

题外话： tensorflow1.2版本之后，定义多层lstm(MultiRNNCell)与原来的版本改变比较大，可以看考PTB tutorials—Stacking multiple LSTMs.

文中涉及到的代码见：github–anna

参考文献

1、tensorflow1.x版本rnn生成cell 报错解决方案
2、ValueError: Attempt to reuse RNNCell
3、How to reuse weights in MultiRNNCell?
4、ValueError: Attempt to reuse RNNCell with a different variable scope than its first use.