Tensorflow - Variable

文章均迁移到我的主页 http://zhenlianghe.com

my github: https://github.com/LynnHo

tf.Variable

1. 变量域？
   • tf.name_scope和tf.variable_scope都会对tf.Variable生成的变量域造成影响，tf.variable_scope中的reuse参数对tf.Variable没有影响（本质上是因为tf.Variable受到了tf.variable_scope中同时创建的tf.name_scope的影响）
2. 重名？
   • 当变量名相同的时候，tf会自动打上序号

     with tf.name_scope('s'):    # or tf.variable_scope('s')
         a = tf.Variable(initial_value=10, name='a')
         b = tf.Variable(initial_value=10, name='a')
         print(a.name)
         print(b.name)
     
     [out]
     s/a:0
     s/a_1:0
     

3. 初始化？
   • tf.Variable是用一个tensor来初始化的，

     a = tf.Variable(initial_value=[1, 2])
     b = tf.Variable(initial_value=tf.constant([1, 2]))
     c = tf.Variable(initial_value=tf.random_uniform(shape=(1, 2)))
     d = tf.Variable(initial_value=tf.zeros_initializer()(shape=(1, 2), dtype=tf.int64))
     e = tf.Variable(initial_value=slim.xavier_initializer()(shape=(1, 2)))
     

   • tf.zeros_initializer()返回的是一个对象，对象对应的类有相应的call函数，这个call函数负责产生一个相应类型的tensor
   • slim.xavier_initializer()则返回的是一个函数，调用这个函数能够产生一个相应类型的tensor
4. 变量共享？
   • 用生成的变量去干不同的事情不就共享了嘛
   • tf.Variable产生的变量不能用tf.variable_scope的reuse设置共享，否则会报错

tf.get_variable

1. 变量域？
   • tf.get_variable产生的变量只会受到tf.variable_scope的影响，不受tf.name_scope的影响

     with tf.name_scope('s'):
         a = tf.get_variable(name='a', shape=(10, 10))
     with tf.variable_scope('s'):
         b = tf.get_variable(name='a', shape=(10, 10))
     print(a.name)
     print(b.name)
     
     [out]
     a:0
     s/a:0
     

2. 重名？变量共享？
   • 在同一个域下，重名是会报错的。

     with tf.variable_scope('s'):
         a = tf.get_variable(name='a', shape=(10, 10))
         b = tf.get_variable(name='a')
     
     [out]
     ValueError: Variable s/a already exists, disallowed. Did you mean to set reuse=True in VarScope?
     

     • 可以在需要复用变量之前改变scope的reuse状态

       with tf.variable_scope('s') as s:
           a = tf.get_variable(name='a', shape=(10, 10))
           s.reuse_variables()
           b = tf.get_variable(name='a')
       print(a == b)
       
       [out]
       True
       

     • 也可以设置tf.variable_scope的reuse参数为True来复用已经定义过的同名变量，但如果没定义过而设置reuse=True也是会报错的

       with tf.variable_scope('s', reuse=True):
           a = tf.get_variable(name='a')
       
       [out]
       ValueError: Variable s/a does not exist, or was not created with tf.get_variable(). Did you mean to set reuse=None in VarScope?
       

       with tf.variable_scope('s'):
           a = tf.get_variable(name='a', shape=(10, 10))
       with tf.variable_scope('s', reuse=True):
           b = tf.get_variable(name='a')
       print(a == b)
       
       [out]
       True
       

3. 初始化？

   a = tf.get_variable(name='a', shape=(1, 2), initializer=tf.constant_initializer([1, 2]), dtype=tf.int64)
   b = tf.get_variable(name='b', shape=(1, 2), initializer=tf.random_uniform_initializer())
   c = tf.get_variable(name='c', shape=(1, 2), initializer=tf.zeros_initializer(), dtype=tf.int64)
   d = tf.get_variable(name='d', shape=(1, 2), initializer=slim.xavier_initializer())
   

   可见，只要给定相应的initializer就可以了，但是要注意dtype的设置，只有设置tf.get_variable的dtype参数才能正确生效，设置initializer的dtype参数是无效的

slim层里面的variable

1. 注意，slim里面的variable生成机制实际上是和tf.get_variable是一样的，所以特性也是一样的，比如说变量域只受tf.variable_scope影响而不受tf.name_scope影响
2. 层的命名
   1. 自动命名变量域，每一个slim层都有一个scope参数，如果不设置这个参数(默认为None)，会有以下两种情况
      • 在同一个上下问管理器中(with tf.variable_scope('s'):)定义层，slim会按生成顺序自动命名变量域(本质上就是因为slim层里面利用了with tf.variable_scope(None, default_name, ...)的机制)

        x = tf.placeholder(tf.float32, shape=[None, 10])
        with tf.variable_scope('s'):
            a = slim.fully_connected(x, 10)
            b = slim.fully_connected(a, 10)
        for var in tf.trainable_variables():
            print(var.name)
        
        [out]  
        s/fully_connected/weights:0
        s/fully_connected/biases:0
        s/fully_connected_1/weights:0
        s/fully_connected_1/biases:0
        

      • 在不同的上下问管理器中定义层，但域名是一样的，slim将报错
        • 报错的例子

          x = tf.placeholder(tf.float32, shape=[None, 10])
          with tf.variable_scope('s'):
              a = slim.fully_connected(x, 10)
          with tf.variable_scope('s'):
              b = slim.fully_connected(x, 10)
          for var in tf.trainable_variables():
              print(var.name)
          
          [out]                
          Variable s/fully_connected/weights already exists, disallowed. Did you mean to set reuse=True in VarScope?
          

        • 报错的例子

          x = tf.placeholder(tf.float32, shape=[None, 10])
          with tf.variable_scope('s'):
              a = slim.layer_norm(x)
          with tf.variable_scope('s'):
              b = slim.layer_norm(x)
          for var in tf.trainable_variables():
              print(var.name)
          
          [out]
          ValueError: Variable s/LayerNorm/beta already exists, disallowed. Did you mean to set reuse=True in VarScope?
          

   2. 手动命名变量域，顾名思义。需要注意以下情况
      • 在同一个域中，如果两个层设置的scope参数是同一个名字，那么slim将报错

        • 报错的例子

          x = tf.placeholder(tf.float32, shape=[None, 2])
          with tf.variable_scope('s'):
              a = slim.fully_connected(x, 2, scope='a')
          with tf.variable_scope('s'): # 在这个例子中，这一行可有可无，效果相同
              b = slim.fully_connected(x, 2, scope='a')
          for var in tf.trainable_variables():
              print(var.name)
          
          [out]
          Variable s/a/weights already exists, disallowed. Did you mean to set reuse=True in VarScope?
          

        • 报错的例子

          x = tf.placeholder(tf.float32, shape=[None, 10])
          with tf.variable_scope('s'):
              a = slim.layer_norm(x, scope='a')
          with tf.variable_scope('s'): # 在这个例子中，这一行可有可无，效果相同
              b = slim.layer_norm(x, scope='a')
          
          [out]
          ValueError: Variable s/a/beta already exists, disallowed. Did you mean to set reuse=True in VarScope?
          

3. 那么最合理的变量共享方式？？？实际上是和tf.get_variable定义的变量的共享机制是一样的，用reuse参数

   x = tf.placeholder(tf.float32, shape=[None, 2])
   with tf.variable_scope('s'):
       y = slim.fully_connected(x, 2, weights_initializer=tf.random_normal_initializer())
       a = slim.layer_norm(y)
   with tf.variable_scope('s', reuse=True):
       y = slim.fully_connected(x, 2)
       b = slim.layer_norm(y)
   for var in tf.trainable_variables():
       print(var.name)
   
   sess = tf.Session()
   sess.run(tf.global_variables_initializer())
   print(sess.run(a, feed_dict={x: [[1, 7]]}))
   print(sess.run(b, feed_dict={x: [[1, 7]]}))
   
   [out] 
   s/fully_connected/weights:0
   s/fully_connected/biases:0
   s/LayerNorm/beta:0
   s/LayerNorm/gamma:0
   [[-1.          1.00000012]]
   [[-1.          1.00000012]]