TensorFlow学习笔记

 @  tf.name_scope(name, default_name=None, values=None) 

定义命名空间（或定义域），可以嵌套使用，用于管理该命名空间下的东西，并用于session graph的可视化

@ Tensorflow 中的 op:

Ops are say the core of tensorflow.
TensorFlow is a programming system in which you represent computations as graphs. Nodes in the graph are called ops (short for operations). An op takes zero or more Tensors, performs some computation, and produces zero or more Tensors.

In [74]: with tf.Session() as sess:
    ...:     zer = tf.zeros(shape=(32, 32))
    ...:     print(zer.op)
    ...:     
name: "zeros_11"
op: "Const"
attr {
  key: "dtype"
  value {
    type: DT_FLOAT
  }
}
attr {
  key: "value"
  value {
    tensor {
      dtype: DT_FLOAT
      tensor_shape {
        dim {
          size: 32
        }
        dim {
          size: 32
        }
      }
      float_val: 0.0
    }
  }
}

zer.op in your example prints out the features of  Tensor zer  in that json-like format

@ Summaries

Summaries provided a way to export condensed information about the model, which is then accessible in tools such as TensorBoard.

@  tf.train.Supervisor()

A training helper that checkpoints models and computes summaries.

  with tf.Graph().as_default():
    ...add operations to the graph...
    # Create a Supervisor that will checkpoint the model in '/tmp/mydir'.
    sv = Supervisor(logdir='/tmp/mydir')
    # Get a TensorFlow session managed by the supervisor.
    with sv.managed_session(FLAGS.master) as sess:
      # Use the session to train the graph.
      while not sv.should_stop():
        sess.run()

 Within the `with sv.managed_session()` block all variables in the graph have   been initialized.  In addition, a few services have been started to
  checkpoint the model and add summaries to the event log.

  If the program crashes and is restarted, the managed session automatically  reinitialize variables from the most recent checkpoint.

  The supervisor is notified of any exception raised by one of the services.   After an exception is raised, `should_stop()` returns `True`.  In that case  the training loop should also stop.  This is why the training loop has to  check for `sv.should_stop()`.

@  tf.add_to_collection(name, value)

可以简单的理解为在Graph下维护了一个dict,key是名字name，value是一些值得list。这个方法就是将value加入到这个dict中key等于name的list中

@ Tensorflow 中 LSTMStateTuple 和zero_state（）的区别

The two are different things. state_is_tuple is used on LSTM cells because the state of LSTM cells is a tuple. cell.zero_state is the initializer of the state for all RNN cells.
You will generally prefer cell.zero_state function as it will initialize the required state class depending on whether state_is_tuple is true or not.

See this GitHub issue where you can see the cell.zero_state recommended - "use the zero_state function on the cell object".

Another reason why you may want cell.zero_state is because it is agnostic of the type of the cell (LSTM, GRU, RNN) and you can do something like this:

if type == 'GRU':
   cell = BasicGRUCell
else:
   cell = BasicLSTMCell(state_is_tuple=True)

init_state = cell.zero_state(batch_size)

with the initial state being set up OK.

LSTMStateTuple will work only on cells that have the state as a tuple.

When to use LSTMStateTuple?

You'll want to use LSTMStateTuple when you're initializing your state with custom values (passed by the trainer). cell.zero_state() will return the state with all the values equal to 0.0. you can do something like this:

        c = tf.zeros([config.num_layers, self.batch_size, config.hidden_size], tf.float32)
        h = tf.zeros([config.num_layers, self.batch_size, config.hidden_size], tf.float32)
        self._initial_state = (rnn.LSTMStateTuple(h=h, c=c),)

If you want to keep state between batches than you'll have to get it after each batch and add it to your feed_dict the next batch.

See this for an explanation on why LSTM state is a tuple.

@  tf.train.export_meta_graph()

Returns MetaGraphDef proto. Optionally writes it to filename.

This function exports the graph, saver, and collection objects into MetaGraphDef protocol buffer with the intention of it being imported at a later time or location to restart training, run inference, or be a subgraph.

@  RNN中为什么要做clip_by_global_norm来得到gradient, 然后再送进优化器？以及这个函数如何执行的？

RNN的误差和梯度是一个连乘的操作，当梯度大于1，会导致梯度爆炸（exploding ），当梯度小于1，就会导致梯度消失（vanishing ）。

采用梯度修剪策略（ gradient norm clipping strategy ）可以很好的解决梯度爆炸问题。参考。

在tensorflow中的函数：tf.clip_by_global_norm（t_list,   clip_norm, use_norm=None, name=None）就是完成这个的，

t_list是tensor的列表，clip_norm是指定的clip比例，函数返回修剪之后的tensor列表，和输入所有tensor的全局范数(global norm)。

To perform the clipping, the values t_list[i] are set to:

t_list[i] * clip_norm / max(global_norm, clip_norm)

where:

global_norm = sqrt(sum([l2norm(t)**2 for t in t_list]))

如果clip_norm大于global_norm，所有tensor保持不变，否则，逐一修剪！

@ tf.reshape()

当做reshape的时候，某个维度值是-1，代表推断，就是保持原始数值个数不变，根据其他维度的数量来推断所在维度的shape大小。