在准备好ChineseNER_input_data后，开始跑NER出现的问题

第一次跑

C:\Users\Administrator\Desktop\ChineseNER>python main.py --train=True --clean=T

Building prefix dict from E:\python3\lib\site-packages\jieba\dict.txt ...
Loading model from cache C:\Users\ADMINI~1\AppData\Local\Temp\jieba.cache
Loading model cost 1.144 seconds.
Prefix dict has been built succesfully.
['O']
Traceback (most recent call last):
  File "main.py", line 225, in
    if __name__ == "__main__":
  File "E:\python3\lib\site-packages\tensorflow\python\platform\app.py", line 48
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "main.py", line 219, in main
    clean(FLAGS)
  File "main.py", line 106, in train
    # load data sets
  File "C:\Users\Administrator\Desktop\ChineseNER\loader.py", line 33, in load_s
    assert len(word) >= 2, print([word[0]])

AssertionError: None

改错：将loader.py", line 33,的

assert len(word) >= 2, print([word[0]]) 

注释掉

第二次跑

C:\Users\Administrator\Desktop\ChineseNER>python main.py --train=True --clean=T

Building prefix dict from E:\python3\lib\site-packages\jieba\dict.txt ...
Loading model from cache C:\Users\ADMINI~1\AppData\Local\Temp\jieba.cache
Loading model cost 1.261 seconds.
Prefix dict has been built succesfully.
Found 1418 unique words (165783 in total)
Loading pretrained embeddings from wiki_100.utf8...
Found 20 unique named entity tags
1 / 0 / 1 sentences in train / dev / test.
2018-07-11 15:55:36,811 - log\train.log - INFO - num_chars      :       1541
2018-07-11 15:55:36,812 - log\train.log - INFO - char_dim       :       100
2018-07-11 15:55:36,812 - log\train.log - INFO - num_tags       :       20
2018-07-11 15:55:36,813 - log\train.log - INFO - seg_dim        :       20
2018-07-11 15:55:36,816 - log\train.log - INFO - lstm_dim       :       100
2018-07-11 15:55:36,817 - log\train.log - INFO - batch_size     :       20
2018-07-11 15:55:36,817 - log\train.log - INFO - emb_file       :       wiki_100
2018-07-11 15:55:36,817 - log\train.log - INFO - clip           :       5
2018-07-11 15:55:36,817 - log\train.log - INFO - dropout_keep   :       0.5
2018-07-11 15:55:36,818 - log\train.log - INFO - optimizer      :       adam
2018-07-11 15:55:36,818 - log\train.log - INFO - lr             :       0.001
2018-07-11 15:55:36,821 - log\train.log - INFO - tag_schema     :       iobes
2018-07-11 15:55:36,821 - log\train.log - INFO - pre_emb        :       True
2018-07-11 15:55:36,821 - log\train.log - INFO - zeros          :       False
2018-07-11 15:55:36,821 - log\train.log - INFO - lower          :       True
2018-07-11 15:55:36.822838: W c:\tf_jenkins\home\workspace\release-win\m\windows
SSE instructions, but these are available on your machine and could speed up CPU
2018-07-11 15:55:36.822946: W c:\tf_jenkins\home\workspace\release-win\m\windows
SSE2 instructions, but these are available on your machine and could speed up CP
2018-07-11 15:55:36.823026: W c:\tf_jenkins\home\workspace\release-win\m\windows
SSE3 instructions, but these are available on your machine and could speed up CP
2018-07-11 15:55:36.823112: W c:\tf_jenkins\home\workspace\release-win\m\windows
SSE4.1 instructions, but these are available on your machine and could speed up
2018-07-11 15:55:36.823194: W c:\tf_jenkins\home\workspace\release-win\m\windows
SSE4.2 instructions, but these are available on your machine and could speed up
2018-07-11 15:55:36.823275: W c:\tf_jenkins\home\workspace\release-win\m\windows
AVX instructions, but these are available on your machine and could speed up CPU
E:\python3\lib\site-packages\tensorflow\python\ops\gradients_impl.py:93: UserWar
nt of memory.
  "Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
2018-07-11 15:55:41,928 - log\train.log - INFO - Created model with fresh parame
Loading pretrained embeddings from wiki_100.utf8...
WARNING: 1 invalid lines
Loaded 16115 pretrained embeddings.
1532 / 1541 (99.4160%) words have been initialized with pretrained embeddings.
1532 found directly, 0 after lowercasing, 0 after lowercasing + zero.
2018-07-11 15:55:46,029 - log\train.log - INFO - Load pre-trained embedding.
2018-07-11 15:55:46,029 - log\train.log - INFO - start training
2018-07-11 15:59:24,735 - log\train.log - INFO - evaluate:dev
Traceback (most recent call last):
  File "main.py", line 225, in
    if __name__ == "__main__":
  File "E:\python3\lib\site-packages\tensorflow\python\platform\app.py", line 48
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "main.py", line 219, in main
    clean(FLAGS)
  File "main.py", line 185, in train


  File "main.py", line 85, in evaluate
    ner_results = model.evaluate(sess, data, id_to_tag)
  File "C:\Users\Administrator\Desktop\ChineseNER\utils.py", line 66, in test_ne
    eval_lines = return_report(output_file)
  File "C:\Users\Administrator\Desktop\ChineseNER\conlleval.py", line 282, in re
    counts = evaluate(f)
  File "C:\Users\Administrator\Desktop\ChineseNER\conlleval.py", line 74, in eva
    for line in iterable:
  File "E:\python3\lib\codecs.py", line 711, in __next__
    return next(self.reader)
  File "E:\python3\lib\codecs.py", line 642, in __next__
    line = self.readline()
  File "E:\python3\lib\codecs.py", line 555, in readline
    data = self.read(readsize, firstline=True)
  File "E:\python3\lib\codecs.py", line 501, in read
    newchars, decodedbytes = self.decode(data, self.errors)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbb in position 0: invalid

第三次跑

C:\Users\Administrator\Desktop\ChineseNER>python main.py --train=True --clean=T
Building prefix dict from E:\python3\lib\site-packages\jieba\dict.txt ...
Loading model from cache C:\Users\ADMINI~1\AppData\Local\Temp\jieba.cache
Loading model cost 1.229 seconds.
Prefix dict has been built succesfully.
Found 1419 unique words (165783 in total)
Loading pretrained embeddings from wiki_100.utf8...
Found 20 unique named entity tags
1 / 0 / 1 sentences in train / dev / test.
2018-07-11 16:09:21,009 - log\train.log - INFO - num_chars      :       1542
2018-07-11 16:09:21,009 - log\train.log - INFO - char_dim       :       100
2018-07-11 16:09:21,010 - log\train.log - INFO - num_tags       :       20
2018-07-11 16:09:21,011 - log\train.log - INFO - seg_dim        :       20
2018-07-11 16:09:21,011 - log\train.log - INFO - lstm_dim       :       100
2018-07-11 16:09:21,011 - log\train.log - INFO - batch_size     :       20
2018-07-11 16:09:21,012 - log\train.log - INFO - emb_file       :       wiki_100
2018-07-11 16:09:21,012 - log\train.log - INFO - clip           :       5
2018-07-11 16:09:21,012 - log\train.log - INFO - dropout_keep   :       0.5
2018-07-11 16:09:21,012 - log\train.log - INFO - optimizer      :       adam
2018-07-11 16:09:21,012 - log\train.log - INFO - lr             :       0.001
2018-07-11 16:09:21,013 - log\train.log - INFO - tag_schema     :       iobes
2018-07-11 16:09:21,013 - log\train.log - INFO - pre_emb        :       True
2018-07-11 16:09:21,013 - log\train.log - INFO - zeros          :       False
2018-07-11 16:09:21,013 - log\train.log - INFO - lower          :       True
2018-07-11 16:09:21.055211: W c:\tf_jenkins\home\workspace\release-win\m\windows
SSE instructions, but these are available on your machine and could speed up CPU
2018-07-11 16:09:21.055343: W c:\tf_jenkins\home\workspace\release-win\m\windows
SSE2 instructions, but these are available on your machine and could speed up CP
2018-07-11 16:09:21.055407: W c:\tf_jenkins\home\workspace\release-win\m\windows
SSE3 instructions, but these are available on your machine and could speed up CP
2018-07-11 16:09:21.055627: W c:\tf_jenkins\home\workspace\release-win\m\windows
SSE4.1 instructions, but these are available on your machine and could speed up
2018-07-11 16:09:21.055688: W c:\tf_jenkins\home\workspace\release-win\m\windows
SSE4.2 instructions, but these are available on your machine and could speed up
2018-07-11 16:09:21.055757: W c:\tf_jenkins\home\workspace\release-win\m\windows
AVX instructions, but these are available on your machine and could speed up CPU
E:\python3\lib\site-packages\tensorflow\python\ops\gradients_impl.py:93: UserWar
nt of memory.
  "Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
2018-07-11 16:09:26,867 - log\train.log - INFO - Created model with fresh parame
Loading pretrained embeddings from wiki_100.utf8...
WARNING: 1 invalid lines
Loaded 16115 pretrained embeddings.
1532 / 1542 (99.3515%) words have been initialized with pretrained embeddings.
1532 found directly, 0 after lowercasing, 0 after lowercasing + zero.
2018-07-11 16:09:31,209 - log\train.log - INFO - Load pre-trained embedding.
2018-07-11 16:09:31,209 - log\train.log - INFO - start training
Traceback (most recent call last):
  File "E:\python3\lib\site-packages\tensorflow\python\client\session.py", line
    return fn(*args)
  File "E:\python3\lib\site-packages\tensorflow\python\client\session.py", line
    status, run_metadata)
  File "E:\python3\lib\contextlib.py", line 66, in __exit__
    next(self.gen)
  File "E:\python3\lib\site-packages\tensorflow\python\framework\errors_impl.py"
    pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: All dimensions exc
         [[Node: optimizer/gradients/char_embedding/concat_grad/ConcatOffset = C
ncat_grad/mod, optimizer/gradients/char_embedding/concat_grad/ShapeN, optimizer/


During handling of the above exception, another exception occurred:


Traceback (most recent call last):
  File "main.py", line 225, in
    if __name__ == "__main__":
  File "E:\python3\lib\site-packages\tensorflow\python\platform\app.py", line 48
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "main.py", line 219, in main
    clean(FLAGS)
  File "main.py", line 176, in train
    for batch in train_manager.iter_batch(shuffle=True):
  File "C:\Users\Administrator\Desktop\ChineseNER\model.py", line 219, in run_st
    feed_dict)
  File "E:\python3\lib\site-packages\tensorflow\python\client\session.py", line
    run_metadata_ptr)
  File "E:\python3\lib\site-packages\tensorflow\python\client\session.py", line
    feed_dict_string, options, run_metadata)
  File "E:\python3\lib\site-packages\tensorflow\python\client\session.py", line
    target_list, options, run_metadata)
  File "E:\python3\lib\site-packages\tensorflow\python\client\session.py", line
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: All dimensions exc
         [[Node: optimizer/gradients/char_embedding/concat_grad/ConcatOffset = C
ncat_grad/mod, optimizer/gradients/char_embedding/concat_grad/ShapeN, optimizer/


Caused by op 'optimizer/gradients/char_embedding/concat_grad/ConcatOffset', defi
  File "main.py", line 225, in
    if __name__ == "__main__":
  File "E:\python3\lib\site-packages\tensorflow\python\platform\app.py", line 48
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "main.py", line 219, in main
    clean(FLAGS)
  File "main.py", line 171, in train
    with tf.Session(config=tf_config) as sess:
  File "C:\Users\Administrator\Desktop\ChineseNER\utils.py", line 172, in create
    model = Model_class(config)
  File "C:\Users\Administrator\Desktop\ChineseNER\model.py", line 80, in __init_
    grads_vars = self.opt.compute_gradients(self.loss)
  File "E:\python3\lib\site-packages\tensorflow\python\training\optimizer.py", l
    colocate_gradients_with_ops=colocate_gradients_with_ops)
  File "E:\python3\lib\site-packages\tensorflow\python\ops\gradients_impl.py", l
    grad_scope, op, func_call, lambda: grad_fn(op, *out_grads))
  File "E:\python3\lib\site-packages\tensorflow\python\ops\gradients_impl.py", l
    return grad_fn()  # Exit early
  File "E:\python3\lib\site-packages\tensorflow\python\ops\gradients_impl.py", l
    grad_scope, op, func_call, lambda: grad_fn(op, *out_grads))
  File "E:\python3\lib\site-packages\tensorflow\python\ops\array_grad.py", line
    op, grad, start_value_index=0, end_value_index=-1, dim_index=-1)
  File "E:\python3\lib\site-packages\tensorflow\python\ops\array_grad.py", line
    offset = gen_array_ops._concat_offset(non_neg_concat_dim, sizes)
  File "E:\python3\lib\site-packages\tensorflow\python\ops\gen_array_ops.py", li
    shape=shape, name=name)
  File "E:\python3\lib\site-packages\tensorflow\python\framework\op_def_library.
    op_def=op_def)
  File "E:\python3\lib\site-packages\tensorflow\python\framework\ops.py", line 2
    original_op=self._default_original_op, op_def=op_def)
  File "E:\python3\lib\site-packages\tensorflow\python\framework\ops.py", line 1
    self._traceback = _extract_stack()


...which was originally created as op 'char_embedding/concat', defined at:
  File "main.py", line 225, in
    if __name__ == "__main__":
[elided 3 identical lines from previous traceback]
  File "C:\Users\Administrator\Desktop\ChineseNER\utils.py", line 172, in create
    model = Model_class(config)
  File "C:\Users\Administrator\Desktop\ChineseNER\model.py", line 54, in __init_
    embedding = self.embedding_layer(self.char_inputs, self.seg_inputs, config)
  File "C:\Users\Administrator\Desktop\ChineseNER\model.py", line 110, in embedd
    embed = tf.concat(embedding, axis=-1)
  File "E:\python3\lib\site-packages\tensorflow\python\ops\array_ops.py", line 1
    name=name)
  File "E:\python3\lib\site-packages\tensorflow\python\ops\gen_array_ops.py", li
    name=name)
  File "E:\python3\lib\site-packages\tensorflow\python\framework\op_def_library.
    op_def=op_def)
  File "E:\python3\lib\site-packages\tensorflow\python\framework\ops.py", line 2
    original_op=self._default_original_op, op_def=op_def)
  File "E:\python3\lib\site-packages\tensorflow\python\framework\ops.py", line 1
...which was originally created as op 'char_embedding/concat', defined at:                                                                                                            
  File "main.py", line 225, in                                                                                                                                               
    if __name__ == "__main__":                                                                                                                                                        
[elided 3 identical lines from previous traceback]                                                                                                                                    
  File "C:\Users\Administrator\Desktop\ChineseNER\utils.py", line 172, in create                                                                                                      
    model = Model_class(config)                                                                                                                                                       
  File "C:\Users\Administrator\Desktop\ChineseNER\model.py", line 54, in __init_                                                                                                      
    embedding = self.embedding_layer(self.char_inputs, self.seg_inputs, config)                                                                                                       
  File "C:\Users\Administrator\Desktop\ChineseNER\model.py", line 110, in embedd
    embed = tf.concat(embedding, axis=-1)
  File "E:\python3\lib\site-packages\tensorflow\python\ops\array_ops.py", line 1
    name=name)
  File "E:\python3\lib\site-packages\tensorflow\python\ops\gen_array_ops.py", li
    name=name)
  File "E:\python3\lib\site-packages\tensorflow\python\framework\op_def_library.
    op_def=op_def)
  File "E:\python3\lib\site-packages\tensorflow\python\framework\ops.py", line 2
    original_op=self._default_original_op, op_def=op_def)
  File "E:\python3\lib\site-packages\tensorflow\python\framework\ops.py", line 1
    if __name__ == "__main__":
[elided 3 identical lines from previous traceback]
  File "C:\Users\Administrator\Desktop\ChineseNER\utils.py", line 172, in create
    model = Model_class(config)
  File "C:\Users\Administrator\Desktop\ChineseNER\model.py", line 54, in __init_
    embedding = self.embedding_layer(self.char_inputs, self.seg_inputs, config)
  File "C:\Users\Administrator\Desktop\ChineseNER\model.py", line 110, in embedd
    embed = tf.concat(embedding, axis=-1)
  File "E:\python3\lib\site-packages\tensorflow\python\ops\array_ops.py", line 1
    name=name)
  File "E:\python3\lib\site-packages\tensorflow\python\ops\gen_array_ops.py", li
    name=name)
  File "E:\python3\lib\site-packages\tensorflow\python\framework\op_def_library.
    op_def=op_def)
  File "E:\python3\lib\site-packages\tensorflow\python\framework\ops.py", line 2
    original_op=self._default_original_op, op_def=op_def)
  File "E:\python3\lib\site-packages\tensorflow\python\framework\ops.py", line 1
    self._traceback = _extract_stack()


InvalidArgumentError (see above for traceback): All dimensions except 2 must mat
         [[Node: optimizer/gradients/char_embedding/concat_grad/ConcatOffset = C
ncat_grad/mod, optimizer/gradients/char_embedding/concat_grad/ShapeN, optimizer/