Debug Tensorflow: tensorflow.python.framework.errors_impl.InvalidArgumentError: OpKernel ‘ConcatV2‘

环境

• Ubuntu 18.04
• Tensorflow-gpu 2.4

报错

Traceback (most recent call last):
  File "nn/cnn.py", line 134, in 
    run()
  File "nn/cnn.py", line 128, in run
    trainer.train(train_dataloader, valid_dataloader, **fit_params)
  File "/home/teletraan/baseline/competition/earth/code/nn/trainer_tf.py", line 28, in train
    valid_epoch_loss, valid_scores = self.valid_loop(valid_dataset)
  File "/home/teletraan/baseline/competition/earth/code/nn/trainer_tf.py", line 81, in valid_loop
    y_valid_preds = tf.concat(y_valid_preds, axis=0)
  File "/home/teletraan/.pyenv/versions/comp/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py", line 201, in wrapper
    return target(*args, **kwargs)
  File "/home/teletraan/.pyenv/versions/comp/lib/python3.7/site-packages/tensorflow/python/ops/array_ops.py", line 1654, in concat
    return gen_array_ops.concat_v2(values=values, axis=axis, name=name)
  File "/home/teletraan/.pyenv/versions/comp/lib/python3.7/site-packages/tensorflow/python/ops/gen_array_ops.py", line 1207, in concat_v2
    _ops.raise_from_not_ok_status(e, name)
  File "/home/teletraan/.pyenv/versions/comp/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 6843, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: OpKernel 'ConcatV2' has constraint on attr 'T' not in NodeDef '[N=0, Tidx=DT_INT32]', KernelDef: 'op: "ConcatV2" device_type: "GPU" constraint { name: "T" allowed_values { list { type: DT_INT32 } } } host_memory_arg: "values" host_memory_arg: "axis" host_memory_arg: "output"' [Op:ConcatV2] name: concat

问题

由于y_valid是空，导致concat操作报错。
我这里的根本原因是batch_size设置太大，大于样本数了，所以出现以上错误。调低batch_size即可