keras 分批训练 详解 - keras进阶系列01

我们今天来讲一讲使用keras框架来进行分批训练
刚入门的深度学习爱好者由于数据量不是很大，倾向于将所有数据读入内存之后直接送入模型进行学习，这样的优点是简单，复杂度小，但是缺点也非常明显：能训练的数据较少，无法训练较大的模型。那些在网上的教程，或者是所谓的“深度学习课程”往往对这个问题避而不谈，估计是商业目的使然。而真正的商用的模型往往需要几天时间甚至几个月时间来训练。那么，用keras怎么来训练这么多的数据呢？我们来一起看一下。
首先，很多入门的同学们都喜欢用model.fit()函数来训练自己的模型，这个函数非常好用，只需要把data和target当作参数传进去就可以了。
而我们今天要介绍的是keras.model类的两个训练方法里面的另外一个：model.fit_genrator()
这个fit_genrator方法就是专门为分批训练打造的，我们需要传入的是一个数据的生成器，以及一些训练的参数：
我们先来具体看一下这个方法的__doc__:（不想看可以直接跳过）

"""Fits the model on data yielded batch-by-batch by a Python generator.

    The generator is run in parallel to the model, for efficiency.
    For instance, this allows you to do real-time data augmentation
    on images on CPU in parallel to training your model on GPU.

    The use of `keras.utils.Sequence` guarantees the ordering
    and guarantees the single use of every input per epoch when
    using `use_multiprocessing=True`.

    Arguments:
        generator: A generator or an instance of `Sequence`
          (`keras.utils.Sequence`)
            object in order to avoid duplicate data
            when using multiprocessing.
            The output of the generator must be either
            - a tuple `(inputs, targets)`
            - a tuple `(inputs, targets, sample_weights)`.
            This tuple (a single output of the generator) makes a single batch.
            Therefore, all arrays in this tuple must have the same length (equal
            to the size of this batch). Different batches may have different
              sizes.
            For example, the last batch of the epoch is commonly smaller than
              the
            others, if the size of the dataset is not divisible by the batch
              size.
            The generator is expected to loop over its data
            indefinitely. An epoch finishes when `steps_per_epoch`
            batches have been seen by the model.
        steps_per_epoch: Total number of steps (batches of samples)
            to yield from `generator` before declaring one epoch
            finished and starting the next epoch. It should typically
            be equal to the number of samples of your dataset
            divided by the batch size.
            Optional for `Sequence`: if unspecified, will use
            the `len(generator)` as a number of steps.
        epochs: Integer, total number of iterations on the data.
        verbose: Verbosity mode, 0, 1, or 2.
        callbacks: List of callbacks to be called during training.
        validation_data: This can be either
            - a generator for the validation data
            - a tuple (inputs, targets)
            - a tuple (inputs, targets, sample_weights).
        validation_steps: Only relevant if `validation_data`
            is a generator. Total number of steps (batches of samples)
            to yield from `generator` before stopping.
            Optional for `Sequence`: if unspecified, will use
            the `len(validation_data)` as a number of steps.
        class_weight: Dictionary mapping class indices to a weight
            for the class.
        max_queue_size: Integer. Maximum size for the generator queue.
            If unspecified, `max_queue_size` will default to 10.
        workers: Integer. Maximum number of processes to spin up
            when using process-based threading.
            If unspecified, `workers` will default to 1. If 0, will
            execute the generator on the main thread.
        use_multiprocessing: Boolean.
            If `True`, use process-based threading.
            If unspecified, `use_multiprocessing` will default to `False`.
            Note that because this implementation relies on multiprocessing,
            you should not pass non-picklable arguments to the generator
            as they can't be passed easily to children processes.
        shuffle: Boolean. Whether to shuffle the order of the batches at
            the beginning of each epoch. Only used with instances
            of `Sequence` (`keras.utils.Sequence`).
            Has no effect when `steps_per_epoch` is not `None`.
        initial_epoch: Epoch at which to start training
            (useful for resuming a previous training run)

    Returns:
        A `History` object.

    Example:

    ```python
        def generate_arrays_from_file(path):
            while 1:
                f = open(path)
                for line in f:
                    # create numpy arrays of input data
                    # and labels, from each line in the file
                    x1, x2, y = process_line(line)
                    yield ({'input_1': x1, 'input_2': x2}, {'output': y})
                f.close()

        model.fit_generator(generate_arrays_from_file('/my_file.txt'),
                            steps_per_epoch=10000, epochs=10)
    ```
    Raises:
        ValueError: In case the generator yields data in an invalid format.
    """

说人话，就是：

1. 我们要传入一个生成器用于在训练的过程中平行的生成训练数据和标签。
2. 这个生成器的返回值的格式可以是：- a tuple (inputs, targets)或者 - a tuple (inputs, targets, sample_weights).那么我们一般是用前者，就是返回data和target，不返回样本权重的那个格式。
3. 我们必须要传入的参数出了上面说的那个generator以外，还有step_per_epoch和epochs。这两个参数是做什么用的呢？epochs就是迭代次数，这个大家都知道。
4. step_per_epoch参数就是每一次迭代（epoch）的步数，说人话就是每一次迭代我要送多少个batch进去进行学习。也就是，这个公式严格成立：step_per_epoch*batch_size==len(train_data)，批大小乘以批数量等于训练集的总长度。
5. 将上述的三个参数generator、step_per_epoch、epochs传进去，我们就可以开始训练了，但是要注意下面的坑：

坑

训练集数据生成器返回的数据格式是两个字典的元组，这两个字典分别只有一个input和一个output的键，如果你的ks模型是Sequential的话，键的名字即是第一层的名字以及最后一层的名字，这个名字我们可以在编译好模型之后打印model.summary()进行查看。更改好之后，整个程序就可以准确无误的跑了～如果对这个还有不明白，我在 https://blog.csdn.net/weixin_42744102/article/details/87193271 中详细提到了这个问题，大家可以看看。

由于本人水平有限，因此如果出现任何错误或者不合理、不得体的地方，欢迎大家积极指出，我将会在最短的时间内进行更正，谢谢大家！本人工作邮箱：[email protected]