[oneAPI] 基于BERT预训练模型的SWAG问答任务

比赛：https://marketing.csdn.net/p/f3e44fbfe46c465f4d9d6c23e38e0517
Intel® DevCloud for oneAPI：https://devcloud.intel.com/oneapi/get_started/aiAnalyticsToolkitSamples/

基于Intel® DevCloud for oneAPI下的Intel® Optimization for PyTorch

在Intel® DevCloud for oneAPI平台上，我们搭建了实验环境，充分发挥其完全虚拟化的优势，使我们能够专注于模型开发和优化，无需过多关心底层配置和维护工作。为了进一步提升我们的实验效果，我们充分利用了Intel® Optimization for PyTorch，将其应用于我们的PyTorch模型中，从而实现了高效的优化。这一优化策略不仅提升了模型的训练速度和推断性能，还使模型在英特尔硬件上的表现更加卓越，从而加速了实验的整体进程。

在我们的实验中，我们选择了Bert预训练模型，并将其应用于SQuAD英文问答任务中。通过数据预处理、Fine-tuning等步骤，我们成功地将Bert模型应用于问答任务中，取得了令人满意的效果。在这一过程中，Intel® Optimization for PyTorch的应用进一步加速了模型的训练和推断过程，提高了整体效率。

这一实验方案不仅仅是技术上的创新，更是在实际应用中带来的价值。通过提升模型的性能，我们不仅可以更快地训练和部署模型，还可以提供更高质量的问答结果，从而提升用户体验。这对于自然语言处理领域的研究和应用都具有重要意义。

基于BERT预训练模型的SWAG问答任务

SWAG任务是一项常见的自然语言处理任务，旨在对给定的句子上下文中的多个选项进行排列，从而确定最可能的下一句。该任务有助于模型理解句子的语境和逻辑，具有广泛的应用价值。下来介绍该任务的我们使用的解决方案：

• 数据准备： 首先，从SWAG数据集中获取句子上下文和多个选项。每个样本包含一个上下文句子以及四个候选选项，其中一个是正确的。需要将这些文本数据转换成Bert模型可以理解的输入格式。
• 预训练模型选择： 选择合适的预训练Bert模型，如BERT-base或BERT-large。这些模型在大规模语料库上进行了预训练，捕捉了丰富的语义信息。
• Fine-tuning： 将预训练的Bert模型应用于SWAG任务，使用已标注的训练数据进行Fine-tuning。在Fine-tuning时，使用正确的选项作为标签，通过最大化正确选项的预测概率来优化模型。
• 模型推断： 使用Fine-tuned的模型对测试数据进行预测，从多个选项中选择最可能的下一句。

数据集下载和描述

在这里，我们使用到的也是论文中所提到的SWAG（The Situations With Adversarial Generations ）数据集，即给定一个情景（一个问题或一句描述），任务是模型从给定的四个选项中预测最有可能的一个。

如下所示便是部分原始示例数据：

1 ,video-id,fold-ind,startphrase,sent1,sent2,gold-source,ending0,ending1,ending2,ending3,label
2 0,anetv_NttjvRpSdsI,19391,The people are in robes. They,The people are in robes.,They,gold,are wearing colorful costumes.,are doing karate moves on the floor.,shake hands on their hips.,do a flip to the bag.,0
3 1,lsmdc3057_ROBIN_HOOD-27684,16344,She smirks at someone and rides off. He,She smirks at someone and rides off.,He,gold,smiles and falls heavily.,wears a bashful smile.,kneels down behind her.,gives him a playful glance.,1

如上所示数据集中一共有12个字段包含两个样本，我们这里需要用到的就是sent1,ending0,ending1,ending2,ending3,label这6个字段。例如对于第一个样本来说，其形式如下：

The people are in robes. They
  A) wearing colorful costumes.# 正确选项
  B) are doing karate moves on the floor.
  C) shake hands on their hips.  
  D) do a flip to the bag.

同时，由于该数据集已经做了训练集、验证集和测试集（没有标签）的划分，所以后续我们也就不需要来手动划分了。

数据集构建

import torch
from torch.utils.data import DataLoader
from tqdm import tqdm
import pandas as pd
import json
import logging
import os
from sklearn.model_selection import train_test_split
import collections
import six


class Vocab:
    """
    根据本地的vocab文件，构造一个词表
    vocab = Vocab()
    print(vocab.itos)  # 得到一个列表，返回词表中的每一个词；
    print(vocab.itos[2])  # 通过索引返回得到词表中对应的词；
    print(vocab.stoi)  # 得到一个字典，返回词表中每个词的索引；
    print(vocab.stoi['我'])  # 通过单词返回得到词表中对应的索引
    print(len(vocab))  # 返回词表长度
    """
    UNK = '[UNK]'

    def __init__(self, vocab_path):
        self.stoi = {}
        self.itos = []
        with open(vocab_path, 'r', encoding='utf-8') as f:
            for i, word in enumerate(f):
                w = word.strip('\n')
                self.stoi[w] = i
                self.itos.append(w)

    def __getitem__(self, token):
        return self.stoi.get(token, self.stoi.get(Vocab.UNK))

    def __len__(self):
        return len(self.itos)


def build_vocab(vocab_path):
    """
    vocab = Vocab()
    print(vocab.itos)  # 得到一个列表，返回词表中的每一个词；
    print(vocab.itos[2])  # 通过索引返回得到词表中对应的词；
    print(vocab.stoi)  # 得到一个字典，返回词表中每个词的索引；
    print(vocab.stoi['我'])  # 通过单词返回得到词表中对应的索引
    """
    return Vocab(vocab_path)


def pad_sequence(sequences, batch_first=False, max_len=None, padding_value=0):
    """
    对一个List中的元素进行padding
    Pad a list of variable length Tensors with ``padding_value``
    a = torch.ones(25)
    b = torch.ones(22)
    c = torch.ones(15)
    pad_sequence([a, b, c],max_len=None).size()
    torch.Size([25, 3])
        sequences:
        batch_first: 是否把batch_size放到第一个维度
        padding_value:
        max_len :
                当max_len = 50时，表示以某个固定长度对样本进行padding，多余的截掉；
                当max_len=None是，表示以当前batch中最长样本的长度对其它进行padding；
    Returns:
    """
    if max_len is None:
        max_len = max([s.size(0) for s in sequences])
    out_tensors = []
    for tensor in sequences:
        if tensor.size(0) < max_len:
            tensor = torch.cat([tensor, torch.tensor([padding_value] * (max_len - tensor.size(0)))], dim=0)
        else:
            tensor = tensor[:max_len]
        out_tensors.append(tensor)
    out_tensors = torch.stack(out_tensors, dim=1)
    if batch_first:
        return out_tensors.transpose(0, 1)
    return out_tensors


def cache(func):
    """
    本修饰器的作用是将SQuAD数据集中data_process()方法处理后的结果进行缓存，下次使用时可直接载入！
    :param func:
    :return:
    """

    def wrapper(*args, **kwargs):
        filepath = kwargs['filepath']
        postfix = kwargs['postfix']
        data_path = filepath.split('.')[0] + '_' + postfix + '.pt'
        if not os.path.exists(data_path):
            logging.info(f"缓存文件 {data_path} 不存在，重新处理并缓存！")
            data = func(*args, **kwargs)
            with open(data_path, 'wb') as f:
                torch.save(data, f)
        else:
            logging.info(f"缓存文件 {data_path} 存在，直接载入缓存文件！")
            with open(data_path, 'rb') as f:
                data = torch.load(f)
        return data

    return wrapper


class LoadSingleSentenceClassificationDataset:
    def __init__(self,
                 vocab_path='./vocab.txt',  #
                 tokenizer=None,
                 batch_size=32,
                 max_sen_len=None,
                 split_sep='\n',
                 max_position_embeddings=512,
                 pad_index=0,
                 is_sample_shuffle=True
                 ):

        """

        :param vocab_path: 本地词表vocab.txt的路径
        :param tokenizer:
        :param batch_size:
        :param max_sen_len: 在对每个batch进行处理时的配置；
                            当max_sen_len = None时，即以每个batch中最长样本长度为标准，对其它进行padding
                            当max_sen_len = 'same'时，以整个数据集中最长样本为标准，对其它进行padding
                            当max_sen_len = 50， 表示以某个固定长度符样本进行padding，多余的截掉；
        :param split_sep: 文本和标签之前的分隔符，默认为'\t'
        :param max_position_embeddings: 指定最大样本长度，超过这个长度的部分将本截取掉
        :param is_sample_shuffle: 是否打乱训练集样本（只针对训练集）
                在后续构造DataLoader时，验证集和测试集均指定为了固定顺序（即不进行打乱），修改程序时请勿进行打乱
                因为当shuffle为True时，每次通过for循环遍历data_iter时样本的顺序都不一样，这会导致在模型预测时
                返回的标签顺序与原始的顺序不一样，不方便处理。

        """
        self.tokenizer = tokenizer
        self.vocab = build_vocab(vocab_path)
        self.PAD_IDX = pad_index
        self.SEP_IDX = self.vocab['[SEP]']
        self.CLS_IDX = self.vocab['[CLS]']
        # self.UNK_IDX = '[UNK]'

        self.batch_size = batch_size
        self.split_sep = split_sep
        self.max_position_embeddings = max_position_embeddings
        if isinstance(max_sen_len, int) and max_sen_len > max_position_embeddings:
            max_sen_len = max_position_embeddings
        self.max_sen_len = max_sen_len
        self.is_sample_shuffle = is_sample_shuffle

    @cache
    def data_process(self, filepath, postfix='cache'):
        """
        将每一句话中的每一个词根据字典转换成索引的形式，同时返回所有样本中最长样本的长度
        :param filepath: 数据集路径
        :return:
        """
        raw_iter = open(filepath, encoding="utf8").readlines()
        data = []
        max_len = 0
        for raw in tqdm(raw_iter, ncols=80):
            line = raw.rstrip("\n").split(self.split_sep)
            s, l = line[0], line[1]
            tmp = [self.CLS_IDX] + [self.vocab[token] for token in self.tokenizer(s)]
            if len(tmp) > self.max_position_embeddings - 1:
                tmp = tmp[:self.max_position_embeddings - 1]  # BERT预训练模型只取前512个字符
            tmp += [self.SEP_IDX]
            tensor_ = torch.tensor(tmp, dtype=torch.long)
            l = torch.tensor(int(l), dtype=torch.long)
            max_len = max(max_len, tensor_.size(0))
            data.append((tensor_, l))
        return data, max_len

    def load_train_val_test_data(self, train_file_path=None,
                                 val_file_path=None,
                                 test_file_path=None,
                                 only_test=False):
        postfix = str(self.max_sen_len)
        test_data, _ = self.data_process(filepath=test_file_path, postfix=postfix)
        test_iter = DataLoader(test_data, batch_size=self.batch_size,
                               shuffle=False, collate_fn=self.generate_batch)
        if only_test:
            return test_iter
        train_data, max_sen_len = self.data_process(filepath=train_file_path,
                                                    postfix=postfix)  # 得到处理好的所有样本
        if self.max_sen_len == 'same':
            self.max_sen_len = max_sen_len
        val_data, _ = self.data_process(filepath=val_file_path,
                                        postfix=postfix)
        train_iter = DataLoader(train_data, batch_size=self.batch_size,  # 构造DataLoader
                                shuffle=self.is_sample_shuffle, collate_fn=self.generate_batch)
        val_iter = DataLoader(val_data, batch_size=self.batch_size,
                              shuffle=False, collate_fn=self.generate_batch)
        return train_iter, test_iter, val_iter

    def generate_batch(self, data_batch):
        batch_sentence, batch_label = [], []
        for (sen, label) in data_batch:  # 开始对一个batch中的每一个样本进行处理。
            batch_sentence.append(sen)
            batch_label.append(label)
        batch_sentence = pad_sequence(batch_sentence,  # [batch_size,max_len]
                                      padding_value=self.PAD_IDX,
                                      batch_first=False,
                                      max_len=self.max_sen_len)
        batch_label = torch.tensor(batch_label, dtype=torch.long)
        return batch_sentence, batch_label


class LoadMultipleChoiceDataset(LoadSingleSentenceClassificationDataset):
    def __init__(self, num_choice=4, **kwargs):
        super(LoadMultipleChoiceDataset, self).__init__(**kwargs)
        self.num_choice = num_choice

    @cache
    def data_process(self, filepath, postfix='cache'):
        data = pd.read_csv(filepath)
        questions = data['startphrase']
        answers0, answers1 = data['ending0'], data['ending1']
        answers2, answers3 = data['ending2'], data['ending3']
        labels = [-1] * len(questions)
        if 'label' in data:  # 测试集中没有标签
            labels = data['label']
        all_data = []
        max_len = 0
        for i in tqdm(range(len(questions)), ncols=80):
            # 将问题中的每个word转换为字典中的token id
            t_q = [self.vocab[token] for token in self.tokenizer(questions[i])]
            t_q = [self.CLS_IDX] + t_q + [self.SEP_IDX]
            # 将答案中的每个word转换为字典中的token id
            t_a0 = [self.vocab[token] for token in self.tokenizer(answers0[i])]
            t_a1 = [self.vocab[token] for token in self.tokenizer(answers1[i])]
            t_a2 = [self.vocab[token] for token in self.tokenizer(answers2[i])]
            t_a3 = [self.vocab[token] for token in self.tokenizer(answers3[i])]
            # 计算最长序列的长度
            max_len = max(max_len, len(t_q) + max(len(t_a0), len(t_a1), len(t_a2), len(t_a3)))
            seg_q = [0] * len(t_q)
            # 加1表示还要加上问题和答案组合后最后一个[SEP]的长度
            seg_a0 = [1] * (len(t_a0) + 1)
            seg_a1 = [1] * (len(t_a1) + 1)
            seg_a2 = [1] * (len(t_a2) + 1)
            seg_a3 = [1] * (len(t_a3) + 1)
            all_data.append((t_q, t_a0, t_a1, t_a2, t_a3, seg_q,
                             seg_a0, seg_a1, seg_a2, seg_a3, labels[i]))
        return all_data, max_len

    def generate_batch(self, data_batch):
        batch_qa, batch_seg, batch_label = [], [], []

        def get_seq(q, a):
            seq = q + a
            if len(seq) > self.max_position_embeddings - 1:
                seq = seq[:self.max_position_embeddings - 1]
            return torch.tensor(seq + [self.SEP_IDX], dtype=torch.long)

        for item in data_batch:
            # 得到 每个问题组合其中一个答案的 input_ids 序列
            tmp_qa = [get_seq(item[0], item[1]),
                      get_seq(item[0], item[2]),
                      get_seq(item[0], item[3]),
                      get_seq(item[0], item[4])]
            # 得到 每个问题组合其中一个答案的 token_type_ids
            seg0 = (item[5] + item[6])[:self.max_position_embeddings]
            seg1 = (item[5] + item[7])[:self.max_position_embeddings]
            seg2 = (item[5] + item[8])[:self.max_position_embeddings]
            seg3 = (item[5] + item[9])[:self.max_position_embeddings]
            tmp_seg = [torch.tensor(seg0, dtype=torch.long),
                       torch.tensor(seg1, dtype=torch.long),
                       torch.tensor(seg2, dtype=torch.long),
                       torch.tensor(seg3, dtype=torch.long)]
            batch_qa.extend(tmp_qa)
            batch_seg.extend(tmp_seg)
            batch_label.append(item[-1])

        batch_qa = pad_sequence(batch_qa,  # [batch_size*num_choice,max_len]
                                padding_value=self.PAD_IDX,
                                batch_first=True,
                                max_len=self.max_sen_len)
        batch_mask = (batch_qa == self.PAD_IDX).view(
            [-1, self.num_choice, batch_qa.size(-1)])
        # reshape 至 [batch_size, num_choice, max_len]
        batch_qa = batch_qa.view([-1, self.num_choice, batch_qa.size(-1)])
        batch_seg = pad_sequence(batch_seg,  # [batch_size*num_choice,max_len]
                                 padding_value=self.PAD_IDX,
                                 batch_first=True,
                                 max_len=self.max_sen_len)
        # reshape 至 [batch_size, num_choice, max_len]
        batch_seg = batch_seg.view([-1, self.num_choice, batch_seg.size(-1)])
        batch_label = torch.tensor(batch_label, dtype=torch.long)
        return batch_qa, batch_seg, batch_mask, batch_label

问答选择模型

我们只需要在原始BERT模型的基础上再加一个分类层即可，因此这部分代码相对来说也比较容易理解

定义一个类以及相应的初始化函数

from model.Bert import BertModel
import torch.nn as nn


class BertForMultipleChoice(nn.Module):
    """
    用于类似SWAG数据集的下游任务
    """

    def __init__(self, config, bert_pretrained_model_dir=None):
        super(BertForMultipleChoice, self).__init__()
        self.num_choice = config.num_labels
        if bert_pretrained_model_dir is not None:
            self.bert = BertModel.from_pretrained(config, bert_pretrained_model_dir)
        else:
            self.bert = BertModel(config)
        self.dropout = nn.Dropout(config.hidden_dropout_prob)
        self.classifier = nn.Linear(config.hidden_size, 1)

    def forward(self, input_ids,
                attention_mask=None,
                token_type_ids=None,
                position_ids=None,
                labels=None):
        """

        :param input_ids: [batch_size, num_choice, src_len]
        :param attention_mask: [batch_size, num_choice, src_len]
        :param token_type_ids: [batch_size, num_choice, src_len]
        :param position_ids:
        :param labels:
        :return:
        """
        flat_input_ids = input_ids.view(-1, input_ids.size(-1)).transpose(0, 1)
        flat_token_type_ids = token_type_ids.view(-1, token_type_ids.size(-1)).transpose(0, 1)
        flat_attention_mask = attention_mask.view(-1, token_type_ids.size(-1))

        pooled_output, _ = self.bert(
            input_ids=flat_input_ids,  # [src_len,batch_size*num_choice]
            attention_mask=flat_attention_mask,  # [batch_size*num_choice,src_len]
            token_type_ids=flat_token_type_ids,  # [src_len,batch_size*num_choice]
            position_ids=position_ids)
        pooled_output = self.dropout(pooled_output)  # [batch_size*num_choice, hidden_size]
        logits = self.classifier(pooled_output)  # [batch_size*num_choice, 1]
        shaped_logits = logits.view(-1, self.num_choice)  # [batch_size, num_choice]
        if labels is not None:
            loss_fct = nn.CrossEntropyLoss()
            loss = loss_fct(shaped_logits, labels.view(-1))
            return loss, shaped_logits
        else:
            return shaped_logits

定义一个ModelConfig类来对分类模型中的超参数以及其它变量进行管理，代码如下所示：

class ModelConfig:
    def __init__(self):
        self.project_dir = os.path.dirname(os.path.abspath(__file__))
        self.dataset_dir = os.path.join(self.project_dir, 'MultipleChoice')
        self.pretrained_model_dir = os.path.join(self.project_dir, "weight")
        self.vocab_path = os.path.join(self.pretrained_model_dir, 'vocab.txt')
        self.device = torch.device('xpu' if torch.cuda.is_available() else 'cpu')
        self.train_file_path = os.path.join(self.dataset_dir, 'train.csv')
        self.val_file_path = os.path.join(self.dataset_dir, 'val.csv')
        self.test_file_path = os.path.join(self.dataset_dir, 'test.csv')
        self.model_save_dir = os.path.join(self.project_dir, 'cache')
        self.logs_save_dir = os.path.join(self.project_dir, 'logs')
        self.is_sample_shuffle = True
        self.batch_size = 16
        self.max_sen_len = None
        self.num_labels = 4  # num_choice
        self.learning_rate = 2e-5
        self.epochs = 10
        self.model_val_per_epoch = 2
        logger_init(log_file_name='choice', log_level=logging.INFO,
                    log_dir=self.logs_save_dir)
        if not os.path.exists(self.model_save_dir):
            os.makedirs(self.model_save_dir)

        # 把原始bert中的配置参数也导入进来
        bert_config_path = os.path.join(self.pretrained_model_dir, "config.json")
        bert_config = BertConfig.from_json_file(bert_config_path)
        for key, value in bert_config.__dict__.items():
            self.__dict__[key] = value
        # 将当前配置打印到日志文件中
        logging.info(" ### 将当前配置打印到日志文件中 ")
        for key, value in self.__dict__.items():
            logging.info(f"### {key} = {value}")

训练

最后，我们便可以通过如下方法完成整个模型的微调：

def train(config):
    model = BertForMultipleChoice(config,
                                  config.pretrained_model_dir)
    model_save_path = os.path.join(config.model_save_dir, 'model.pt')
    if os.path.exists(model_save_path):
        loaded_paras = torch.load(model_save_path)
        model.load_state_dict(loaded_paras)
        logging.info("## 成功载入已有模型，进行追加训练......")
    model = model.to(config.device)
    optimizer = torch.optim.Adam(model.parameters(), lr=config.learning_rate)

    '''
    Apply Intel Extension for PyTorch optimization against the model object and optimizer object.
    '''
    model, optimizer = ipex.optimize(model, optimizer=optimizer)

    model.train()
    bert_tokenize = BertTokenizer.from_pretrained(model_config.pretrained_model_dir).tokenize
    data_loader = LoadMultipleChoiceDataset(
        vocab_path=config.vocab_path,
        tokenizer=bert_tokenize,
        batch_size=config.batch_size,
        max_sen_len=config.max_sen_len,
        max_position_embeddings=config.max_position_embeddings,
        pad_index=config.pad_token_id,
        is_sample_shuffle=config.is_sample_shuffle,
        num_choice=config.num_labels)
    train_iter, test_iter, val_iter = \
        data_loader.load_train_val_test_data(config.train_file_path,
                                             config.val_file_path,
                                             config.test_file_path)
    max_acc = 0
    for epoch in range(config.epochs):
        losses = 0
        start_time = time.time()
        for idx, (qa, seg, mask, label) in enumerate(train_iter):
            qa = qa.to(config.device)  # [src_len, batch_size]
            label = label.to(config.device)
            seg = seg.to(config.device)
            mask = mask.to(config.device)
            loss, logits = model(input_ids=qa,
                                 attention_mask=mask,
                                 token_type_ids=seg,
                                 position_ids=None,
                                 labels=label)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
            losses += loss.item()
            acc = (logits.argmax(1) == label).float().mean()
            if idx % 10 == 0:
                logging.info(f"Epoch: {epoch}, Batch[{idx}/{len(train_iter)}], "
                             f"Train loss :{loss.item():.3f}, Train acc: {acc:.3f}")
            if idx % 100 == 0:
                y_pred = logits.argmax(1).cpu()
                show_result(qa, y_pred, data_loader.vocab.itos, num_show=1)
        end_time = time.time()
        train_loss = losses / len(train_iter)
        logging.info(f"Epoch: {epoch}, Train loss: "
                     f"{train_loss:.3f}, Epoch time = {(end_time - start_time):.3f}s")
        if (epoch + 1) % config.model_val_per_epoch == 0:
            acc, _ = evaluate(val_iter, model,
                              config.device, inference=False)
            logging.info(f"Accuracy on val {acc:.3f}")
            if acc > max_acc:
                max_acc = acc
                torch.save(model.state_dict(), model_save_path)

结果

参考资料

基于BERT预训练模型的SWAG问答任务:https://mp.weixin.qq.com/s/GqsbMBNt9XcFIjmumR04Pg