【实作】CNN-微表情识别

 微表情识别

一、数据集说明

使用的数据集是FER2013

kaggle FER2013 https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge/data 

该数据集官方的下载链接目前失效了,可通过这个链接下载:https://www.kaggle.com/shawon10/facial-expression-detection-cnn/data?select=fer2013.csv

FER2013数据集下载下来是一个总 3w5k 行左右的csv文件:

【实作】CNN-微表情识别_第1张图片

 ......

【实作】CNN-微表情识别_第2张图片

......

【实作】CNN-微表情识别_第3张图片

第一列emotion是图像标签,即 y:[0, 6]。分别代表7种emotion:0 - ‘angry’(4953),1 - ‘disgusted’(547),2 - ‘feaful’(5121),3 - ‘happy’(8989),4 - ‘sad’(6077),5 - ‘surprised’(4002),6 - ‘neutral’(6198)  (数据集不是那么的平衡)

第二列是人脸图像的灰度像素值:[0, 255],大小是48*48

第三列是图像用途分类。根据第三列将图像分为 训练集(training),验证集(public test),测试集(private test)

 

二、预处理数据

读数据:

1 import pandas as pd
2 import numpy as np
3 
4 data_path = "../DATA/fer2013.csv"
5 df = pd.read_csv(data_path)
View Code

其中pixel列是一个字符串,将pixel列转为 int型 的list,表示像素灰度值

1 df['pixels'] = df['pixels'].str.split(' ').apply(lambda x: [int(i) for i in x])
View Code

在此步直接将pixel reshape成48*48的:

df['pixels'] = df['pixels'].str.split(' ').apply(lambda x: np.array([int(i) for i in x]).reshape(48,48))
View Code

注:后训练模型时发现不支持int64型数据,所以应该转为np.unit8,如下:

df['pixels'] = df['pixels'].str.split(' ').apply(lambda x: np.array([np.uint8(i) for i in x]).reshape(48,48))
View Code

 

将FER2013.csv根据用途(usage)分为train、val、test

分完后记得把原来的dataframe的index重设,否则val、test的dataframe序号不连续,后面y_val转成torch.LongTensor时会报错

 1 def devide_x_y(my_df):
 2     return my_df['pixels'].reset_index(drop=True), my_df['emotion'].reset_index(drop=True)
 3 
 4 # 分别筛选出training、publictest、privatetest对应的行
 5 training = df['Usage']=="Training"
 6 publicTest = df['Usage'] == "PublicTest"
 7 privateTest = df["Usage"] == "PrivateTest"
 8 
 9 # 读取对应行的数据
10 train = df[training]
11 public_t = df[publicTest]
12 private_t = df[privateTest]
13 
14 # 分为 X 和 y
15 train_x, train_y = devide_x_y(train)
16 val_x, val_y = devide_x_y(public_t)
17 test_x, test_y = devide_x_y(private_t)
View Code

分完后,各数据集的行数如下:

train:28709

val:3589

test:3589

len(train_x[0])每个图片长度为 2304 --> 48 * 48

画一张出来看一下

1 import matplotlib.pyplot as plt
2 
3 img_test = train_x[0].reshape(48,48)
4 plt.figure()
5 plt.imshow(img_test,cmap='gray')
6 plt.show()
View Code

显示如下:

【实作】CNN-微表情识别_第4张图片

 注:如果画图时没有注明是灰度图:cmap=gray,则会显示如下:

 【实作】CNN-微表情识别_第5张图片

可以看到,这个48*48的图的画质不太清晰,马赛克略有点严重

 

三、制作数据集

 1 import torchvision.transforms as transforms
 2 from torch.utils.data import Dataset, DataLoader
 3 import torch
 4 
 5 train_transform = transforms.Compose([
 6     transforms.ToPILImage(),
 7     transforms.RandomHorizontalFlip(),
 8     transforms.RandomRotation(5),
 9     transforms.ToTensor(),
10 ])
11 
12 test_transform = transforms.Compose([
13     transforms.ToPILImage(),
14     transforms.ToTensor(),
15 ])
16 
17 class MicroExpreDataset(Dataset):
18     def __init__(self, x, y=None, transform=None):
19         self.x = x
20         self.y = y
21         if y is not None:
22             # label needs to be a LongTensor
23             self.y = torch.LongTensor(y)
24         self.transform = transform
25     
26     def __len__(self):
27         return len(self.x)
28     
29     def __getitem__(self, index):
30         X = self.x[index]
31         if self.transform is not None:
32             X = self.transform(X)
33         if self.y is not None:
34             Y = self.y[index]
35             return X, Y
36         else: return X
37 
38 
39 batch_size = 128
40 train_set = MicroExpreDataset(train_x, train_y, train_transform)
41 val_set = MicroExpreDataset(val_x, val_y, test_transform)
42 
43 train_loader = DataLoader(train_set, batch_size=batch_size, shuffle=True)
44 val_loader = DataLoader(val_set, batch_size=batch_size, shuffle=False)
View Code

 

四、设置模型

 1 import torch.nn as nn
 2 
 3 class Classifier(nn.Module):
 4     def __init__(self):
 5         # super继承父类
 6         super(Classifier, self).__init__()
 7         
 8         # conv2d( in_channels, out_channels, kernel_size, stide, padding)
 9         # input shape: [1, 48, 48]
10         self.cnn = nn.Sequential(
11             nn.Conv2d(1, 64, 2, 1, 1), # [64, 49, 49]
12             nn.BatchNorm2d(64),
13             nn.ReLU(),
14             nn.MaxPool2d(2, 2, 0), #[64, 25, 25]
15             
16             nn.Conv2d(64, 128, 2, 1, 1), # [128, 26, 26]
17             nn.BatchNorm2d(128),
18             nn.ReLU(),
19             nn.MaxPool2d(2,2,0), # [128, 13, 13]
20             
21             nn.Conv2d(128, 256, 3, 1, 1), # [256, 13, 13]
22             nn.BatchNorm2d(256),
23             nn.ReLU(),
24             nn.MaxPool2d(2, 2, 0), # [256, 7, 7]
25             
26             nn.Conv2d(256, 256, 3, 1, 1), #[256, 7, 7]
27             nn.BatchNorm2d(256),
28             nn.ReLU(),
29             nn.MaxPool2d(2, 2, 0), #[256, 4, 4]        
30         )
31         
32         self.fc = nn.Sequential(
33             nn.Linear(2304, 512),
34             nn.ReLU(),
35             nn.Linear(512, 256),
36             nn.Linear(256, 7)
37         )
38         
39     def forward(self, x):
40         out = self.cnn(x)
41         out = out.view(out.size()[0], -1)
42         return self.fc(out)
View Code

这个模型注释里面,我算maxpool时算错了。我以为尽管maxpool的padding是0,但是当边界不足时,也会自动补0。但事实是好像并没有。。

注:

通过下面这个代码可以打印模型各层输入和输出

  1 def show_summary(my_model, my_input_size):
  2     from collections import OrderedDict
  3     import pandas as pd
  4     import numpy as np
  5 
  6     import torch
  7     from torch.autograd import Variable
  8     import torch.nn.functional as F
  9     from torch import nn
 10 
 11 
 12     def get_names_dict(model):
 13         """
 14         Recursive walk to get names including path
 15         """
 16         names = {}
 17         def _get_names(module, parent_name=''):
 18             for key, module in module.named_children():
 19                 name = parent_name + '.' + key if parent_name else key
 20                 names[name]=module
 21                 if isinstance(module, torch.nn.Module):
 22                     _get_names(module, parent_name=name)
 23         _get_names(model)
 24         return names
 25 
 26 
 27     def torch_summarize_df(input_size, model, weights=False, input_shape=True, nb_trainable=False):
 28         """
 29         Summarizes torch model by showing trainable parameters and weights.
 30         
 31         author: wassname
 32         url: https://gist.github.com/wassname/0fb8f95e4272e6bdd27bd7df386716b7
 33         license: MIT
 34         
 35         Modified from:
 36         - https://github.com/pytorch/pytorch/issues/2001#issuecomment-313735757
 37         - https://gist.github.com/wassname/0fb8f95e4272e6bdd27bd7df386716b7/
 38         
 39         Usage:
 40             import torchvision.models as models
 41             model = models.alexnet()
 42             df = torch_summarize_df(input_size=(3, 224,224), model=model)
 43             print(df)
 44             
 45             #              name class_name        input_shape       output_shape  nb_params
 46             # 1     features=>0     Conv2d  (-1, 3, 224, 224)   (-1, 64, 55, 55)      23296#(3*11*11+1)*64
 47             # 2     features=>1       ReLU   (-1, 64, 55, 55)   (-1, 64, 55, 55)          0
 48             # ...
 49         """
 50 
 51         def register_hook(module):
 52             def hook(module, input, output):
 53                 name = ''
 54                 for key, item in names.items():
 55                     if item == module:
 56                         name = key
 57                 #
 58                 class_name = str(module.__class__).split('.')[-1].split("'")[0]
 59                 module_idx = len(summary)
 60 
 61                 m_key = module_idx + 1
 62 
 63                 summary[m_key] = OrderedDict()
 64                 summary[m_key]['name'] = name
 65                 summary[m_key]['class_name'] = class_name
 66                 if input_shape:
 67                     summary[m_key][
 68                         'input_shape'] = (-1, ) + tuple(input[0].size())[1:]
 69                 summary[m_key]['output_shape'] = (-1, ) + tuple(output.size())[1:]
 70                 if weights:
 71                     summary[m_key]['weights'] = list(
 72                         [tuple(p.size()) for p in module.parameters()])
 73 
 74     #             summary[m_key]['trainable'] = any([p.requires_grad for p in module.parameters()])
 75                 if nb_trainable:
 76                     params_trainable = sum([torch.LongTensor(list(p.size())).prod() for p in module.parameters() if p.requires_grad])
 77                     summary[m_key]['nb_trainable'] = params_trainable
 78                 params = sum([torch.LongTensor(list(p.size())).prod() for p in module.parameters()])
 79                 summary[m_key]['nb_params'] = params
 80                 
 81 
 82             if  not isinstance(module, nn.Sequential) and \
 83                 not isinstance(module, nn.ModuleList) and \
 84                 not (module == model):
 85                 hooks.append(module.register_forward_hook(hook))
 86                 
 87         # Names are stored in parent and path+name is unique not the name
 88         names = get_names_dict(model)
 89 
 90         # check if there are multiple inputs to the network
 91         if isinstance(input_size[0], (list, tuple)):
 92             x = [Variable(torch.rand(1, *in_size)) for in_size in input_size]
 93         else:
 94             x = Variable(torch.rand(1, *input_size))
 95 
 96         if next(model.parameters()).is_cuda:
 97             x = x.cuda()
 98 
 99         # create properties
100         summary = OrderedDict()
101         hooks = []
102 
103         # register hook
104         model.apply(register_hook)
105 
106         # make a forward pass
107         model(x)
108 
109         # remove these hooks
110         for h in hooks:
111             h.remove()
112 
113         # make dataframe
114         df_summary = pd.DataFrame.from_dict(summary, orient='index')
115 
116         return df_summary
117 
118 
119     # Test on alexnet
120     import torchvision.models as models
121     df = torch_summarize_df(input_size=my_input_size, model=my_model)
122     print(df)
View Code

实际的模型输入输出应该是这样

      name   class_name        input_shape       output_shape        nb_params
1    cnn.0       Conv2d    (-1, 1, 48, 48)   (-1, 64, 49, 49)      tensor(320)
2    cnn.1  BatchNorm2d   (-1, 64, 49, 49)   (-1, 64, 49, 49)      tensor(128)
3    cnn.2         ReLU   (-1, 64, 49, 49)   (-1, 64, 49, 49)                0
4    cnn.3    MaxPool2d   (-1, 64, 49, 49)   (-1, 64, 24, 24)                0
5    cnn.4       Conv2d   (-1, 64, 24, 24)  (-1, 128, 25, 25)    tensor(32896)
6    cnn.5  BatchNorm2d  (-1, 128, 25, 25)  (-1, 128, 25, 25)      tensor(256)
7    cnn.6         ReLU  (-1, 128, 25, 25)  (-1, 128, 25, 25)                0
8    cnn.7    MaxPool2d  (-1, 128, 25, 25)  (-1, 128, 12, 12)                0
9    cnn.8       Conv2d  (-1, 128, 12, 12)  (-1, 256, 12, 12)   tensor(295168)
10   cnn.9  BatchNorm2d  (-1, 256, 12, 12)  (-1, 256, 12, 12)      tensor(512)
11  cnn.10         ReLU  (-1, 256, 12, 12)  (-1, 256, 12, 12)                0
12  cnn.11    MaxPool2d  (-1, 256, 12, 12)    (-1, 256, 6, 6)                0
13  cnn.12       Conv2d    (-1, 256, 6, 6)    (-1, 256, 6, 6)   tensor(590080)
14  cnn.13  BatchNorm2d    (-1, 256, 6, 6)    (-1, 256, 6, 6)      tensor(512)
15  cnn.14         ReLU    (-1, 256, 6, 6)    (-1, 256, 6, 6)                0
16  cnn.15    MaxPool2d    (-1, 256, 6, 6)    (-1, 256, 3, 3)                0
17    fc.0       Linear         (-1, 2304)          (-1, 512)  tensor(1180160)
18    fc.1      Dropout          (-1, 512)          (-1, 512)                0
19    fc.2         ReLU          (-1, 512)          (-1, 512)                0
20    fc.3       Linear          (-1, 512)          (-1, 256)   tensor(131328)
21    fc.4      Dropout          (-1, 256)          (-1, 256)                0
22    fc.5       Linear          (-1, 256)            (-1, 7)     tensor(1799)
model parameters

 

五、训练

因为工作站有两张卡,仅一张可用,需打印看一下可用的那张的编号

torch.cuda.get_device_name(0)

训练30个epoch

 1 import os
 2 import time
 3 
 4 os.environ["CUDA_VISIBLE_DEVICES"] = '0'
 5 model = Classifier().cuda()
 6 
 7 # classification task, so use crossEntropyLoss
 8 loss = nn.CrossEntropyLoss()
 9 # optimizer use Adam
10 optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
11 num_epoch = 30
12 
13 for epoch in range(num_epoch):
14     epoch_start_time = time.time()
15     train_acc = 0.0
16     train_loss = 0.0
17     val_acc = 0.0
18     val_loss = 0.0
19     
20     # open train mode
21     model.train()
22     for i, data in enumerate(train_loader):
23         # set gradient to zero, otherwise it will be added each time
24         optimizer.zero_grad()
25         
26         # train model to get the predict possibility. Actually, it is to call the forward function
27         train_pred = model(data[0].cuda()) # data[0] is x
28         batch_loss = loss(train_pred, data[1].cuda()) # data[1] is y
29         
30         # use backward to calculate the gradient for each parameter
31         batch_loss.backward()
32         # update the parameters by optimizer
33         optimizer.step()
34         
35         train_acc += np.sum(np.argmax(train_pred.cpu().data.numpy(), axis=1) == data[1].numpy())
36         train_loss += batch_loss.item()
37         
38     # turn model to eval mode
39     model.eval()
40     with torch.no_grad():
41         for i, data in enumerate(val_loader):
42             val_pred = model(data[0].cuda())
43             batch_loss = loss(val_pred, data[1].cuda())
44             
45             val_acc += np.sum(np.argmax(val_pred.cpu().data.numpy(), axis=1) == data[1].numpy())
46             val_loss += batch_loss.item()
47         
48         # print out the results
49         print('[%03d/%03d] %2.2f sec(s) Train Acc: %3.6f Loss: %3.6f | Val Acc: %3.6f loss: %3.6f' % \
50             (epoch + 1, num_epoch, time.time()-epoch_start_time, \
51              train_acc/train_set.__len__(), train_loss/train_set.__len__(), val_acc/val_set.__len__(), val_loss/val_set.__len__()))
52         
View Code

训练结果val acc大概 0.6左右

[001/030] 10.69 sec(s) Train Acc: 0.309450 Loss: 0.013407 | Val Acc: 0.404848 loss: 0.012436
[002/030] 11.58 sec(s) Train Acc: 0.467449 Loss: 0.010781 | Val Acc: 0.494845 loss: 0.010486
[003/030] 10.64 sec(s) Train Acc: 0.522728 Loss: 0.009736 | Val Acc: 0.533018 loss: 0.009739
[004/030] 10.57 sec(s) Train Acc: 0.552440 Loss: 0.009222 | Val Acc: 0.550850 loss: 0.009390
[005/030] 10.58 sec(s) Train Acc: 0.579087 Loss: 0.008683 | Val Acc: 0.564224 loss: 0.009504
[006/030] 10.59 sec(s) Train Acc: 0.598558 Loss: 0.008282 | Val Acc: 0.566732 loss: 0.009202
[007/030] 10.62 sec(s) Train Acc: 0.613710 Loss: 0.007971 | Val Acc: 0.580106 loss: 0.009015
[008/030] 10.60 sec(s) Train Acc: 0.636421 Loss: 0.007589 | Val Acc: 0.591808 loss: 0.009277
[009/030] 10.61 sec(s) Train Acc: 0.647950 Loss: 0.007321 | Val Acc: 0.580663 loss: 0.009388
[010/030] 10.64 sec(s) Train Acc: 0.664635 Loss: 0.007020 | Val Acc: 0.589301 loss: 0.008910
[011/030] 10.93 sec(s) Train Acc: 0.680449 Loss: 0.006731 | Val Acc: 0.600446 loss: 0.009198
[012/030] 10.48 sec(s) Train Acc: 0.695531 Loss: 0.006364 | Val Acc: 0.610198 loss: 0.009257
[013/030] 10.46 sec(s) Train Acc: 0.711972 Loss: 0.006073 | Val Acc: 0.610198 loss: 0.008954
[014/030] 11.79 sec(s) Train Acc: 0.726636 Loss: 0.005780 | Val Acc: 0.609641 loss: 0.008695
[015/030] 10.62 sec(s) Train Acc: 0.741579 Loss: 0.005495 | Val Acc: 0.604068 loss: 0.009682
[016/030] 10.55 sec(s) Train Acc: 0.760633 Loss: 0.005133 | Val Acc: 0.596266 loss: 0.009784
[017/030] 10.53 sec(s) Train Acc: 0.771256 Loss: 0.004891 | Val Acc: 0.613820 loss: 0.009530
[018/030] 10.51 sec(s) Train Acc: 0.785956 Loss: 0.004585 | Val Acc: 0.609083 loss: 0.009737
[019/030] 10.49 sec(s) Train Acc: 0.798460 Loss: 0.004316 | Val Acc: 0.607412 loss: 0.010229
[020/030] 10.49 sec(s) Train Acc: 0.812150 Loss: 0.003993 | Val Acc: 0.605461 loss: 0.010734
[021/030] 10.47 sec(s) Train Acc: 0.822913 Loss: 0.003782 | Val Acc: 0.576484 loss: 0.011127
[022/030] 10.53 sec(s) Train Acc: 0.836184 Loss: 0.003529 | Val Acc: 0.594595 loss: 0.012074
[023/030] 10.55 sec(s) Train Acc: 0.846459 Loss: 0.003334 | Val Acc: 0.604904 loss: 0.012387
[024/030] 10.48 sec(s) Train Acc: 0.857780 Loss: 0.003108 | Val Acc: 0.612427 loss: 0.011957
[025/030] 10.61 sec(s) Train Acc: 0.866070 Loss: 0.002917 | Val Acc: 0.598495 loss: 0.012693
[026/030] 10.63 sec(s) Train Acc: 0.869100 Loss: 0.002839 | Val Acc: 0.599053 loss: 0.013177
[027/030] 10.60 sec(s) Train Acc: 0.883800 Loss: 0.002563 | Val Acc: 0.608247 loss: 0.012588
[028/030] 10.58 sec(s) Train Acc: 0.889094 Loss: 0.002421 | Val Acc: 0.609083 loss: 0.012604
[029/030] 10.57 sec(s) Train Acc: 0.889338 Loss: 0.002420 | Val Acc: 0.606297 loss: 0.014425
[030/030] 10.58 sec(s) Train Acc: 0.898743 Loss: 0.002211 | Val Acc: 0.600446 loss: 0.013294
result

 

六、调优模型

1) 在FC层增加dropout

1 self.fc = nn.Sequential(
2             nn.Linear(2304, 512),
3             nn.Dropout(0.3),
4             nn.ReLU(),
5             nn.Linear(512, 256),
6             nn.Dropout(0.3),
7             nn.Linear(256, 7)
8         )
View Code

效果没有显著提升,val acc大概还是0.6左右

[001/030] 11.63 sec(s) Train Acc: 0.307534 Loss: 0.013370 | Val Acc: 0.412928 loss: 0.012034
[002/030] 10.04 sec(s) Train Acc: 0.445191 Loss: 0.011217 | Val Acc: 0.495403 loss: 0.010660
[003/030] 11.71 sec(s) Train Acc: 0.490961 Loss: 0.010363 | Val Acc: 0.514349 loss: 0.010133
[004/030] 10.32 sec(s) Train Acc: 0.509805 Loss: 0.009945 | Val Acc: 0.544999 loss: 0.009528
[005/030] 11.72 sec(s) Train Acc: 0.536452 Loss: 0.009538 | Val Acc: 0.553636 loss: 0.009448
[006/030] 10.73 sec(s) Train Acc: 0.552823 Loss: 0.009249 | Val Acc: 0.553079 loss: 0.009476
[007/030] 11.18 sec(s) Train Acc: 0.563134 Loss: 0.009051 | Val Acc: 0.551407 loss: 0.009214
[008/030] 10.37 sec(s) Train Acc: 0.575882 Loss: 0.008729 | Val Acc: 0.559487 loss: 0.009178
[009/030] 10.68 sec(s) Train Acc: 0.592149 Loss: 0.008511 | Val Acc: 0.585678 loss: 0.008921
[010/030] 10.49 sec(s) Train Acc: 0.600683 Loss: 0.008264 | Val Acc: 0.570354 loss: 0.009281
[011/030] 10.57 sec(s) Train Acc: 0.611725 Loss: 0.008074 | Val Acc: 0.586236 loss: 0.008738
[012/030] 10.60 sec(s) Train Acc: 0.625727 Loss: 0.007824 | Val Acc: 0.597660 loss: 0.008503
[013/030] 10.58 sec(s) Train Acc: 0.635306 Loss: 0.007631 | Val Acc: 0.583171 loss: 0.008948
[014/030] 10.60 sec(s) Train Acc: 0.644606 Loss: 0.007423 | Val Acc: 0.583728 loss: 0.008670
[015/030] 11.75 sec(s) Train Acc: 0.655300 Loss: 0.007204 | Val Acc: 0.583449 loss: 0.009537
[016/030] 11.72 sec(s) Train Acc: 0.668536 Loss: 0.006962 | Val Acc: 0.613820 loss: 0.008576
[017/030] 11.78 sec(s) Train Acc: 0.677383 Loss: 0.006736 | Val Acc: 0.582056 loss: 0.010016
[018/030] 10.39 sec(s) Train Acc: 0.686753 Loss: 0.006568 | Val Acc: 0.604068 loss: 0.009147
[019/030] 10.35 sec(s) Train Acc: 0.696402 Loss: 0.006376 | Val Acc: 0.586514 loss: 0.009409
[020/030] 10.37 sec(s) Train Acc: 0.701870 Loss: 0.006202 | Val Acc: 0.602675 loss: 0.009220
[021/030] 10.39 sec(s) Train Acc: 0.714758 Loss: 0.005984 | Val Acc: 0.595152 loss: 0.009192
[022/030] 10.44 sec(s) Train Acc: 0.722178 Loss: 0.005780 | Val Acc: 0.601282 loss: 0.009065
[023/030] 10.65 sec(s) Train Acc: 0.735623 Loss: 0.005579 | Val Acc: 0.621064 loss: 0.009224
[024/030] 10.46 sec(s) Train Acc: 0.743809 Loss: 0.005383 | Val Acc: 0.618278 loss: 0.010290
[025/030] 10.44 sec(s) Train Acc: 0.752447 Loss: 0.005239 | Val Acc: 0.616606 loss: 0.010834
[026/030] 10.32 sec(s) Train Acc: 0.758403 Loss: 0.005108 | Val Acc: 0.589858 loss: 0.009424
[027/030] 10.32 sec(s) Train Acc: 0.766728 Loss: 0.004936 | Val Acc: 0.608526 loss: 0.010024
[028/030] 10.33 sec(s) Train Acc: 0.771814 Loss: 0.004792 | Val Acc: 0.609083 loss: 0.010309
[029/030] 10.29 sec(s) Train Acc: 0.780487 Loss: 0.004629 | Val Acc: 0.600167 loss: 0.011290
[030/030] 10.26 sec(s) Train Acc: 0.794106 Loss: 0.004395 | Val Acc: 0.606854 loss: 0.012611
result

2)在第一层的maxpool中stride设成1,并周围补一圈0 --> 不缩小图像

因为48*48这个图还是比较马赛克化的

这样操作后,模型从最后一层conv后输出3*3变成了6*6,模型具体结构如下:

      name   class_name        input_shape       output_shape        nb_params
1    cnn.0       Conv2d    (-1, 1, 48, 48)   (-1, 64, 49, 49)      tensor(320)
2    cnn.1  BatchNorm2d   (-1, 64, 49, 49)   (-1, 64, 49, 49)      tensor(128)
3    cnn.2         ReLU   (-1, 64, 49, 49)   (-1, 64, 49, 49)                0
4    cnn.3    MaxPool2d   (-1, 64, 49, 49)   (-1, 64, 50, 50)                0
5    cnn.4       Conv2d   (-1, 64, 50, 50)  (-1, 128, 51, 51)    tensor(32896)
6    cnn.5  BatchNorm2d  (-1, 128, 51, 51)  (-1, 128, 51, 51)      tensor(256)
7    cnn.6         ReLU  (-1, 128, 51, 51)  (-1, 128, 51, 51)                0
8    cnn.7    MaxPool2d  (-1, 128, 51, 51)  (-1, 128, 25, 25)                0
9    cnn.8       Conv2d  (-1, 128, 25, 25)  (-1, 256, 25, 25)   tensor(295168)
10   cnn.9  BatchNorm2d  (-1, 256, 25, 25)  (-1, 256, 25, 25)      tensor(512)
11  cnn.10         ReLU  (-1, 256, 25, 25)  (-1, 256, 25, 25)                0
12  cnn.11    MaxPool2d  (-1, 256, 25, 25)  (-1, 256, 12, 12)                0
13  cnn.12       Conv2d  (-1, 256, 12, 12)  (-1, 256, 12, 12)   tensor(590080)
14  cnn.13  BatchNorm2d  (-1, 256, 12, 12)  (-1, 256, 12, 12)      tensor(512)
15  cnn.14         ReLU  (-1, 256, 12, 12)  (-1, 256, 12, 12)                0
16  cnn.15    MaxPool2d  (-1, 256, 12, 12)    (-1, 256, 6, 6)                0
17    fc.0       Linear         (-1, 9216)          (-1, 512)  tensor(4719104)
18    fc.1      Dropout          (-1, 512)          (-1, 512)                0
19    fc.2         ReLU          (-1, 512)          (-1, 512)                0
20    fc.3       Linear          (-1, 512)          (-1, 256)   tensor(131328)
21    fc.4      Dropout          (-1, 256)          (-1, 256)                0
22    fc.5       Linear          (-1, 256)            (-1, 7)     tensor(1799)
model parameters

但训练结果没有提升,反而看起来略有点下降了

[001/030] 15.70 sec(s) Train Acc: 0.254276 Loss: 0.014978 | Val Acc: 0.329061 loss: 0.013547
[002/030] 14.21 sec(s) Train Acc: 0.359678 Loss: 0.012589 | Val Acc: 0.418222 loss: 0.011816
[003/030] 14.17 sec(s) Train Acc: 0.417883 Loss: 0.011663 | Val Acc: 0.472276 loss: 0.011007
[004/030] 14.30 sec(s) Train Acc: 0.451844 Loss: 0.011097 | Val Acc: 0.479521 loss: 0.010857
[005/030] 14.30 sec(s) Train Acc: 0.461388 Loss: 0.010875 | Val Acc: 0.509334 loss: 0.010366
[006/030] 14.32 sec(s) Train Acc: 0.484726 Loss: 0.010492 | Val Acc: 0.522708 loss: 0.010089
[007/030] 14.32 sec(s) Train Acc: 0.500644 Loss: 0.010178 | Val Acc: 0.533296 loss: 0.009900
[008/030] 14.31 sec(s) Train Acc: 0.509979 Loss: 0.010009 | Val Acc: 0.547785 loss: 0.009626
[009/030] 14.32 sec(s) Train Acc: 0.522937 Loss: 0.009812 | Val Acc: 0.543327 loss: 0.009497
[010/030] 14.32 sec(s) Train Acc: 0.529416 Loss: 0.009635 | Val Acc: 0.565060 loss: 0.009770
[011/030] 14.31 sec(s) Train Acc: 0.537810 Loss: 0.009511 | Val Acc: 0.561995 loss: 0.009458
[012/030] 14.33 sec(s) Train Acc: 0.544115 Loss: 0.009311 | Val Acc: 0.538033 loss: 0.009728
[013/030] 14.32 sec(s) Train Acc: 0.552928 Loss: 0.009205 | Val Acc: 0.560323 loss: 0.009435
[014/030] 14.32 sec(s) Train Acc: 0.561218 Loss: 0.009048 | Val Acc: 0.573976 loss: 0.009330
[015/030] 14.32 sec(s) Train Acc: 0.568254 Loss: 0.008903 | Val Acc: 0.573976 loss: 0.009089
[016/030] 14.35 sec(s) Train Acc: 0.575743 Loss: 0.008777 | Val Acc: 0.571468 loss: 0.009189
[017/030] 14.34 sec(s) Train Acc: 0.581838 Loss: 0.008667 | Val Acc: 0.567846 loss: 0.009613
[018/030] 14.32 sec(s) Train Acc: 0.589954 Loss: 0.008496 | Val Acc: 0.578991 loss: 0.009191
[019/030] 14.32 sec(s) Train Acc: 0.593124 Loss: 0.008375 | Val Acc: 0.590415 loss: 0.009168
[020/030] 14.32 sec(s) Train Acc: 0.603191 Loss: 0.008238 | Val Acc: 0.598217 loss: 0.009239
[021/030] 14.32 sec(s) Train Acc: 0.609704 Loss: 0.008045 | Val Acc: 0.598774 loss: 0.008932
[022/030] 14.32 sec(s) Train Acc: 0.613327 Loss: 0.008028 | Val Acc: 0.598774 loss: 0.008821
[023/030] 14.32 sec(s) Train Acc: 0.623881 Loss: 0.007821 | Val Acc: 0.598495 loss: 0.008917
[024/030] 14.32 sec(s) Train Acc: 0.631056 Loss: 0.007649 | Val Acc: 0.599053 loss: 0.008865
[025/030] 14.34 sec(s) Train Acc: 0.637082 Loss: 0.007531 | Val Acc: 0.597660 loss: 0.008953
[026/030] 14.32 sec(s) Train Acc: 0.639834 Loss: 0.007439 | Val Acc: 0.606576 loss: 0.009263
[027/030] 14.31 sec(s) Train Acc: 0.652200 Loss: 0.007253 | Val Acc: 0.597381 loss: 0.008817
[028/030] 14.30 sec(s) Train Acc: 0.657076 Loss: 0.007143 | Val Acc: 0.586236 loss: 0.009791
[029/030] 14.31 sec(s) Train Acc: 0.662301 Loss: 0.007033 | Val Acc: 0.612984 loss: 0.009230
[030/030] 14.30 sec(s) Train Acc: 0.672577 Loss: 0.006849 | Val Acc: 0.598495 loss: 0.009119
result

 

七、训练+验证一起重新训练一遍最优模型

这里还是用最开始的model

合并train,test

1 train_val_x = np.concatenate((train_x, val_x), axis=0)
2 train_val_y = np.concatenate((train_y, val_y), axis=0)
3 train_val_set = MicroExpreDataset(train_val_x, train_val_y, train_transform)
4 train_val_loader = DataLoader(train_val_set, batch_size=batch_size, shuffle=True)
View Code

用best_model进行训练

 1 model_best = Classifier().cuda()
 2 loss = nn.CrossEntropyLoss()
 3 optimizer = torch.optim.Adam(model_best.parameters(), lr=0.001)
 4 num_epoch = 30
 5 
 6 for epoch in range(num_epoch):
 7     epoch_start_time = time.time()
 8     train_acc = 0.0
 9     train_loss = 0.0
10     
11     model_best.train()
12     for i, data in enumerate(train_val_loader):
13         optimizer.zero_grad()
14         train_pred = model_best(data[0].cuda())
15         batch_loss = loss(train_pred, data[1].cuda())
16         batch_loss.backward()
17         optimizer.step()
18         
19         train_acc += np.sum(np.argmax(train_pred.cpu().data.numpy(), axis=1) == data[1].numpy())
20         train_loss+= batch_loss.item()
21         
22         # print out the result
23     print('[%03d/%03d] %2.2f sec(s) Train Acc: %3.6f Loss: %3.6f' % \
24       (epoch + 1, num_epoch, time.time()-epoch_start_time, \
25       train_acc/train_val_set.__len__(), train_loss/train_val_set.__len__()))
View Code

训练结果大概89%左右,但应该是有一点过拟合了

 1 [001/030] 11.02 sec(s) Train Acc: 0.349867 Loss: 0.012808
 2 [002/030] 10.87 sec(s) Train Acc: 0.494024 Loss: 0.010299
 3 [003/030] 10.91 sec(s) Train Acc: 0.540529 Loss: 0.009427
 4 [004/030] 10.97 sec(s) Train Acc: 0.573101 Loss: 0.008838
 5 [005/030] 10.97 sec(s) Train Acc: 0.594774 Loss: 0.008422
 6 [006/030] 10.97 sec(s) Train Acc: 0.610224 Loss: 0.008127
 7 [007/030] 11.07 sec(s) Train Acc: 0.625735 Loss: 0.007769
 8 [008/030] 11.06 sec(s) Train Acc: 0.643476 Loss: 0.007467
 9 [009/030] 10.97 sec(s) Train Acc: 0.656604 Loss: 0.007177
10 [010/030] 10.99 sec(s) Train Acc: 0.675367 Loss: 0.006820
11 [011/030] 10.97 sec(s) Train Acc: 0.688928 Loss: 0.006606
12 [012/030] 10.96 sec(s) Train Acc: 0.708279 Loss: 0.006197
13 [013/030] 10.91 sec(s) Train Acc: 0.719549 Loss: 0.005955
14 [014/030] 10.98 sec(s) Train Acc: 0.726144 Loss: 0.005742
15 [015/030] 10.97 sec(s) Train Acc: 0.748746 Loss: 0.005355
16 [016/030] 10.96 sec(s) Train Acc: 0.768654 Loss: 0.004975
17 [017/030] 10.97 sec(s) Train Acc: 0.775714 Loss: 0.004779
18 [018/030] 10.99 sec(s) Train Acc: 0.795281 Loss: 0.004470
19 [019/030] 10.97 sec(s) Train Acc: 0.804694 Loss: 0.004222
20 [020/030] 10.98 sec(s) Train Acc: 0.818286 Loss: 0.003909
21 [021/030] 10.98 sec(s) Train Acc: 0.828720 Loss: 0.003722
22 [022/030] 10.97 sec(s) Train Acc: 0.837204 Loss: 0.003552
23 [023/030] 10.97 sec(s) Train Acc: 0.851167 Loss: 0.003269
24 [024/030] 10.98 sec(s) Train Acc: 0.857360 Loss: 0.003158
25 [025/030] 10.96 sec(s) Train Acc: 0.863057 Loss: 0.002980
26 [026/030] 10.99 sec(s) Train Acc: 0.874915 Loss: 0.002788
27 [027/030] 10.98 sec(s) Train Acc: 0.878166 Loss: 0.002646
28 [028/030] 10.98 sec(s) Train Acc: 0.886835 Loss: 0.002485
29 [029/030] 10.98 sec(s) Train Acc: 0.893771 Loss: 0.002362
30 [030/030] 10.98 sec(s) Train Acc: 0.896278 Loss: 0.002323
View Code

保存模型

torch.save(model_best, 'model_best.pkl')

 

八、测试

用测试集再算一遍看下acc到达多少

读取模型

# load model
test_model = torch.load('model_best.pkl')
 1 test_set = MicroExpreDataset(test_x, test_y, test_transform)
 2 test_loader = DataLoader(test_set, batch_size=batch_size, shuffle=False)
 3 
 4 test_model.eval()
 5 prediction = []
 6 with torch.no_grad():
 7     for i, data in enumerate(test_loader):
 8         test_pred = test_model(data[0].cuda())
 9         test_label = np.argmax(test_pred.cpu().data.numpy(), axis=1)
10         for y in test_label:
11             prediction.append(y)
12 
13 test_acc = np.sum(prediction == test_y) / test_set.__len__()
14 test_acc
15 
16 # test_acc= 0.62245
View Code

最后test acc = 0.622

合并了训练集和验证集后效果略提升了一些。

 

九、调用摄像头,在线demo

步骤:

1)调用opencv自带的人脸识别器

路径(mac参考路径) /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/sitepackages/cv2/data/

【实作】CNN-微表情识别_第6张图片

 

一般使用:

【实作】CNN-微表情识别_第7张图片

 

 在这里我们用alt2来实现:

 

 

 

2)调用系统摄像头拍摄出人脸图片

3)对人脸图片进行预处理

4)将处理完的图片传入模型

5)将模型计算出来的结果反馈至运行窗口界面

 

 

 

 

 

 ---------------ref-----------------

(2)网上的考察资料中对于调用 cascade_classifier.detectMultiScale这个函数时,对其一个参数scaleFactor设置为1.3,这直接导致对于人脸的识别不够敏感,上网参阅相关资料后,得知这个值一般设置为1.1较为合适,且对于人脸的识别也比较敏感

(3)因为对于opencv这个库使用不够熟练,一开始一直碰壁,所以你要是和我一样,建议先去这个网站了解一下OPENCV的大致使用方法:https://docs.opencv.org/trunk/d6/d00/tutorial_py_root.html

4.该模型的不足以及日后的发展

(1)目前划分的微表情类别较少,难以覆盖人类多种交织的表情。

(2)对于某人表情识别的准备率还不够高,例如厌恶。

(3)模型未能处理多帧连续表情的变化,这将限制我们对少数微表情的识别的准确率。

5.完善算法以及平台的应用

(1)采用LSTM(长短时记忆算法),新增加一个时间维度用于记录多帧微表情的变化,然后在结果DNN模型完善算法。

(2)可以将此项目应用于检测疲劳驾驶以及刑侦等领域

 

 

 

 

参考:

https://www.cnblogs.com/XDU-Lakers/p/10587894.html

https://blog.csdn.net/mathlxj/article/details/87920084

https://github.com/tgpcai/Microexpression_recognition

你可能感兴趣的:(【实作】CNN-微表情识别)