天池CIKM2018 - Sentence Similarity

 

Github Link: https://github.com/MarvinLSJ/LSTM-siamese

cikm 2018 - Sentence Similarity

AliMe is a chatbot for online shopping in a global context, this task is solving the short text matching problem with different language (Spanish & English)

Website:https://tianchi.aliyun.com/competition/introduction.htm?spm=5176.100150.711.6.600e2784M5Z0uW&raceId=231661

Competition Introduction

Data Description

Training Data

21400 Labeled Spanish sentence pairs & English sentence pairs are provided;

55669 Unlabeled Spanish sentences & corresponding English translations are provided.

Test Data

5000 Spanish sentence pairs

Goal and Evaluation

Predicting the similarity of Spanish sentence pairs in test set.

Evaluated result by logloss.

ML Model

Developed by freedomwyl in Link

Deep Model

Common thoughts would be finding a way to represent sentences and calculate their similarity, with a little elaboration, here comes the basic model.

Basic Model: LSTM-Siamese

Name Origin

The name comes from Siamese twins in Thailand, the conjoined twins whose body is partially shared with each other. Later the word "Siamese" refers to the phenomenon of twin structures, like this neural network.

Main Idea

This model takes in one sentence pair, encoding each sentence into vector representation through LSTM word by word (which gives the sentence embedding the information of word sequences). Then generate some vector features from them, feed into classifier to get the similarity.

Baseline

With standard parameter settings as follows, the validation loss can be 0.3366, which is a pretty well off-line score.

Baseline configuration

experiment_name: 'siamese-baseline'

task: 'train'make_dict: Falsedata_preprocessing: False

ckpt_dir: 'ckpt/'

training: num_epochs: 20 learning_rate: 0.01 # options = ['adam', 'adadelta', 'rmsprop'] optimizer: 'sgd'

embedding: full_embedding_path: 'input/wiki.es.vec' cur_embedding_path: 'input/embedding.pkl'

model: fc_dim: 100 name: 'siamese' embed_size: 300 batch_size: 1 embedding_freeze: False encoder: hidden_size: 150 num_layers: 1 bidirectional: False dropout: 0.5

result: filename: 'result.txt' filepath: 'res/'

 

Some Attempts

Tuning Parameters

1. Classifier

   fc_dim: classifier fully connected layer size

2. Encoder

   hidden_size: lstm hidden size

   num_layers: lstm layer

   bidirectional: bidirectional lstm can get more info

   dropout: avoid overfitting

3. Embedding

   embedding_freeze: Set it to false, then the embedding will participate backpropogation. Not so good from my experience, especially small training dataset.

 

Structure

1. Classifier

   fc layers, non-linear fc layers(add ReLU)

2. Encoder

   Features generating method, current method is (v1, v2, abs(v1-v2), v1*v2), more features with different vector distance measurement?

    

Training

1. Early stopping

   Stop training whenever the |valid loss - train loss| <= 0.02

2. Optimizer

   Default SGD;

   Rmsprop for self-adaptive learning rate;

   Adam for self-adaptive learning rate and momentum to get out of local optima;

3. Learning rate

   It should be small enough to avoid oscillation. Furthur exploration can be dynamic learning rate clipping.

Baseline result

The basic model turns out to perform bad online, the reason is probably:

1. This test set is very different from train set, no matter from class distribution (pos:neg = 1:3 for train set), or sentence features.

2. This deep neural model is too sophisticated, with so much weights in LSTM and fully connected classifier, it overfits and get overtrained easily.

    

Data Augmentation

Based one the baseline result, we need to consider other path to avoid overfitting. The amount of data can always give us a surprise. We have a unexploited treasure - 55669 unlabeled data sentences which can be critical with proper use.

Main Idea

Here's how we do it:

Constructing Spanish sentence pairs by aligning them in rows and columns, and calculating their similarities in a unsupervised way.

First question is how to embedding the sentences.

Following the simple and effective fashion, the first choice would be averaging every word embeddings in the sentence.

Alternatively, it could be done in a more elaborate way, using AutoEncoder to train a sentence encoder. As the data amount is large enough, the encoder may be able to capture proper representation.

Secondly, the similarity between two sentences can be measured by several kind of distances, I prefer the cosine and the word mover's distance. Here are a example done during my intern applying these two method to calculate phrases'(store tags) similarity. Phrase Similarity

Here are some other thoughts about the data augmentation, in a traditional way with synonym substitution, and an effective but not so practical way of double translation. Data Augmentation

Problems

In doing so, I encountered a large problem when calculating the huge similarity matrix. In this calculation, we need to do O(n^2) to get the similarity matrix, at best O(n)*O(logn) to select the k best and worse result for every sentence, while the n is near 50k, that is impossible to run on single PC, and still haven't figured out how to do it now.

Thus, I run this augmentation with some twitching on 700 to get 13216 positive samples and 11569 negtive samples, and had another run on 1000 sentences.

Augmentation result

Local loss is really good, but online still not ideal.

That may cause by the selection from the similarity matrix, selecting 10 best and 10 worse to be positive and negative examples makes the augmented data looks good on amounts, using only 700 sentences to get 24000 boosting on training data. But it actually has so many repeating data like (s1, s2) (s2, s1), that leads to a even more servere overfitting.

The ideal way of doing so is using all sentences to find top and bottom 1 and not duplicated sentence pairs. But how to do this efficiently is still puzzling me, hope readers can give me some hints. After doing so, the amount to be added into train set is still a problem to be discussed, how much is suitable to alleviate the overfitting?

 

Transfer Learning

As we are provided labeled English data, another thoughts would be using transfer learning.

A number of animal words went directly from Indian languages into Spanish and then English, (Puma originated in Quechua, while jaguar comes from yaguar). So I thought transfer may be useful on this task.

Main Idea

The idea is rather simple, train the siamese-LSTM on English labeled data first, and transfer neural network's weight to initialize Spanish model.

Transfer result

That is a quick and not fully extended attempt. Here are some after-thoughts: After transfer, there should be some frozen and unfrozen layers, especially the classifier layers, the English siamese may learn different features from Spanish input, so the classifier is doing a totally different job, which lead to a worse loss. Maybe we can freeze the classifier first and train encoder part, and then fine-tune the encoder part.

 

Ensemble

1. Weighted Average

   The result is probabilities which is a number between [0,1], the simplest way to do ensemble is the weighted average on this probabilities. The weight on each model can be manully adjusted according to single model performance. As the deep and ML models may perform well on different part of the data, this simple way renders a good result and our final submission is based on 0.5 weights on each model.

    

2. Stacking

Stacking can be more comprehensive, using the first level model to extract different features.

Implementation Details

 

Basic Model

Step by step Jupyter Notebook explanation: Explanation

Main : Run this to train model and inference

Configuration File : All configurations and parameters are set in here

Model : Siamese-LSTM model in PyTorch

Dataset : How samples are stored and extracted

Pre-processing for Sentences & Embedding : Pre-processing from raw data, embedding

 

Data Augmentation

Data Augmentation Jupyter notebook : Details in data augmentation using unlabeled data

Train with augmented data : Using augmented data with 700 unlabeled sentences to train model

Other Augmentation Methods: Augmentation with synonym substitution and double-translation

 

Transfer Learning

Transfer learning Jupyter Notebook explanation: Transfer Explanation

Transfer Main : Run this to train transfering model and inference

Transfer Configuration : Configuration file for transfer learning