论文阅读12：Spatio-Temporal Tuples Transformer forSkeleton-Based Action Recognition-2021STTFormer

论文阅读12：Spatio-Temporal Tuples Transformer forSkeleton-Based Action Recognition-2021STTFormer

Abstract

• problems currently : the existing Transformer-based meth-ods cannot capture the correlation of different joints between frames

• The skeleton sequence is divided into several parts, and several consecu-tive frames contained in each part are encoded. And then a spatio-temporal tuplesself-attention module is proposed to capture the relationship of different joints inconsecutive frames. In addition, a feature aggregation module is introduced be-tween non-adjacent frames to enhance the ability to distinguish similar actions.

  the three points of this paper.

Introduction

• the above methods(RNN,CNN,GCN) cannot effectively model the long-term dependence of sequences and the global correlation of spatio-temporal joints. 是否可以使用例如non-local, slow-fast 之类的策略来进行skeleton based action recognition.

• 本文重点：extract the related features of different joints between adjacent frames

  这算是一种解释，而非出发点，感觉。

• 两个模块：

  • STTA - spatio-temporal tuple self-attention. 进行局部连续帧的注意力。

  • IFFA - 非连续帧间的注意力，信息整合。

Related Work

• Self-Attention Mechanism

• Skeleton-Based Action Recognition

• Context Aware-based Methods

Method

1. Overall Architecture

   

2. Spatio-Temporal Tuples Encoding
   

   • $InputX\in {\mathbb{R}}^{C_0\times T_0\times V_0}$

   • $conv1=conv1+bathNorm+LeakyReLU$.

     $ X$ shape : $C_0,T_0,V_0\to C_1,T_0,V_0$

   • skeleton sequence divide : $X=X.reshape(C_1,T,n,V_0),T_0=n\times T_0)$ 这里，$T\times n$ 表示，将原来序列分成 $T$段，每一段是原始序列中连续的$n$个序列。

   • flatten : $X\leftarrow X.reshape(C1,T,n\ast V_0)$

   • $conv2=convolutionallayer+LeakyRelu()$

3. Positional Encoding

   $\begin{array}{l}PE(p,2i)=\sin \left(p/10000^{2i/C_{in}}\right)\\ PE(p,2i+1)=\cos \left(p/10000^{2i/C_{in}}\right)\end{array}$

   其中$p$表示位置，$i$ 表示维度

4. Spatio-Temporal Tuples Transformer

   

   • Spatio-Temporal Tuples Attention

   1. 求Q,K,V

      $\mathbf{Q},\mathbf{K},\mathbf{V}=\mathrm{Conv}\ast 2D(1\times 1)\left(\mathbf{X}\ast in\right)$

   2. 求X,多头注意力

      • ${\mathbf{X}}_{attn}=\mathrm{Tanh}\left(\frac{{\mathbf{Q}\mathbf{K}}^{\mathbf{T}}}{\sqrt{C}}\right)\mathbf{V}$

      • ${\mathbf{X}}_{\text{Attn }}=\mathrm{Concat}\left({\mathbf{X}}_{attn}^1,\cdots ,{\mathbf{X}}_{\text{attn }}^h\right)$

      • ${\mathbf{X}}_{STTA}={\mathrm{Conv}}_{2D\left(1\times k_1\right)}\left({\mathbf{X}}_{Attn}\right)$

   3. 在feed forword 前进行残差，残差为 $conv1\times 1$

• • Inter-Frame Feature Aggregation

  ${\mathbf{X}}_{IFFA}={\mathrm{Conv}}_{2D\left(k_2\times 1\right)}\left({\mathbf{X}}_{STTA}\right)$

  $Xsahpe:C,T,V$, conv on temporal dimmension.

  Aggregation $T$ sub actions.

  当然残差链接依然是需要的。

Experiment

。。。。

Aggregation $T$ sub actions.

当然残差链接依然是需要的。

Experiment

。。。。