FairMOT多目标跟踪（安装+代码解读）

---

提示：这里可以添加本文要记录的大概内容：

论文：
FairMOT: On the Fairness of Detection and Re-Identification in Multiple Object Tracking**](http://arxiv.org/abs/2004.01888）

代码：https://github.com/ifzhang/FairMOT

---

0.安装步骤

1.首先创建虚拟环境，并安装相应包

conda create -n FairMOT
conda activate FairMOT
conda install pytorch==1.7.0 torchvision==0.8.0 cudatoolkit=10.2 -c pytorch
cd ${FAIRMOT_ROOT}
pip install cython
pip install -r requirements.txt

2.编译DCNv2(可变形卷积)

这里必须用pytorch1.7.1或者1.7.0。因为作者提供的dcnv2编译代码是基于这两个torch版本的
克隆链接： https://github.com/ifzhang/DCNv2/tree/pytorch_1.7

git clone -b pytorch_1.7 https://github.com/ifzhang/DCNv2.git
cd DCNv2
./make.sh

3.安装ffmppeg(测试用)

ffmpeg主要用于编辑连续帧图片与视频，实现转换：
1.下载代码： http://ffmpeg.org/download.html
2.安装 yasm

 sudo apt-get install yasm

3.安装sdl1.2

sudo apt-get install libsdl1.2-dev

4.安装 sdl2.0

sudo apt-get install libsdl2-dev

5.编译安装ffmpeg

进入到解压之后的 ffmpeg文件夹，依次执行以下命令：

 ./configure
make
sudo make install

6.测试是否安装成功

ffmpeg -version
ffplay -version

7.ffmpeg简单使用

ffmpeg 抽取视频帧 https://blog.csdn.net/weixin_43804210/article/details/107964643
ffmpeg 压缩视频 https://blog.csdn.net/weixin_43804210/article/details/108109386

#.连续帧图像转视频，也可以使用opencv-python。可参考连接：
https://blog.csdn.net/kxh123456/article/details/121692474
（该方法可能读取的连续帧图像是乱序的，需要自己加个排序代码。想要完整代码的可以私信我）

1.读入第一帧

代码如下（示例）track.py line55：

    for path, img, img0 in dataloader:
        timer.tic()
        blob = torch.from_numpy(img).cuda().unsqueeze(0)              # ([1, 3, 608, 1088])
        online_targets = tracker.update(blob, img0)

online_targets 已经得到56个结果，后面是结果可视化。下面看tracker.update做了什么。

1.检测

主要得到三个结果，hm(热图，2维)、wh(2维)、id_feature(512维)、reg(偏移量，2维)
在4倍下采样特征图上。随后经过后处理等操作，得到56个检测目标

output = self.model(im_blob)[-1]          # (1,1,152,272 ) (1,2,152,272 ) (1,512,152,272 ) (1,2,152,272 )
 hm = output['hm'].sigmoid_()
wh = output['wh']
id_feature = output['id']
id_feature = F.normalize(id_feature, dim=1)

reg = output['reg'] if self.opt.reg_offset else None
dets, inds = mot_decode(hm, wh, reg=reg, cat_spec_wh=self.opt.cat_spec_wh, K=self.opt.K)   # (1,128,6) (1,128)
        
dets = self.post_process(dets, meta)                 # (1,128,6)     -->  dets[1]: ( 128,5 )
dets = self.merge_outputs([dets])[1]

2.跟踪

依次经过以下2步。

Step 2: First association, with embedding
Step 3: Second association, with IOU

由于此前 self.tracked_stracks 与 self.lost_stracks 为空，以上两步省略，直接创建新目标

        for inew in u_detection:
            track = detections[inew]
            if track.score < self.det_thresh:
                continue
            track.activate(self.kalman_filter, self.frame_id)
            activated_starcks.append(track)

这里的u_detection 与 detections就是刚才的检测结果。activate代码如下：

    def activate(self, kalman_filter, frame_id):
        """Start a new tracklet"""
        self.kalman_filter = kalman_filter
        self.track_id = self.next_id()
        self.mean, self.covariance = self.kalman_filter.initiate(self.tlwh_to_xyah(self._tlwh))   # (8) (8,8)

        self.tracklet_len = 0
        self.state = TrackState.Tracked          # 1
        if frame_id == 1:
            self.is_activated = True
        #self.is_activated = True
        self.frame_id = frame_id
        self.start_frame = frame_id

然后就是可视化。这里第一帧检测结果就开始跟踪了，新创建的tracker: state=True，并且is_activated（*到后面的新检测结果就不激活了，这里注意 *）.

2.第二帧

1.对第二帧检测

这里省略，代码同上。检测得到57个结果：dets、 detections、inds

2.跟踪

    unconfirmed = []
    tracked_stracks = []  # type: list[STrack]
    for track in self.tracked_stracks:              # 上一帧的56个结果
        if not track.is_activated:
            unconfirmed.append(track)               # 不执行
        else:
            tracked_stracks.append(track)           # 执行

Step 2: First association, with embedding ：目标间的特征距离

1.dists = matching.embedding_distance(strack_pool, detections)

    strack_pool = joint_stracks(tracked_stracks, self.lost_stracks)
    # Predict the current location with KF
    #for strack in strack_pool:
        #strack.predict()
    STrack.multi_predict(strack_pool)          # 更新卡尔曼滤波参数(更新后的mean conv等于旧的)
    dists = matching.embedding_distance(strack_pool, detections)   
    # 计算56个跟踪结果和57个检测结果，对应特征的余弦距离。计算代码如下：

matching.***embedding_distance***：计算跟踪目标与新检测目标的余弦距离：

def embedding_distance(tracks, detections, metric='cosine'):
    cost_matrix = np.zeros((len(tracks), len(detections)), dtype=np.float)
    det_features = np.asarray([track.curr_feat for track in detections], dtype=np.floa   # (57,512)
    track_features = np.asarray([track.smooth_feat for track in tracks], dtype=np.float) # (56,512)
    cost_matrix = np.maximum(0.0, cdist(track_features, det_features, metric))  # Nomalized features
    return cost_matrix                # （56，57）

2.dists = matching.fuse_motion(self.kalman_filter, dists, strack_pool, detections)

def fuse_motion(kf, cost_matrix, tracks, detections, only_position=False, lambda_=0.98):
    if cost_matrix.size == 0:
        return cost_matrix
    gating_dim = 2 if only_position else 4            # 4
    gating_threshold = kalman_filter.chi2inv95[gating_dim]           # 9.48
    measurements = np.asarray([det.to_xyah() for det in detections])    # ( 57,4 )
    for row, track in enumerate(tracks):
        gating_distance = kf.gating_distance(
            track.mean, track.covariance, measurements, only_position, metric='maha')
        # 门距离，根据预测值(跟踪器)与观测值(检测结果)的位置和距离计算，维度（57）
        
        cost_matrix[row, gating_distance > gating_threshold] = np.inf
        cost_matrix[row] = lambda_ * cost_matrix[row] + (1 - lambda_) * gating_distance
        # 按照0.98和0.02的权重分配： 余弦距离与gating_distance
    return cost_matrix                                      #(56, 57)

3.matches, u_track, u_detection = matching.linear_assignment(dists, thresh=0.7)
matchs有56个，u_track无，u_detection有一个

def linear_assignment(cost_matrix, thresh):
    matches, unmatched_a, unmatched_b = [], [], []
    cost, x, y = lap.lapjv(cost_matrix, extend_cost=True, cost_limit=thresh) 
    # cost:1.63 x:(56) y:(57)
    
    for ix, mx in enumerate(x):
        if mx >= 0:
            matches.append([ix, mx])
    unmatched_a = np.where(x < 0)[0]        # 无
    unmatched_b = np.where(y < 0)[0]        # [56]：第56个值为-1，没有找到匹配目标
    matches = np.asarray(matches)
    return matches, unmatched_a, unmatched_b     # （56，2） （）  [56]

lap.lapjv：做匈牙利匹配。输入是一个n*m得分矩阵，返回三个值：
c: 赋值的代价(越小越好)，如果return_cost为False，则不返回。
x: 一个大小为n的数组，用于指定每一行被分配到哪一列。
y: 一个大小为m的数组，用于指定每列被分配到哪一行。
5.

  for itracked, idet in matches:                        # 循环56个已匹配目标
      track = strack_pool[itracked]
      det = detections[idet]
      if track.state == TrackState.Tracked:
          track.update(detections[idet], self.frame_id)  # 根据匹配结果，更新卡尔曼滤波(包括特征值)
          activated_starcks.append(track)
      else:
          track.re_activate(det, self.frame_id, new_id=False)
          refind_stracks.append(track)

Step 3: Second association, with IOU

u_track为空，因此r_tracked_stracks为空，dists为空。

    detections = [detections[i] for i in u_detection]           # ( 56 ) * OT_O_(0-0)
    r_tracked_stracks = [strack_pool[i] for i in u_track if strack_pool[i].state == TrackState.Tracked]   # []
    dists = matching.iou_distance(r_tracked_stracks, detections)                # []
    matches, u_track, u_detection = matching.linear_assignment(dists, thresh=0.5)    # [] () (0)

Step 3.5：Deal with unconfirmed tracks

没做任何操作，因为 unconfirmed 为空

 detections = [detections[i] for i in u_detection]                         # 【56】
 dists = matching.iou_distance(unconfirmed, detections)        # unconfirmed为[],dists为[]
 matches, u_unconfirmed, u_detection = matching.linear_assignment(dists, thresh=0.7)    # [] () (0)

Step 4: Init new stracks

为 u_detection 中的一个目标创建跟踪器。

    for inew in u_detection:
        track = detections[inew]
        if track.score < self.det_thresh:
            continue
        track.activate(self.kalman_filter, self.frame_id)
        activated_starcks.append(track)

Step 5: Update state
没执行，因为 self.lost_stracks 为空

3.更新整个跟踪器

        self.tracked_stracks = [t for t in self.tracked_stracks if t.state == TrackState.Tracked]   # 56个（之前的）
        self.tracked_stracks = joint_stracks(self.tracked_stracks, activated_starcks)                   # 57个（加上新的检测）
        self.tracked_stracks = joint_stracks(self.tracked_stracks, refind_stracks)                          # 无增加：refind_stracks为空
        self.lost_stracks = sub_stracks(self.lost_stracks, self.tracked_stracks)                        # 无
        self.lost_stracks.extend(lost_stracks)                                                                                         # 无
        self.lost_stracks = sub_stracks(self.lost_stracks, self.removed_stracks)                     # 无
        self.removed_stracks.extend(removed_stracks)                                                                   # 无
        self.tracked_stracks, self.lost_stracks = remove_duplicate_stracks(self.tracked_stracks, self.lost_stracks)
       
        output_stracks = [track for track in self.tracked_stracks if track.is_activated]        # （56）个

3.第三帧

检测结果为55个目标

        detections = [STrack(STrack.tlbr_to_tlwh(tlbrs[:4]), tlbrs[4], f, 30) for
                                  (tlbrs, f) in zip(dets[:, :5], id_feature)]                    # 55
                                  
        unconfirmed = []
        tracked_stracks = []  # type: list[STrack]
        for track in self.tracked_stracks:                          # 上一帧57个目标
            if not track.is_activated:
                unconfirmed.append(track)                         # 一个目标
            else:
                tracked_stracks.append(track)                  # 上一帧56个目标

Step 2: First association, with embedding

        strack_pool = joint_stracks(tracked_stracks, self.lost_stracks)    # 56

        STrack.multi_predict(strack_pool)
        dists = matching.embedding_distance(strack_pool, detections)       # ( 56,55)
        #dists = matching.gate_cost_matrix(self.kalman_filter, dists, strack_pool, detections)
        dists = matching.fuse_motion(self.kalman_filter, dists, strack_pool, detections)
        matches, u_track, u_detection = matching.linear_assignment(dists, thresh=0.7)   # (55，2)  [52]  []

然后对 matches, u_track, u_detection 依次处理
更新已匹配的55个跟踪器：

        for itracked, idet in matches:
            track = strack_pool[itracked]
            det = detections[idet]
            if track.state == TrackState.Tracked:
                track.update(detections[idet], self.frame_id)
                activated_starcks.append(track)                                    # 55个
            else:
                track.re_activate(det, self.frame_id, new_id=False)
                refind_stracks.append(track)                                           # 无

Step 3: Second association, with IOU

计算 多出(未配对)的检测结果，与多出(未配对)的跟踪器，之间的iou距离：

        detections = [detections[i] for i in u_detection]             # 空：多出(未配对)的检测结果
        r_tracked_stracks = [strack_pool[i] for i in u_track if strack_pool[i].state == TrackState.Tracked]   # u_track：[52]
        dists = matching.iou_distance(r_tracked_stracks, detections)                # []
        matches, u_track, u_detection = matching.linear_assignment(dists, thresh=0.5)    # （） (0) （）

处理未配对的跟踪器（跟踪结果未与检测结果匹配）

        for it in u_track:
            track = r_tracked_stracks[it]                              # 上一帧未匹配的那一个跟踪器
            if not track.state == TrackState.Lost:             # 执行
                track.mark_lost()                                               # track.state = 2
                lost_stracks.append(track)                           # lost_stracks由 [] 增加一个

处理上一帧未激活的跟踪器（也就是第二帧中(57)，未与第一帧(56)配对的那一个检测结果）

        detections = [detections[i] for i in u_detection]                                                # []
        dists = matching.iou_distance(unconfirmed, detections)                            # []
        matches, u_unconfirmed, u_detection = matching.linear_assignment(dists, thresh=0.7)    # () (0) ()
      
        for it in u_unconfirmed:
            track = unconfirmed[it]
            track.mark_removed()                                      # track.state = 1  -->  =3 状态转为3
            removed_stracks.append(track)                  # 由[] 添加1

Step 4: Init new stracks ：由于u_detection为空，不执行
Step 5: Update state 由于self.lost_stracks为空，不执行

更新整个跟踪器

        self.tracked_stracks = [t for t in self.tracked_stracks if t.state == TrackState.Tracked]   # 55个
        self.tracked_stracks = joint_stracks(self.tracked_stracks, activated_starcks)                   # 55个
        self.tracked_stracks = joint_stracks(self.tracked_stracks, refind_stracks)                          # 55个 无增加：refind_stracks为空
        self.lost_stracks = sub_stracks(self.lost_stracks, self.tracked_stracks)                        # 无
        self.lost_stracks.extend(lost_stracks)                                                                                         # 增一个：未匹配的跟踪[OT_53_(1-2)]
        self.lost_stracks = sub_stracks(self.lost_stracks, self.removed_stracks)                     # 无增加，还是1个
        self.removed_stracks.extend(removed_stracks)                                                                   # 一个:第2帧中多出来的检测
        self.tracked_stracks, self.lost_stracks = remove_duplicate_stracks(self.tracked_stracks, self.lost_stracks)
        # （55）（1）
        
        output_stracks = [track for track in self.tracked_stracks if track.is_activated]        # （55）个

4.第四帧

检测结果为55个目标
每次跟踪开始前，先初始化：

        self.frame_id += 1
        activated_starcks = []
        refind_stracks = []
        lost_stracks = []
        removed_stracks = []

unconfirmed 没有，tracked_stracks有55个：

        unconfirmed = []
        tracked_stracks = []  
        for track in self.tracked_stracks:
            if not track.is_activated:
                unconfirmed.append(track)
            else:
                tracked_stracks.append(track)

Step 2: First association, with embeddig

strack_pool有56个（添加了上一帧中的self.lost_stracks）

        strack_pool = joint_stracks(tracked_stracks, self.lost_stracks)    # 56

        STrack.multi_predict(strack_pool)
        dists = matching.embedding_distance(strack_pool, detections)       # ( 56,55)
        dists = matching.fuse_motion(self.kalman_filter, dists, strack_pool, detections)    # ( 56,55)
        matches, u_track, u_detection = matching.linear_assignment(dists, thresh=0.7)   # (55，2)  [34]  []

处理配对的55个跟踪器：
activated_starcks：54个 ，refind_stracks：1个（[53]）

        for itracked, idet in matches:
            track = strack_pool[itracked]
            det = detections[idet]
            if track.state == TrackState.Tracked:
                track.update(detections[idet], self.frame_id)
                activated_starcks.append(track)
            else:
                track.re_activate(det, self.frame_id, new_id=False)
                refind_stracks.append(track)                                                 # 上一帧的self.lost_stracks

Step 3: Second association, with IOU

detections 为空，u_track 有1个，r_tracked_stracks 1个：

        detections = [detections[i] for i in u_detection]           # ( 56 ) * OT_O_(0-0)
        r_tracked_stracks = [strack_pool[i] for i in u_track if strack_pool[i].state == TrackState.Tracked]   # []
        dists = matching.iou_distance(r_tracked_stracks, detections)                # []
        matches, u_track, u_detection = matching.linear_assignment(dists, thresh=0.5)    # [] (0) ()

对u_track（为跟检测结果匹配）的跟踪器，标为lost：

        for it in u_track:
            track = r_tracked_stracks[it]
            if not track.state == TrackState.Lost:
                track.mark_lost()
                lost_stracks.append(track)

Deal with unconfirmed tracks, usually tracks with only one beginning frame
Step 4: Init new stracks

因为没有u_detection，这步跳过，直接第5步骤：

Step 5: Update state

self.lost_stracks是上一帧的未匹配跟踪器，不是刚才的lost_stracks：

       for track in self.lost_stracks:
            if self.frame_id - track.end_frame > self.max_time_lost:        # 判断是否丢失超过15秒
                track.mark_removed()
                removed_stracks.append(track)                                                   # 未执行

更新整个跟踪器

        self.tracked_stracks = [t for t in self.tracked_stracks if t.state == TrackState.Tracked]   # 54
        self.tracked_stracks = joint_stracks(self.tracked_stracks, activated_starcks)                  # 54
        self.tracked_stracks = joint_stracks(self.tracked_stracks, refind_stracks)                         # 55
        
        self.lost_stracks = sub_stracks(self.lost_stracks, self.tracked_stracks)                              #     []
        self.lost_stracks.extend(lost_stracks)                                                                                               # 1个：【35】
        self.lost_stracks = sub_stracks(self.lost_stracks, self.removed_stracks)                           # 1个：【35】
        self.removed_stracks.extend(removed_stracks)                                                                         #  1个 [57]
        self.tracked_stracks, self.lost_stracks = remove_duplicate_stracks(self.tracked_stracks, self.lost_stracks) # (55) [35]
        # get scores of lost tracks
        output_stracks = [track for track in self.tracked_stracks if track.is_activated]               # [55]

因为上一帧的self.lost_stracks，在这帧已经匹配了，所以变成了refind_stracks

5.第5帧

检测结果为56个目标
上一帧跟踪器为55个，unconfirmed为0

        unconfirmed = []
        tracked_stracks = []  
        for track in self.tracked_stracks:
            if not track.is_activated:
                unconfirmed.append(track)
            else:
                tracked_stracks.append(track)

Step 2: First association, with embedding

        strack_pool = joint_stracks(tracked_stracks, self.lost_stracks)    # 56

        STrack.multi_predict(strack_pool)
        dists = matching.embedding_distance(strack_pool, detections)            # ( 56,56)
        dists = matching.fuse_motion(self.kalman_filter, dists, strack_pool, detections)     # ( 56,56)
        matches, u_track, u_detection = matching.linear_assignment(dists, thresh=0.7)    # (55，2)  [55]  [48]

更新已匹配的55个match：

        for itracked, idet in matches:
            track = strack_pool[itracked]
            det = detections[idet]
            if track.state == TrackState.Tracked:
                track.update(detections[idet], self.frame_id)
                activated_starcks.append(track)                                               # 55 个
            else:
                track.re_activate(det, self.frame_id, new_id=False)
                refind_stracks.append(track)                                                     # 【】

Step 3: Second association, with IOU

        detections = [detections[i] for i in u_detection]           #  OT_O_(0-0)
        r_tracked_stracks = [strack_pool[i] for i in u_track if strack_pool[i].state == TrackState.Tracked]    # []
        dists = matching.iou_distance(r_tracked_stracks, detections)                # []
        matches, u_track, u_detection = matching.linear_assignment(dists, thresh=0.5)    # [] () (0)

'Deal with unconfirmed tracks, usually tracks with only one beginning frame

        detections = [detections[i] for i in u_detection]                                                      # 1个
        dists = matching.iou_distance(unconfirmed, detections)                                 # []
        matches, u_unconfirmed, u_detection = matching.linear_assignment(dists, thresh=0.7)    # [] () (0)
        for itracked, idet in matches:                                                                                          # 不执行
            unconfirmed[itracked].update(detections[idet], self.frame_id)
            activated_starcks.append(unconfirmed[itracked])
        for it in u_unconfirmed:                                                       
            track = unconfirmed[it]
            track.mark_removed()           # track.state = 1  --> track.state = 3
            removed_stracks.append(track)

Step 4: Init new stracks

activated_starcks 增加1，其state = 1 ：

        for inew in u_detection:
            track = detections[inew]
            if track.score < self.det_thresh:
                continue
            track.activate(self.kalman_filter, self.frame_id)
            activated_starcks.append(track)

更新整个跟踪器

        self.tracked_stracks = [t for t in self.tracked_stracks if t.state == TrackState.Tracked]       # 55
        self.tracked_stracks = joint_stracks(self.tracked_stracks, activated_starcks)                       # 56
        self.tracked_stracks = joint_stracks(self.tracked_stracks, refind_stracks)                              # 55
        self.lost_stracks = sub_stracks(self.lost_stracks, self.tracked_stracks)
        self.lost_stracks.extend(lost_stracks)
        self.lost_stracks = sub_stracks(self.lost_stracks, self.removed_stracks)
        self.removed_stracks.extend(removed_stracks)
        self.tracked_stracks, self.lost_stracks = remove_duplicate_stracks(self.tracked_stracks, self.lost_stracks)
        # get scores of lost tracks
        output_stracks = [track for track in self.tracked_stracks if track.is_activated]               # 55个（其中有个跟踪器，状态是未激活的）