[实现] 安装 TadasBaltrusaitis/OpenFace 并修改代码逻辑

最近需要用 TadasBaltrusaitis/OpenFace 做一个 Demo，安装与修改代码的过程十分繁琐，所以以本文进行记录，给需要的人留个参考。
链接：https://github.com/TadasBaltrusaitis/OpenFace

目录

1. 安装 OpenFace
2. 使用 OpenFace
3. 修改 OpenFace 代码逻辑

1. 安装 OpenFace

Windows Installation：https://github.com/TadasBaltrusaitis/OpenFace/wiki/Windows-Installation
Ubuntu Installation：https://github.com/TadasBaltrusaitis/OpenFace/wiki/Unix-Installation
Model Download：https://github.com/TadasBaltrusaitis/OpenFace/wiki/Model-download

（建议先认真看完作者写的 Wiki，再进行安装，这样即使报错也可以心里有数）

OpenFace 在 Windows 下安装非常简单，只需要先将文件下载好（链接），再执行脚本文件 download_models.ps1，下载相应的模型文件即可。

如果是要在 Ubuntu 下安装 OpenFace，就得直接将 Github 上的项目文件下载下来，先执行脚本文件 download_models.sh，下载相应的模型文件，再执行脚本文件 install.sh，进行项目的编译即可。这里需要提醒，由于 OpenFace 的依赖比较多，导致我在之前的环境下一直无法安装完成，直到重装了系统之后才安装完成，我个人感觉 OpenFacee 有点坑。

此外，我之前安装好后，运行时会报错 Could not find CEN patch experts，网上有博客指出去官网上下更多的的模型文件（链接），但是我是通过重新执行脚本文件 install.sh 解决的。

以下是网上所说的解决办法：

下载完后放到对应的文件夹内，如果按照官网的路径保存 .dat 文件依然报错，其实是保存路径不对，应该：

• 把12个.mat文件存放到：/home/username/OpenFace/matlab_version/models/cen下即可
• 把4个.dat文件存放到：/home/username/OpenFace/build/lib/local/LandmarkDetector/model/patch_experts即可

2. 使用 OpenFace

在 Windows 下，OpenFace 可以使用 GUI 进行操作，分别是：OpenFaceDemo 和 OpenFaceOffline。其中，OpenFaceOffline 可供调整的选项更多，一般都使用该程序。

此外，OpenFace 在 Ubuntu 和 Windows 下都支持命令行操作，可执行文件分别是：FaceLandmarkImg, FaceLandmarkVid, FaceLandmarkVidMulti, FeatureExtraction了，关于命令行参数的详情可以查看：Command line arguments。

3. 修改 OpenFace 代码逻辑

在 Ubuntu 下修改 OpenFace 的代码逻辑后，需要在 build 文件夹下，执行make命令，重新进行项目的编译。

四个可执行文件的源码路径：

OpenFace-master\exe\FaceLandmarkImg\FaceLandmarkImg.cpp
OpenFace-master\exe\FaceLandmarkVidFaceLandmarkVid.cpp
OpenFace-master\exe\FaceLandmarkVidMulti\FaceLandmarkVidMulti.cpp
OpenFace-master\exe\FeatureExtractionFeatureExtraction.cpp

以修改 FaceLandmarkVidMulti 的代码逻辑为例：

• 修改人脸数量限制设置

OpenFace-master\exe\FaceLandmarkVidMulti\FaceLandmarkVidMulti.cpp
Line 129:	int num_faces_max = 4;//修改此处即可对人脸数量设置进行修改

• 修改检测置信度阈值设置

OpenFace-master\lib\local\Utilities\include\Visualizer.h
Line 96:	double visualisation_boundary = 0.9;//当检测结果置信度低于阈值时，结果将不进行展示

• 限制程序展示 LandMark 结果

OpenFace-master\exe\FaceLandmarkVidMulti\FaceLandmarkVidMulti.cpp
Line 357:	visualizer.SetObservationLandmarks(face_models[model].detected_landmarks, face_models[model].detection_certainty);//此语句用于展示 LandMark 结果
Line 365:   open_face_rec.SetObservationLandmarks(face_models[model].detected_landmarks, face_models[model].GetShape(sequence_reader.fx, sequence_reader.fy, sequence_reader.cx, sequence_reader.cy),face_models[model].params_global, face_models[model].params_local, face_models[model].detection_certainty, face_models[model].detection_success);//此语句用于记录 LandMark 结果

• 修改 LandMark 可视化函数

OpenFace-master\lib\local\Utilities\src\Visualizer.cpp
Line 170:    void Visualizer::SetObservationLandmarks(const cv::Mat_<float>& landmarks_2D, double confidence, const cv::Mat_<int>& visibilities) //修改该函数即可

如果想修改 AU 与 LandMark 的展示逻辑，作者给出了一些详细的信息：Output Format，Facial Action Coding System。

Code:

void Visualizer::SetObservationActionUnits_New(const std::vector<std::pair<std::string, double> >& au_intensities, const std::vector<std::pair<std::string, double> >& au_occurences, const cv::Mat_<float>& landmarks_2D, double confidence, const cv::Mat_<int>& visibilities)
{
    if (au_intensities.size() > 0 || au_occurences.size() > 0)
    {

        std::set<std::string> au_names;
        std::map<std::string, bool> occurences_map;
        std::map<std::string, double> intensities_map;

        for (size_t idx = 0; idx < au_intensities.size(); idx++)
        {
            au_names.insert(au_intensities[idx].first);
            intensities_map[au_intensities[idx].first] = au_intensities[idx].second;
        }

        for (size_t idx = 0; idx < au_occurences.size(); idx++)
        {
            au_names.insert(au_occurences[idx].first);
            occurences_map[au_occurences[idx].first] = au_occurences[idx].second > 0;
        }

        const int AU_TRACKBAR_LENGTH = 400;
        const int AU_TRACKBAR_HEIGHT = 10;

        const int MARGIN_X = 185;
        const int MARGIN_Y = 10;

        const int nb_aus = (int) au_names.size();

        // Do not reinitialize
        if (action_units_image.empty())
        {
            action_units_image = cv::Mat(nb_aus * (AU_TRACKBAR_HEIGHT + 10) + MARGIN_Y * 2, AU_TRACKBAR_LENGTH + MARGIN_X, CV_8UC3, cv::Scalar(255, 255, 255));
        }
        else
        {
            action_units_image.setTo(255);
        }

        std::map<std::string, std::pair<bool, double>> aus;

        // first, prepare a mapping "AU name" -> { present, intensity }
        for (auto au_name : au_names)
        {
            // Insert the intensity and AU presense (as these do not always overlap check if they exist first)
            bool occurence = false;
            if (occurences_map.find(au_name) != occurences_map.end())
            {
                occurence = occurences_map[au_name] != 0;
            }
            else
            {
                // If we do not have an occurence label, trust the intensity one
                occurence = intensities_map[au_name] > 1;
            }
            double intensity = 0.0;
            if (intensities_map.find(au_name) != intensities_map.end())
            {
                intensity = intensities_map[au_name];
            }
            else
            {
                // If we do not have an intensity label, trust the occurence one
                intensity = occurences_map[au_name] == 0 ? 0 : 5;
            }

            aus[au_name] = std::make_pair(occurence, intensity);
        }
	// then, build the graph
	unsigned int idx = 0;
	for (auto& au : aus)
	{
		std::string name = au.first;
		bool present = au.second.first;
		double intensity = au.second.second;

		int offset = MARGIN_Y + idx * (AU_TRACKBAR_HEIGHT + 10);
		std::ostringstream au_i;
		au_i << std::setprecision(2) << std::setw(4) << std::fixed << intensity;
		cv::putText(action_units_image, name, cv::Point(10, offset + 10), cv::FONT_HERSHEY_SIMPLEX, 0.5, CV_RGB(present ? 0 : 200, 0, 0), 1, cv::LINE_AA);
		cv::putText(action_units_image, AUS_DESCRIPTION.at(name), cv::Point(55, offset + 10), cv::FONT_HERSHEY_SIMPLEX, 0.3, CV_RGB(0, 0, 0), 1, cv::LINE_AA);

		if (present)
		{
			cv::putText(action_units_image, au_i.str(), cv::Point(160, offset + 10), cv::FONT_HERSHEY_SIMPLEX, 0.3, CV_RGB(0, 100, 0), 1, cv::LINE_AA);
			cv::rectangle(action_units_image, cv::Point(MARGIN_X, offset),
				cv::Point((int)(MARGIN_X + AU_TRACKBAR_LENGTH * intensity / 5.0), offset + AU_TRACKBAR_HEIGHT),
				cv::Scalar(128, 128, 128),
				cv::FILLED);
		}
		else
		{
			cv::putText(action_units_image, "0.00", cv::Point(160, offset + 10), cv::FONT_HERSHEY_SIMPLEX, 0.3, CV_RGB(0, 0, 0), 1, cv::LINE_AA);
		}
		idx++;
	}
        // Draw the AU Land
        int n = landmarks_2D.rows / 2;
	int thickness = (int)std::ceil(3.0* ((double)captured_image.cols) / 640.0);
        int thickness_2 = (int)std::ceil(1.0* ((double)captured_image.cols) / 640.0);

        std::vector<cv::Point> Points;

	for (auto& au : aus)
        {
            std::string name = au.first;
            bool present = au.second.first;
            double intensity = au.second.second;

            if (present && intensity>=1.5)
            {
                if (name=="AU01")
                {
                    cv::Point featurePoint1(cvRound(landmarks_2D.at<float>(21) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(21 + n) * (float)draw_multiplier));
                    cv::Point featurePoint2(cvRound(landmarks_2D.at<float>(22) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(22 + n) * (float)draw_multiplier));
                    Points.push_back(featurePoint1);
		    Points.push_back(featurePoint2);
                }
                else if (name=="AU02")
                {
                    cv::Point featurePoint1(cvRound(landmarks_2D.at<float>(18) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(18 + n) * (float)draw_multiplier));
                    cv::Point featurePoint2(cvRound(landmarks_2D.at<float>(25) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(25 + n) * (float)draw_multiplier));
		    Points.push_back(featurePoint1);
		    Points.push_back(featurePoint2);
                }
                else if (name=="AU04")
                {
                    cv::Point featurePoint1(cvRound(landmarks_2D.at<float>(19) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(19 + n) * (float)draw_multiplier));
                    cv::Point featurePoint2(cvRound(landmarks_2D.at<float>(21) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(21 + n) * (float)draw_multiplier));
                    cv::Point featurePoint3(cvRound(landmarks_2D.at<float>(22) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(22 + n) * (float)draw_multiplier));
                    cv::Point featurePoint4(cvRound(landmarks_2D.at<float>(24) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(24 + n) * (float)draw_multiplier));
                    Points.push_back(featurePoint1);
		    Points.push_back(featurePoint2);
		    Points.push_back(featurePoint3);
		    Points.push_back(featurePoint4);
                }
                else if (name=="AU05")
                {
		    cv::Point featurePoint1(cvRound(landmarks_2D.at<float>(37) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(37 + n) * (float)draw_multiplier));
                    cv::Point featurePoint2(cvRound(landmarks_2D.at<float>(38) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(38 + n) * (float)draw_multiplier));
                    cv::Point featurePoint3(cvRound(landmarks_2D.at<float>(43) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(43 + n) * (float)draw_multiplier));
                    cv::Point featurePoint4(cvRound(landmarks_2D.at<float>(44) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(44 + n) * (float)draw_multiplier));
                    Points.push_back(featurePoint1);
		    Points.push_back(featurePoint2);
		    Points.push_back(featurePoint3);
		    Points.push_back(featurePoint4);
                }
                else if (name=="AU06")
                {
                    cv::Point featurePoint1(cvRound(landmarks_2D.at<float>(41) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(41 + n) * (float)draw_multiplier));
                    cv::Point featurePoint2(cvRound(landmarks_2D.at<float>(46) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(46 + n) * (float)draw_multiplier));
                    Points.push_back(featurePoint1);
		    Points.push_back(featurePoint2);
                }
                else if (name=="AU07")
                {
                    cv::Point featurePoint1(cvRound(landmarks_2D.at<float>(38) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(38 + n) * (float)draw_multiplier));
                    cv::Point featurePoint2(cvRound(landmarks_2D.at<float>(43) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(43 + n) * (float)draw_multiplier));
                    Points.push_back(featurePoint1);
		    Points.push_back(featurePoint2);
                }
                else if (name=="AU09")
                {
		    cv::Point featurePoint1(cvRound(landmarks_2D.at<float>(27) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(27 + n) * (float)draw_multiplier));
                    cv::Point featurePoint2(cvRound(landmarks_2D.at<float>(28) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(28 + n) * (float)draw_multiplier));
                    Points.push_back(featurePoint1);
		    Points.push_back(featurePoint2);
                }
                else if (name=="AU10")
                {
                    cv::Point featurePoint1(cvRound(landmarks_2D.at<float>(31) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(31 + n) * (float)draw_multiplier));
                    cv::Point featurePoint2(cvRound(landmarks_2D.at<float>(35) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(35 + n) * (float)draw_multiplier));
                    Points.push_back(featurePoint1);
		    Points.push_back(featurePoint2);
                }
                else if (name=="AU12")
                {
                    cv::Point featurePoint1(cvRound(landmarks_2D.at<float>(48) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(48 + n) * (float)draw_multiplier));
                    cv::Point featurePoint2(cvRound(landmarks_2D.at<float>(54) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(54 + n) * (float)draw_multiplier));
		    Points.push_back(featurePoint1);
		    Points.push_back(featurePoint2);
                }
                else if (name=="AU14")
                {
                    cv::Point featurePoint1(cvRound(landmarks_2D.at<float>(48) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(48 + n) * (float)draw_multiplier));
                    cv::Point featurePoint2(cvRound(landmarks_2D.at<float>(54) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(54 + n) * (float)draw_multiplier));
                    Points.push_back(featurePoint1);
		    Points.push_back(featurePoint2);
                }
                else if (name=="AU15")
                {
                    cv::Point featurePoint1(cvRound(landmarks_2D.at<float>(48) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(48 + n) * (float)draw_multiplier));
                    cv::Point featurePoint2(cvRound(landmarks_2D.at<float>(54) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(54 + n) * (float)draw_multiplier));
                    Points.push_back(featurePoint1);
		    Points.push_back(featurePoint2);
                }
                else if (name=="AU17")
                {
		    cv::Point featurePoint1(cvRound(landmarks_2D.at<float>(8) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(8 + n) * (float)draw_multiplier));
                    cv::Point featurePoint2(cvRound(landmarks_2D.at<float>(57) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(57 + n) * (float)draw_multiplier));
		    cv::Point featurePoint3(cvRound(landmarks_2D.at<float>(48) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(48 + n) * (float)draw_multiplier));
                    cv::Point featurePoint4(cvRound(landmarks_2D.at<float>(54) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(54 + n) * (float)draw_multiplier));
		    Points.push_back(featurePoint1);
		    Points.push_back(featurePoint2);
		    Points.push_back(featurePoint3);
		    Points.push_back(featurePoint4);
                }
                else if (name=="AU20")
                {
		    cv::Point featurePoint1(cvRound(landmarks_2D.at<float>(8) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(8 + n) * (float)draw_multiplier));
                    cv::Point featurePoint2(cvRound(landmarks_2D.at<float>(57) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(57 + n) * (float)draw_multiplier));
		    cv::Point featurePoint3(cvRound(landmarks_2D.at<float>(48) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(48 + n) * (float)draw_multiplier));
                    cv::Point featurePoint4(cvRound(landmarks_2D.at<float>(54) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(54 + n) * (float)draw_multiplier));
		    Points.push_back(featurePoint1);
		    Points.push_back(featurePoint2);
		    Points.push_back(featurePoint3);
		    Points.push_back(featurePoint4);
                }
                else if (name=="AU23")
                {
                    cv::Point featurePoint1(cvRound(landmarks_2D.at<float>(66) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(66 + n) * (float)draw_multiplier));
                    cv::Point featurePoint2(cvRound(landmarks_2D.at<float>(62) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(62 + n) * (float)draw_multiplier));
		    Points.push_back(featurePoint1);
		    Points.push_back(featurePoint2);
                    
                }
                else if (name=="AU25")
                {
                    cv::Point featurePoint1(cvRound(landmarks_2D.at<float>(61) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(61 + n) * (float)draw_multiplier));
                    cv::Point featurePoint2(cvRound(landmarks_2D.at<float>(63) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(63 + n) * (float)draw_multiplier));
		    cv::Point featurePoint3(cvRound(landmarks_2D.at<float>(65) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(65 + n) * (float)draw_multiplier));
                    cv::Point featurePoint4(cvRound(landmarks_2D.at<float>(67) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(67 + n) * (float)draw_multiplier));
		    Points.push_back(featurePoint1);
		    Points.push_back(featurePoint2);
		    Points.push_back(featurePoint3);
		    Points.push_back(featurePoint4);
                }
                else if (name=="AU26")
                {
                    cv::Point featurePoint1(cvRound(landmarks_2D.at<float>(61) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(61 + n) * (float)draw_multiplier));
                    cv::Point featurePoint2(cvRound(landmarks_2D.at<float>(63) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(63 + n) * (float)draw_multiplier));
		    cv::Point featurePoint3(cvRound(landmarks_2D.at<float>(65) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(65 + n) * (float)draw_multiplier));
                    cv::Point featurePoint4(cvRound(landmarks_2D.at<float>(67) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(67 + n) * (float)draw_multiplier));
		    Points.push_back(featurePoint1);
		    Points.push_back(featurePoint2);
		    Points.push_back(featurePoint3);
		    Points.push_back(featurePoint4);
                }
                else if (name=="AU28")
                {
                    cv::Point featurePoint1(cvRound(landmarks_2D.at<float>(61) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(61 + n) * (float)draw_multiplier));
                    cv::Point featurePoint2(cvRound(landmarks_2D.at<float>(63) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(63 + n) * (float)draw_multiplier));
		    cv::Point featurePoint3(cvRound(landmarks_2D.at<float>(65) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(65 + n) * (float)draw_multiplier));
                    cv::Point featurePoint4(cvRound(landmarks_2D.at<float>(67) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(67 + n) * (float)draw_multiplier));
		    Points.push_back(featurePoint1);
		    Points.push_back(featurePoint2);
		    Points.push_back(featurePoint3);
		    Points.push_back(featurePoint4);
                }
                else if (name=="AU45")
                {
                    cv::Point featurePoint1(cvRound(landmarks_2D.at<float>(38) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(38 + n) * (float)draw_multiplier));
                    cv::Point featurePoint2(cvRound(landmarks_2D.at<float>(43) * (float)draw_multiplier), cvRound(landmarks_2D.at<float>(43 + n) * (float)draw_multiplier));
		    Points.push_back(featurePoint1);
		    Points.push_back(featurePoint2);
                }

            }
        }
	    int count = Points.size();
    	for (int i = 0; i < count;i++)
    	{
        	cv::circle(captured_image, Points[i], 1 * draw_multiplier, cv::Scalar(0, 0, 255), thickness, cv::LINE_AA, draw_shiftbits);
        	cv::circle(captured_image, Points[i], 1 * draw_multiplier, cv::Scalar(255, 0, 0), thickness_2, cv::LINE_AA, draw_shiftbits);
		
    	}
	


    }
}

参考资料：

• OpenFace安装问题—Could not find CEN patch experts
• OpenFace

---

如果你看到了这篇文章的最后，并且觉得有帮助的话，麻烦你花几秒钟时间点个赞，或者受累在评论中指出我的错误。谢谢！

作者信息：
LeetCode：Tao Pu
CSDN：Code_Mart
Github：Bojack-want-drink