DynaSLAM-7 DynaSLAM中双目运行流程（Ⅰ）：加载Mask R-CNN网络部分MaskNet.cc

目录

1.执行流程

2. SegmentDynObject::SegmentDynObject

3. SegmentDynObject::GetSegmentation 

---

1.执行流程

        我们输入到命令行五个参数：

stereo_kitti path_to_vocabulary path_to_settings path_to_sequence (path_to_masks)

        分别是DynaSLAM双目例程中的可执行文件stereo_kittt、ORB字典文件、配置文件、序列文件、以及可选的存储mask的文件。

        输入完成后，我们便执行起了双目例程，首先进入的是LoadImages函数：

ORB-SLAM2 ---- LoadImages函数解析https://blog.csdn.net/qq_41694024/article/details/126285573        LoadImages函数的主要作用是将双目图像分割：将左目图像存储进vstrImageLeft容器中、右目图像存储进vstrImageRight容器中、对应的时间戳存储进vTimestamps中。

        随后我们就开始加载MaskRCNN网络了：

    // Initialize Mask R-CNN
    cout << "Loading Mask R-CNN. This could take a while..." << endl;
    DynaSLAM::SegmentDynObject *MaskNet;
    if (argc==5){
         MaskNet = new DynaSLAM::SegmentDynObject();
    }
    cout << "Mask net loaded!" << endl;

2. SegmentDynObject::SegmentDynObject

        这个类方法首先做的就是通过ImportSettings加载Mask R-CNN网络配置：

void SegmentDynObject::ImportSettings(){
    std::string strSettingsFile = "./Examples/RGB-D/MaskSettings.yaml";
    cv::FileStorage fs(strSettingsFile.c_str(), cv::FileStorage::READ);
    fs["py_path"] >> this->py_path;
    fs["module_name"] >> this->module_name;
    fs["class_name"] >> this->class_name;
    fs["get_dyn_seg"] >> this->get_dyn_seg;

%YAML:1.0
py_path: "./src/python/"
module_name: "MaskRCNN"
class_name: "Mask"
get_dyn_seg: "GetDynSeg"

        主要是加载了几个参数：

        @py_path ： src/python目录

        @module_name ： MaskRCNN文件

        @class_name：Mask类

        @get_dyn_seg：GetDynSeg MaskRCNN中的方法获取mask信息

def GetDynSeg(image,image2=None):
	h = image.shape[0]
	w = image.shape[1]
	if len(image.shape) == 2:
		im = np.zeros((h,w,3))
		im[:,:,0]=image
		im[:,:,1]=image
		im[:,:,2]=image
		image = im
	#if image2 is not None:
	#	args+=[image2]
	# Run detection
	results = model.detect([image], verbose=0)
	# Visualize results
	r = results[0]
	i = 0
	mask = np.zeros((h,w))
	for roi in r['rois']:
		if class_names[r['class_ids'][i]] == 'person':
			image_m = r['masks'][:,:,i]
			mask[image_m == 1] = 1.
		if class_names[r['class_ids'][i]] == 'bicycle':
			image_m = r['masks'][:,:,i]
			mask[image_m == 1] = 1.
		if class_names[r['class_ids'][i]] == 'car':
			image_m = r['masks'][:,:,i]
			mask[image_m == 1] = 1.
		if class_names[r['class_ids'][i]] == 'motorcycle':
			image_m = r['masks'][:,:,i]
			mask[image_m == 1] = 1.
		if class_names[r['class_ids'][i]] == 'airplane':
			image_m = r['masks'][:,:,i]
			mask[image_m == 1] = 1.
		if class_names[r['class_ids'][i]] == 'bus':
			image_m = r['masks'][:,:,i]
			mask[image_m == 1] = 1.
		if class_names[r['class_ids'][i]] == 'train':
			image_m = r['masks'][:,:,i]
			mask[image_m == 1] = 1.
		if class_names[r['class_ids'][i]] == 'truck':
			image_m = r['masks'][:,:,i]
			mask[image_m == 1] = 1.
		if class_names[r['class_ids'][i]] == 'boat':
			image_m = r['masks'][:,:,i]
			mask[image_m == 1] = 1.
		if class_names[r['class_ids'][i]] == 'bird':
			image_m = r['masks'][:,:,i]
			mask[image_m == 1] = 1.
		if class_names[r['class_ids'][i]] == 'cat':
			image_m = r['masks'][:,:,i]
			mask[image_m == 1] = 1.
		if class_names[r['class_ids'][i]] == 'dog':
			image_m = r['masks'][:,:,i]
			mask[image_m == 1] = 1.
		if class_names[r['class_ids'][i]] == 'horse':
			image_m = r['masks'][:,:,i]
			mask[image_m == 1] = 1.
		if class_names[r['class_ids'][i]] == 'sheep':
			image_m = r['masks'][:,:,i]
			mask[image_m == 1] = 1.
		if class_names[r['class_ids'][i]] == 'cow':
			image_m = r['masks'][:,:,i]
			mask[image_m == 1] = 1.
		if class_names[r['class_ids'][i]] == 'elephant':
			image_m = r['masks'][:,:,i]
			mask[image_m == 1] = 1.
		if class_names[r['class_ids'][i]] == 'bear':
			image_m = r['masks'][:,:,i]
			mask[image_m == 1] = 1.
		if class_names[r['class_ids'][i]] == 'zebra':
			image_m = r['masks'][:,:,i]
			mask[image_m == 1] = 1.
		if class_names[r['class_ids'][i]] == 'giraffe':
			image_m = r['masks'][:,:,i]
			mask[image_m == 1] = 1.		
		i+=1
	#print('GetSeg mask shape:',mask.shape)

	return mask

        setenv起到改变或增加环境变量的作用，用法如下：

        int setenv(const char *name,const char * value,int overwrite);

        @name为环境变量名称字符串。
        @value则为变量内容。

        @overwrite用来决定是否要改变已存在的环境变量。

        如果overwrite不为0，则改变环境变量原有内容，原有内容会被改为参数value所指的变量内容。如果overwrite为0，且该环境变量已有内容，则参数value会被忽略。
        @reuturn 执行成功则返回0，有错误发生时返回-1。

#include
main()
{
　　char * p;
　　if((p=getenv(“USER”)))
　　    printf(“USER =%s\n”,p);
　　setenv(“USER”,”test”,1);
　　printf(“USER=%s\n”,getenv(“USEr”));
　　unsetenv(“USER”);
　　printf(“USER=%s\n”,getenv(“USER”));
}

        执行结果：USER = root        USER = test          USER = (null)

setenv("PYTHONPATH", this->py_path.c_str(), 1);

        这里就是将./src/python文件夹设置为this->py_path。初始化Python虚拟环境。

namespace DynaSLAM
{

class SegmentDynObject{
private:
	NDArrayConverter *cvt; 	/*!< Converter to NumPy Array from cv::Mat */
	PyObject *py_module; 	/*!< Module of python where the Mask algorithm is implemented */
	PyObject *py_class; 	/*!< Class to be instanced */
	PyObject *net; 			/*!< Instance of the class */
	std::string py_path; 	/*!< Path to be included to the environment variable PYTHONPATH */
	std::string module_name; /*!< Detailed description after the member */
	std::string class_name; /*!< Detailed description after the member */
        std::string get_dyn_seg; 	/*!< Detailed description after the member */

	void ImportSettings();
public:

	SegmentDynObject();
        ~SegmentDynObject();
        cv::Mat GetSegmentation(cv::Mat &image, std::string dir="no_save", std::string rgb_name="no_file");
};

        初始化Numpy向cv::Mat的转化器cvt、导入py文库文件的过程：

        知识点请见博客：

如何在C++中使用一个Python类-[PyImport_ImportModule、PyModule_GetDict、PyDict_GetItemString、PyObject_CallFuncti]https://blog.csdn.net/Windgs_YF/article/details/91416252

SegmentDynObject::SegmentDynObject(){
    std::cout << "Importing Mask R-CNN Settings..." << std::endl;
    //fs["py_path"] >> this->py_path;
    //fs["module_name"] >> this->module_name;
    //fs["class_name"] >> this->class_name;
    //fs["get_dyn_seg"] >> this->get_dyn_seg;
    ImportSettings();
    std::string x;
    setenv("PYTHONPATH", this->py_path.c_str(), 1);
    x = getenv("PYTHONPATH");
    Py_Initialize();
    this->cvt = new NDArrayConverter();
    this->py_module = PyImport_ImportModule(this->module_name.c_str());
    assert(this->py_module != NULL);
    this->py_class = PyObject_GetAttrString(this->py_module, this->class_name.c_str());
    assert(this->py_class != NULL);
    this->net = PyInstance_New(this->py_class, NULL, NULL);
    assert(this->net != NULL);
    std::cout << "Creating net instance..." << std::endl;
    cv::Mat image  = cv::Mat::zeros(480,640,CV_8UC3); //Be careful with size!!
    std::cout << "Loading net parameters..." << std::endl;
    GetSegmentation(image);
}

        py_module 的类型是PyObject *类型，通过PyImport_ImportModule方法读入src/python目录下的MaskRCNN.py文件，如果读入文件为空报错。

        py_class 的类型是PyObject *类型，通过PyObject_GetAttrString方法导入已导入模块（MaskRCNN.py）中的方法或类（MaskRCNN.py中的Mask类）。

        net  的类型是PyObject *类型，通过PyInstance_New方法使用类构造对象。

        这样我们顺利地构造了MaskRCNN.py中的Mask类。

        下面我们创建网络实例：Creating net instance....，首先创建一个480*640的空白图像。

        调用GetSegmentation函数对该张图片进行处理。

3. SegmentDynObject::GetSegmentation 

cv::Mat SegmentDynObject::GetSegmentation(cv::Mat &image,std::string dir, std::string name){
    cv::Mat seg = cv::imread(dir+"/"+name,CV_LOAD_IMAGE_UNCHANGED);
    if(seg.empty()){
        PyObject* py_image = cvt->toNDArray(image.clone());
        assert(py_image != NULL);
        PyObject* py_mask_image = PyObject_CallMethod(this->net, const_cast(this->get_dyn_seg.c_str()),"(O)",py_image);
        seg = cvt->toMat(py_mask_image).clone();
        seg.cv::Mat::convertTo(seg,CV_8U);//0 background y 1 foreground
        if(dir.compare("no_save")!=0){
            DIR* _dir = opendir(dir.c_str());
            if (_dir) {closedir(_dir);}
            else if (ENOENT == errno)
            {
                const int check = mkdir(dir.c_str(), S_IRWXU | S_IRWXG | S_IROTH | S_IXOTH);
                if (check == -1) {
                    std::string str = dir;
                    str.replace(str.end() - 6, str.end(), "");
                    mkdir(str.c_str(), S_IRWXU | S_IRWXG | S_IROTH | S_IXOTH);
                }
            }
            cv::imwrite(dir+"/"+name,seg);
        }
    }
    return seg;
}

        本方法将传入的image进行语义分割，得到mask图。

        首先将image图片转换成tensor格式存储在py_image在，判断是否读取成功。

        通过PyObject_CallMethod方法调用（this -> net 是MaskRCNN.py中Mask类实例化的一个对象，this->get_dyn_seg 是该类中的一个方法求得图像的mask信息，再将其转化成图片格式保存起来。返回图像。

        现在Mask R-CNN网络初始化完毕，屏幕输出Mask net loaded!。