C++ 调用 Python 总结(一)

#PS：要转载请注明出处，本人版权所有

#PS:这个只是 《 我自己 》理解，如果和你的

#原则相冲突，请谅解，勿喷

序

我们有个任务，需要在C++里面调用pycaffe的算法来做相关的检测。（不要问我为啥不直接用caffe的c++接口，因为后面还要调tensorflow和pytorch，导致了算法组为了统(偷)一（懒），就直接怼了一个pycaffe）。

我还是第一次遇到这类问题，我去查了查，还真的有相关的内容，真的是存在即合理。

本文大部分内容都是主要参考python官方手册，其次也看了网络上的一些资料，我这里做了一些汇总和衍生。(官方手册https://docs.python.org/zh-cn/3/c-api/index.html)

阅读本文需要一定的c++基本‘’姿势‘’和了解一些最最最简单的py‘’姿势‘’。

本文所有实例环境为：
win10+py35

Python C Hello World

先来一个全世界通用的例子。(没做异常处理)

#ifdef __cplusplus
extern "C"{
#endif //
/*
Note Since Python may define some pre-processor definitions which affect the standard headers on some systems, you must include Python.h before any standard headers are included.
It is recommended to always define PY_SSIZE_T_CLEAN before including Python.h. See Parsing arguments and building values for a description of this macro.
*/
#define PY_SSIZE_T_CLEAN
#include 

#ifdef __cplusplus
}
#endif //

#include 
int main(int argc, char * argv[]){

    //This function works like Py_Initialize() if initsigs is 1. 
    //If initsigs is 0, it skips initialization registration of signal handlers, which might be useful when Python is embedded.
    Py_InitializeEx(1);
    PyRun_SimpleString("print('hello world.')");
    Py_Finalize();
#ifdef _WIN32 || _WIN64

    system("pause");

#elif __linux__ || __linux


#endif //    
	return 0;
}

编译，设定python的头文件和库文件路径。（注意python的库名字），我这里是在win上做的演示，实际是部署到linux.相关结果如图：

Python C 常用接口

python 基本数据类型对象

更加详细的，请查看官方doc中关于Py_BuildValue()接口的描述。

    PyObject * func_name = Py_BuildValue("s","test_add");
    PyObject * arg1 = Py_BuildValue("l",3);

python 序列或者容器对象

更多内容，详见官方doc

//tuple
PyObject* PyTuple_New(Py_ssize_t len)
//Return value: New reference.
//Return a new tuple object of size len, or NULL on failure.

PyObject* PyTuple_GetItem(PyObject *p, Py_ssize_t pos)
//Return value: Borrowed reference.
//Return the object at position pos in the tuple pointed to by p. If pos is out of bounds, return NULL and set an IndexError exception.

int PyTuple_SetItem(PyObject *p, Py_ssize_t pos, PyObject *o)
//Insert a reference to object o at position pos of the tuple pointed to by p. Return 0 on success. If pos is out of bounds, return -1 and set an IndexError exception.
//list
//原理同tuple
PyObject* PyList_New(Py_ssize_t len)
int PyList_SetItem(PyObject *list, Py_ssize_t index, PyObject *item)
PyObject* PyList_GetItem(PyObject *list, Py_ssize_t index)
//dict
//原理同tuple
PyObject* PyDict_New()
int PyDict_SetItem(PyObject *p, PyObject *key, PyObject *val)
PyObject* PyDict_GetItem(PyObject *p, PyObject *key)

对于这些python中的数据类型对象，都有相关的c api来创建、操作，更多的内容可以查看官方doc。这部分内容较简单，不做示例。

python 模块对象

PyObject* PyImport_ImportModuleEx(const char *name, PyObject *globals, PyObject *locals, PyObject *fromlist)
/*
Return value: New reference.
Import a module. This is best described by referring to the built-in Python function __import__().

The return value is a new reference to the imported module or top-level package, or NULL with an exception set on failure. Like for __import__(), the return value when a submodule of a package was requested is normally the top-level package, unless a non-empty fromlist was given.

Failing imports remove incomplete module objects, like with PyImport_ImportModule().
*/
PyObject* PyModule_GetDict(PyObject *module)
/*
Return value: Borrowed reference.
Return the dictionary object that implements module's namespace; this object is the same as the __dict__ attribute of the module object. If module is not a module object (or a subtype of a module object), SystemError is raised and NULL is returned.

It is recommended extensions use other PyModule_*() and PyObject_*() functions rather than directly manipulate a module's __dict__.
*/

这些api主要是导入python模块，类似关键字import，同时返回模块的属性dict。这些属性包含当前命名空间下的：全局变量，全局函数，类等等。

python callable object , python class object, python object function

/*
Return value: New reference.
Call a callable Python object callable, with arguments given by the tuple args. If no arguments are needed, then args can be NULL.

Return the result of the call on success, or raise an exception and return NULL on failure.

This is the equivalent of the Python expression: callable(*args).
*/
PyObject* PyObject_CallObject(PyObject *callable, PyObject *args)


/*
Return value: New reference.
Return a new instance method object, with func being any callable object func is the function that will be called when the instance method is called.
*/
PyObject* PyInstanceMethod_New(PyObject *func)

//call obj.func
PyObject* PyObject_CallMethod(PyObject *obj, const char *name, const char *format, ...)
PyObject* PyObject_CallMethodObjArgs(PyObject *obj, PyObject *name, ..., NULL)

PyObject 对象

这个对象是pythonc c api 所有对象的wrapper，有许许多多需要注意的事项，其中最重要的就是其内存管理问题。详细细节参考官方doc，这里给出最重要的几条我遇到的结论：

1. pyobject 是依靠引用计数来管理对象的。
2. pyobject 的创建者需要对这个obj负责，创建者可以传递，存储，和调用 Py_DECREF()
3. python c api 都有关于pyobject的处理说明，比如：借用，新建。如下图。

本章特别重要，有兴趣的去看官方文档。当你乱用Py_INCREF和Py_DECREF时，你写的程序会在调用Py_Finalize时崩溃，这个时候，你需要看下文，去看看你哪个地方用错了。
（https://docs.python.org/3.8/extending/extending.html#ownership-rules）

实例

上一个实例来展示本文所有内容。
这是我要调用的python module

'''
'''
@Description: 
@Author: Sky
@Date: 2019-12-05 16:23:29
@LastEditors  : Sky
@LastEditTime : 2020-01-09 18:04:41
@FilePath: \Test_C_Call_Python\py_modules\py_normal_test.py
'''
import sys

class TestPythonNormal:
    cls_attr = 'I am cls attr'
    # def __new__(class_ins):
    #     print('call TestPythonNormal.__new__')
    #     return class_ins
    def __init__(self):
        print('call TestPythonNormal.__init__')
        self.obj_attr = 'I am obj attr'

    def test_add(self, a = 0 , b = 0):
        print("a = ", a)
        print("b = ",  b)
        print("a + b = ", (a + b))
        return (a + b)


g_num = 22222
g_str = 'hello f***'

def g_func(a):
    print("g_func arg = ", a - 1)

if __name__ == '__main__':
    test = TestPythonNormal()
    test.test_add(2, 3)
    print(TestPythonNormal.__dict__)
    # print(TestPythonNormal.__dir__(test))

/*
 * @Description: 
 * @Author: Sky
 * @Date: 2020-01-09 11:26:28
 * @LastEditors  : Sky
 * @LastEditTime : 2020-01-09 18:21:53
 * @FilePath: \Test_C_Call_Python\test_tmp.cpp
 * @Github: 
 */
#ifdef __cplusplus
extern "C"{
#endif //

#define PY_SSIZE_T_CLEAN
#include 

#ifdef __cplusplus
}
#endif //

#include 

int main(int argc, char * argv[]){

    //This function works like Py_Initialize() if initsigs is 1. 
    //If initsigs is 0, it skips initialization registration of signal handlers, which might be useful when Python is embedded.
    Py_InitializeEx(1);
    PyRun_SimpleString("print('hello world.')");

    //add path of module-py_normal_test to sys.path
    PyRun_SimpleString("import sys");
    PyRun_SimpleString("import numpy as np");
    PyRun_SimpleString("sys.path.append('E://TestDir//Test_C_Call_Python//py_modules')");
    PyRun_SimpleString("print(sys.path)");

    //import 
    //Return value: New reference.
    PyObject * p_module = PyImport_ImportModuleEx("py_normal_test", NULL, NULL, NULL);//import module

    //get module attr, they are g_var, g_func, class and so on.
    //Return value: Borrowed reference.
    PyObject * p_dict_mo = PyModule_GetDict(p_module);//get module's all attr
    //Return value: Borrowed reference.
    PyObject * p_module_g_num = PyDict_GetItemString(p_dict_mo, "g_num");// global var

    PyObject * p_module_cls = PyDict_GetItemString(p_dict_mo, "TestPythonNormal");//class

    PyObject * p_module_g_func = PyDict_GetItemString(p_dict_mo, "g_func");// global function

    //print g_var
    std::cout<<"g_num is "<<PyLong_AsLong(p_module_g_num)<<std::endl;

    //call g_func
    //Return value: New reference.
    PyObject * arg_tuple = PyTuple_New(1);
    PyTuple_SetItem(arg_tuple, 0, p_module_g_num);//prepare arg
    PyObject_CallObject(p_module_g_func, arg_tuple);//call g_func
    

    //instance a class
    //Return value: New reference.
    PyObject * p_cls_instance = PyInstanceMethod_New(p_module_cls);//call builtin.__new__, new a instance
    //Return value: New reference.
    p_cls_instance = PyObject_CallObject(p_cls_instance, NULL);//call __init__,construct function,return None
    // Py_XDECREF(p_none);

    //print class-var
    //Return value: New reference.
    //PyObject* PyObject_GetAttrString(PyObject *o, const char *attr_name)
    PyObject * cls_attr = PyObject_GetAttrString(p_module_cls, "cls_attr");
    std::cout<<"cls_attr is "<<PyUnicode_AsUTF8(cls_attr)<<std::endl;
    
    //print obj.var
    PyObject * obj_attr = PyObject_GetAttrString(p_cls_instance, "obj_attr");
    std::cout<<"obj_attr is "<<PyUnicode_AsUTF8(obj_attr)<<std::endl;

    //call obj.func
    //Way0: This is the equivalent of the Python expression: obj.name(arg1, arg2, ...).
    //Return value: New reference.Return value: New reference.
    PyObject * ret0 = PyObject_CallMethod(p_cls_instance, "test_add", "ii", 10, 10);//test_add(self, 10, 10)
    std::cout<<"call obj.func, Way0 ret =  "<<PyLong_AsLong(ret0)<<std::endl;

    
    //Return value: New reference.
    PyObject * func_name = Py_BuildValue("s","test_add");//
    PyObject * arg1 = Py_BuildValue("l",3);
    PyObject * arg2 = Py_BuildValue("l",3);
    //Return value: New reference.
    PyObject * ret1 = PyObject_CallMethodObjArgs(p_cls_instance, func_name, arg1, arg2,  NULL);//test_add(self, 3, 3)
    PyErr_PrintEx(1);
    std::cout<<"call obj.func, Way1 ret =  "<<PyLong_AsLong(ret1)<<std::endl;
        
    



    Py_XDECREF(ret1);
    Py_XDECREF(arg2);
    Py_XDECREF(arg1);
    Py_XDECREF(func_name);
    Py_XDECREF(ret0);
    Py_XDECREF(obj_attr);
    Py_XDECREF(cls_attr);
    Py_XDECREF(p_cls_instance);
    Py_XDECREF(p_module);
    //Py_XDECREF(ret0);

    Py_Finalize();
#ifdef _WIN32 || _WIN64

    system("pause");

#elif __linux__ || __linux


#endif //    
	return 0;
}

效果：

以下的内容希望重点关注：
1. 其实这个看起来是很简单的，如果有一定的c++基础，整个流程就是导入模块，获取模块中的属性，这些属性包括函数，全局变量，类等等。
2. 操作全局的函数和变量
3. 这里最难的还是对于python class 的操作，第1点中获取的类属性是一个callable object，这个时候需要实例化class(这里调用了builtin的__new__，而不是class中重载的__new__)，然后需要手动调用__init__方法。到此，class的实例化完成
4. 通过api获取class的属性，就是class的变量
5. 通过api调用object method，传入obj和函数名，包括参数，直接调用即可，主要某些api要关注self这个参数。

调试常用手段

1. 如果Py_Finalize()调用时，崩溃了，请检查自己的Py_INCREF和Py_DECREF各个引用增减是否正确。
2. 如果某些python c api调用失败了，可以尝试使用 PyErr_PrintEx(1);

#PS：请尊重原创，不喜勿喷

#PS：要转载请注明出处，本人版权所有.

有问题请留言，看到后我会第一时间回复