在配一篇detection论文时,安装README文档进行配置,由于论文代码采用的mmdetection(2.X.X)版本中使用了mmpycocotools这个模块(也就是pycocotools外面再套了一层),我环境当中安装了mmdet 3.0.1运行起来会出现如下的错误:
(oln) jiang@jiang:~/model/object_localization_network/object_localization_network-main$ bash tools/dist_train.sh configs/oln_box/oln_box.py 1
config file is : configs/oln_box/oln_box.py
the number of gpus is : 1
the port is : 29500
Traceback (most recent call last):
File "tools/train.py", line 15, in <module>
from mmdet.apis import set_random_seed, train_detector
File "/home/jiang/model/object_localization_network/object_localization_network-main/mmdet/apis/__init__.py", line 1, in <module>
from .inference import (async_inference_detector, inference_detector,
File "/home/jiang/model/object_localization_network/object_localization_network-main/mmdet/apis/inference.py", line 11, in <module>
from mmdet.datasets import replace_ImageToTensor
File "/home/jiang/model/object_localization_network/object_localization_network-main/mmdet/datasets/__init__.py", line 2, in <module>
from .coco import CocoDataset
File "/home/jiang/model/object_localization_network/object_localization_network-main/mmdet/datasets/coco.py", line 21, in <module>
assert pycocotools.__version__ >= '12.0.2'
AttributeError: module 'pycocotools' has no attribute '__version__'
报错的源码如下(assert引起的版本校验错误):
try:
import pycocotools
if not hasattr(pycocotools, '__sphinx_mock__'): # for doc generation
assert pycocotools.__version__>='12.0.2'
except AssertionError:
raise AssertionError('Incompatible version of pycocotools is installed.'
'Run pip uninstall pycocotools first. Then run pip'
'install mmpycocotools to install open-mmlab forked'
'pycocotools.')
起初mmdet拥有自己的一个mmpycocotools工具包,功能和pycocotools类似。
最新版的mmdet框架已经遗弃了mmpycocotools这个包,也是采用通用的pycocotools。(emmm,话不多说)
现有的方案有两个:
下面来一一说明两种方案的结果。
将代码中出现pycocotools版本验证的源码进行注释,就不会出现这种问题,但是由于mmpycocotools是在pycocotools上层(外面套了一层),所以mmpycocotools有的方法pycocotools不一定有,就会存在差异。
这个时候再继续按照这个方案进行的话,就需要对代码中的大部分与coco有关的部分进行“大手术”。所以这个方法被我pass掉了。
采用pip进行在线安装,命令如下:
(oln) root@jiang:/home/jiang/model/object_localization_network/mmpycocotools-12.0.3# pip install mmpycocotools > /home/jiang/桌面/a.txt
error: subprocess-exited-with-error
× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [25 lines of output]
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-cpython-37
creating build/lib.linux-x86_64-cpython-37/pycocotools
copying pycocotools/__init__.py -> build/lib.linux-x86_64-cpython-37/pycocotools
copying pycocotools/mask.py -> build/lib.linux-x86_64-cpython-37/pycocotools
copying pycocotools/coco.py -> build/lib.linux-x86_64-cpython-37/pycocotools
copying pycocotools/cocoeval.py -> build/lib.linux-x86_64-cpython-37/pycocotools
running build_ext
Compiling pycocotools/_mask.pyx because it changed.
[1/1] Cythonizing pycocotools/_mask.pyx
/home/jiang/anaconda3/envs/oln/lib/python3.7/site-packages/Cython/Compiler/Main.py:381: FutureWarning: Cython directive 'language_level' not set, using '3str' for now (Py3). This has changed from earlier releases! File: /tmp/pip-install-a5br_mr2/mmpycocotools_0a6b2c42ae12414093ca041fe7591fa0/pycocotools/_mask.pyx
tree = Parsing.p_module(s, pxd, full_module_name)
building 'pycocotools._mask' extension
sources:['../common/maskApi.c', 'common/maskApi.c', 'pycocotools/_mask.c']
creating build/common
creating build/temp.linux-x86_64-cpython-37
creating build/temp.linux-x86_64-cpython-37/common
creating build/temp.linux-x86_64-cpython-37/pycocotools
build:{'build/temp.linux-x86_64-cpython-37/../common/maskApi.o': ('../common/maskApi.c', '.c'), 'build/temp.linux-x86_64-cpython-37/common/maskApi.o': ('common/maskApi.c', '.c'), 'build/temp.linux-x86_64-cpython-37/pycocotools/_mask.o': ('pycocotools/_mask.c', '.c')}
gcc -pthread -B /home/jiang/anaconda3/envs/oln/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/jiang/anaconda3/envs/oln/lib/python3.7/site-packages/numpy/core/include -Icommon -I/home/jiang/anaconda3/envs/oln/include/python3.7m -c ../common/maskApi.c -o build/temp.linux-x86_64-cpython-37/../common/maskApi.o
gcc: error: ../common/maskApi.c: 没有那个文件或目录
error: command '/usr/bin/gcc' failed with exit code 1
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for mmpycocotools
ERROR: Could not build wheels for mmpycocotools, which is required to install pyproject.toml-based projects
(oln) root@jiang:/home/jiang/model/object_localization_network/mmpycocotools-12.0.3#
就会出现一个 " gcc: error: …/common/maskApi.c: 没有那个文件或目录"
下载mmpycocotools源码进行源码编译安装:
执行python setup.py build出现如下错误:
(oln) root@jiang:/home/jiang/model/object_localization_network/mmpycocotools-12.0.3# python setup.py build
running build
running build_py
running build_ext
building 'pycocotools._mask' extension
sources:['../common/maskApi.c', 'common/maskApi.c', 'pycocotools/_mask.c']
build:{'build/temp.linux-x86_64-cpython-37/../common/maskApi.o': ('../common/maskApi.c', '.c'), 'build/temp.linux-x86_64-cpython-37/common/maskApi.o': ('common/maskApi.c', '.c'), 'build/temp.linux-x86_64-cpython-37/pycocotools/_mask.o': ('pycocotools/_mask.c', '.c')}
gcc -pthread -B /home/jiang/anaconda3/envs/oln/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/jiang/anaconda3/envs/oln/lib/python3.7/site-packages/numpy/core/include -Icommon -I/home/jiang/anaconda3/envs/oln/include/python3.7m -c ../common/maskApi.c -o build/temp.linux-x86_64-cpython-37/../common/maskApi.o
gcc: error: ../common/maskApi.c: 没有那个文件或目录
error: command '/usr/bin/gcc' failed with exit code 1
(oln) root@jiang:/home/jiang/model/object_localization_network/mmpycocotools-12.0.3#
这个错误跟pip在线安装一样
后来我就尝试着在mmpycocotools根目录上一层复制一份common文件夹,本以为能够编译成功,但是接着就出现了如下的错误:
(oln) root@jiang:/home/jiang/model/object_localization_network/mmpycocotools-12.0.3# python setup.py build
running build
running build_py
running build_ext
building 'pycocotools._mask' extension
sources:['../common/maskApi.c', 'common/maskApi.c', 'pycocotools/_mask.c']
build:{'build/temp.linux-x86_64-cpython-37/../common/maskApi.o': ('../common/maskApi.c', '.c'), 'build/temp.linux-x86_64-cpython-37/common/maskApi.o': ('common/maskApi.c', '.c'), 'build/temp.linux-x86_64-cpython-37/pycocotools/_mask.o': ('pycocotools/_mask.c', '.c')}
gcc -pthread -B /home/jiang/anaconda3/envs/oln/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/jiang/anaconda3/envs/oln/lib/python3.7/site-packages/numpy/core/include -Icommon -I/home/jiang/anaconda3/envs/oln/include/python3.7m -c ../common/maskApi.c -o build/temp.linux-x86_64-cpython-37/../common/maskApi.o
../common/maskApi.c: In function ‘rleDecode’:
../common/maskApi.c:48:4: warning: this ‘for’ clause does not guard... [-Wmisleading-indentation]
48 | for (k = 0; k < R[i].cnts[j]; k++) *(M++) = v; v = !v;
| ^~~
../common/maskApi.c:48:51: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the ‘for’
48 | for (k = 0; k < R[i].cnts[j]; k++) *(M++) = v; v = !v;
| ^
../common/maskApi.c: In function ‘rleFrPoly’:
../common/maskApi.c:171:2: warning: this ‘for’ clause does not guard... [-Wmisleading-indentation]
171 | for (j = 0; j < k; j++) x[j] = (int)(scale * xy[j * 2 + 0] + .5); x[k] = x[0];
| ^~~
gcc -pthread -B /home/jiang/anaconda3/envs/oln/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/jiang/anaconda3/envs/oln/lib/python3.7/site-packages/numpy/core/include -Icommon -I/home/jiang/anaconda3/envs/oln/include/python3.7m -c common/maskApi.c -o build/temp.linux-x86_64-cpython-37/common/maskApi.o
common/maskApi.c: In function ‘rleDecode’:
common/maskApi.c:48:4: warning: this ‘for’ clause does not guard... [-Wmisleading-indentation]
48 | for (k = 0; k < R[i].cnts[j]; k++) *(M++) = v; v = !v;
| ^~~
common/maskApi.c:48:51: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the ‘for’
235 | if (m > 2) x += (long)cnts[m - 2]; cnts[m++] = (uint)x;
| ^~~~
gcc -pthread -B /home/jiang/anaconda3/envs/oln/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/jiang/anaconda3/envs/oln/lib/python3.7/site-packages/numpy/core/include -Icommon -I/home/jiang/anaconda3/envs/oln/include/python3.7m -c pycocotools/_mask.c -o build/temp.linux-x86_64-cpython-37/pycocotools/_mask.o
In file included from /home/jiang/anaconda3/envs/oln/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1969,
from /home/jiang/anaconda3/envs/oln/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
from /home/jiang/anaconda3/envs/oln/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
from pycocotools/_mask.c:1111:
/home/jiang/anaconda3/envs/oln/lib/python3.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
17 | #warning "Using deprecated NumPy API, disable it with " \
| ^~~~~~~
gcc -pthread -shared -B /home/jiang/anaconda3/envs/oln/compiler_compat -L/home/jiang/anaconda3/envs/oln/lib -Wl,-rpath=/home/jiang/anaconda3/envs/oln/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-cpython-37/../common/maskApi.o build/temp.linux-x86_64-cpython-37/common/maskApi.o build/temp.linux-x86_64-cpython-37/pycocotools/_mask.o -o build/lib.linux-x86_64-cpython-37/pycocotools/_mask.cpython-37m-x86_64-linux-gnu.so
/home/jiang/anaconda3/envs/oln/compiler_compat/ld: build/temp.linux-x86_64-cpython-37/common/maskApi.o: in function `rleArea':
/home/jiang/model/object_localization_network/mmpycocotools-12.0.3/common/maskApi.c:76: multiple definition of `rleArea'; build/temp.linux-x86_64-cpython-build/temp.linux-x86_64-cpython-37/../common/maskApi.o:/home/jiang/model/object_localization_network/mmpycocotools-12.0.3/../common/maskApi.c:225: first defined here
collect2: error: ld returned 1 exit status
error: command '/usr/bin/gcc' failed with exit code 1
(oln) root@jiang:/home/jiang/model/object_localization_network/mmpycocotools-12.0.3#
大概意思就是c语言一个头文件编译了两次到同一个.o文件当中,在链接的时候会出现错误。
分析:
gcc命令看不懂的可以参考 这儿
gcc -pthread -B /home/jiang/anaconda3/envs/oln/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/jiang/anaconda3/envs/oln/lib/python3.7/site-packages/numpy/core/include -Icommon -I/home/jiang/anaconda3/envs/oln/include/python3.7m -c ../common/maskApi.c -o build/temp.linux-x86_64-cpython-37/../common/maskApi.o
gcc编译命令里面,编译的源代码(c文件)相对路径是 ‘-c …/common/maskApi.c’,代表gcc在读取命令的时候出现绝对路径映射错误。
mmpycocotools中setup.py源码如下:
import numpy as np
from setuptools import Extension, setup
# To compile and install locally run "python setup.py build_ext --inplace"
# To install library to Python site-packages run
# "python setup.py build_ext install"
# Note that the original compile flags below are GCC flags unsupported by
# the Visual C++ 2015 build tools.
# They can safely be removed.
ext_modules = [
Extension(
'pycocotools._mask',
sources=['common/maskApi.c', 'pycocotools/_mask.pyx'],
include_dirs=[np.get_include(), 'common'],
# extra_compile_args=['-Wno-cpp', '-Wno-unused-function', '-std=c99'],
extra_compile_args=[],
)
]
setup(name='mmpycocotools',
packages=['pycocotools'],
package_dir={'pycocotools': 'pycocotools'},
install_requires=[
'setuptools>=18.0', 'cython>=0.27.3', 'matplotlib>=2.1.0'
],
version='12.0.3',
ext_modules=ext_modules)
试过使用pdb工具进行调试,但是不太直观,很难发现问题所在
然后我就使用pycharm对这个模块建立项目,进行debug:
发现如下:
主要的问题出在:sources=[‘common/maskApi.c’, ‘pycocotools/_mask.pyx’],的处理上,记录如下:
问题出现地方:
经过步骤1图中【122】【123】行函数的处理之后,得到的新值如下:
中间添加了一个"…/common/maskApi.c"源码到sources当中
最后换了另外的一个conda虚拟环境进行编译就没有这个错误
在重新编译mmpycocotools源码时发现,两者调用的build_extension(self, ext)这个函数不同,编译失败找不到"…/common/maskApi.c"是由于其调用的是Cython里面的build_extension()函数,【Cython是对原生的disutils工具包二次封装了的】,而另外一个能编译通过的环境中没有安装Cython,直接调用disutils的build_extension()函数,调用位置如下:
这个问题主要虚拟环境中的Cython包所导致的;
当删除Cython包时就可以正常编译
Tips:
所使用的setuptools版本是61.2.0;
mmpycocotools版本为:12.0.3(好像mmpycocotools已经停止维护了)废了老大劲,唉!!!