UE4 OpenCV+Dlib插件（支持Windows Android）人脸检测（一）

本篇是OpenCV插件的编写，使用

参考

https://github.com/Brandon-Wilson/OpenCV-Plugin
https://github.com/TianxingWu/OpenVHead
https://unreal.gg-labs.com/wiki-archives/ar-vr/integrating-opencv-into-unreal-engine-4
https://github.com/Batname/UE4OpenCV

相关配置

• opencv 3.4
• UE4.24
• 
• Android端的话需要申请权限代码，这里要注意申请前需要先检查一下是否已经具备权限，否则会报错，为例省事，直接申请了存储的写入，读取还有摄像机权限

OpenCV.cpp
//Request Android Permission
	TArray AndroidTotalPermissions = {
		TEXT("android.permission.CAMERA"),
       TEXT("android.permission.READ_EXTERNAL_STORAGE"),
       TEXT("android.permission.WRITE_EXTERNAL_STORAGE"),
       TEXT("android.permission.MOUNT_UNMOUNT_FILESYSTEMS"),
       };
	TArray AndroidNeedReqPermissions;
	for(int i = 0; i < AndroidTotalPermissions.Num(); i++)
	{
		if(!UAndroidPermissionFunctionLibrary::CheckPermission(AndroidTotalPermissions[i]))
		{
			AndroidNeedReqPermissions.Add(AndroidTotalPermissions[i]);
		}	
	}
	UAndroidPermissionFunctionLibrary::AcquirePermissions(AndroidNeedReqPermissions);

源码

源码
有可能会有遗漏，以下所有内容最终以源码中的结果为准

插件的目录结构

OpenCV SDK准备

1. UE4新建Blank插件或者ThirdPartyLibrary插件，我这里是直接Blank

2. 下载OpenCV SDK，放置Include library到指定目录中

   • OpenCV SDK 根据需要下载指定的版本，我这里下载的是3.4版本的WIndows和Android
   • 需要注意要修改include\opencv2\core\utility.hpp的关于Check代码，否则的话，编译会报错误

     // cofolict with UE4
     // #if defined(check)
     // #  warning Detected Apple 'check' macro definition, it can cause build conflicts. Please, include this header before any Apple headers.
     // #endif
     

       // cofolict with UE4
       // bool check() const;
     

   • Windows相对简单拷贝dll lib
     opencv\build\bin中的dll 还有opencv\build\x64\vc15\lib的lib拷贝到 Plugins\OpenCV\Library\Win64中 注意要将两个dll文件拷贝到项目根目录的Binaries/Win64中，否则打开Editor会崩溃
     
   • 拷贝opencv\build\include到Plugins\OpenCV\Include\Win64中
   • Android相对麻烦一些 将OpenCV-android-sdk\sdk\native\3rdparty拷贝到Plugins\OpenCV\Library\Android\3rdparty中
     对应的libs拷贝到对应的libs目录
     注意将liblibpng.a删除掉，会和UE4冲突
     
   • 上面jni目录中只保存libopenc_java3.so 具体看源码中的内容 src目录是后面例子使用的
   • OpenCV-android-sdk\sdk\native\jni\include拷贝到Plugins\OpenCV\Include\Android中，这里注意Windows和Android的.h文件是不同的

   这里插件需要创建对应的Library Include目录

.build.cs的配置

• 这里两个模块是用来将OpenCV的Mat数据转换成UTexture2D用的，没有需求可以不加

   if (Target.Platform == UnrealTargetPlatform.Android)
        {
        	//for jni call
    		PrivateDependencyModuleNames.Add("Launch");
        }

• 这是用来在执行Android JNI函数的，没有需求可以不加

   string OpenCVIncludePath = Path.Combine(ModuleDirectory, "../../Include");
        string OpenCVLibraryPath = Path.Combine(ModuleDirectory, "../../Library");
        
        System.Console.WriteLine("----Architecture -----" + Target.Architecture);
        
        //Load OpenCV Include

        if (Target.Platform == UnrealTargetPlatform.Win64)
        {
            OpenCVIncludePath = Path.Combine(OpenCVIncludePath, "Win64");
            
        }
        else if (Target.Platform == UnrealTargetPlatform.Android)
        {
            OpenCVIncludePath = Path.Combine(OpenCVIncludePath, "Android");
        }
        else if (Target.Platform == UnrealTargetPlatform.IOS)
        {
            OpenCVIncludePath = Path.Combine(OpenCVIncludePath, "IOS");
        }
        System.Console.WriteLine("----Include -----" + OpenCVIncludePath);
        PublicIncludePaths.AddRange(new string[] { OpenCVIncludePath });

• 根据不同的平台 将不同的Include目录添加到PublicIncludePaths中

//Load OpenCV Library
        string OpenCVFinalLibraryPath = "";
        if(Target.Platform == UnrealTargetPlatform.Win64)
        {
	        OpenCVFinalLibraryPath = Path.Combine(OpenCVLibraryPath, "Win64");
	        // Add Library Path 
	        PublicLibraryPaths.Add(OpenCVFinalLibraryPath);
	        
	        PublicAdditionalLibraries.Add(Path.Combine(OpenCVFinalLibraryPath, "opencv_world340.lib"));
	        
            //add dynamic library
            PublicDelayLoadDLLs.Add("opencv_ffmpeg340_64.dll");
            PublicDelayLoadDLLs.Add("opencv_world340.dll");
            RuntimeDependencies.Add(new RuntimeDependency(Path.Combine(OpenCVFinalLibraryPath, "opencv_ffmpeg340_64.dll"))); //for package
            RuntimeDependencies.Add(new RuntimeDependency(Path.Combine(OpenCVFinalLibraryPath, "opencv_world340.dll")));  //for package
            
            System.Console.WriteLine("----Library -----" + OpenCVFinalLibraryPath);
        }

• Windows平台 加载lib dll

else if(Target.Platform == UnrealTargetPlatform.Android)
        {
	        
            string[] AndroidTarget = { "arm64-v8a", "armeabi", "armeabi-v7a", "mips", "mips64", "x86", "x86_64"  };
            
            string[] ThirdLibName = { "libcpufeatures.a", "libIlmImf.a", "liblibjasper.a", "liblibjpeg.a", 
                "liblibprotobuf.a", "liblibtiff.a", "liblibwebp.a", "libtbb.a", "libtegra_hal.a"
            };
            string[] ThirdLibName_x86 = { "libcpufeatures.a", "libIlmImf.a", "libippicv.a", "libippiw.a", "libittnotify.a",
	            "liblibjasper.a", "liblibjpeg.a",  "liblibprotobuf.a", "liblibtiff.a", "liblibwebp.a", "libtbb.a"
            };
            string[] OpenCVLibName = { "libopencv_calib3d.a", "libopencv_core.a", "libopencv_flann.a", "libopencv_highgui.a",
                "libopencv_imgcodecs.a", "libopencv_imgproc.a", "libopencv_ml.a", "libopencv_objdetect.a",
                "libopencv_photo.a", "libopencv_shape.a", "libopencv_stitching.a", "libopencv_superres.a", "libopencv_video.a",
                "libopencv_videoio.a", "libopencv_videostab.a", "libopencv_dnn.a", "libopencv_features2d.a"
            };
            OpenCVFinalLibraryPath = Path.Combine(OpenCVLibraryPath, "Android");
            
            //add Third lib
            for (int i = 0; i < 3; i++)
            {
	            for (int j = 0; j < ThirdLibName.Length; j++)
	            {
		            PublicAdditionalLibraries.Add(Path.Combine(OpenCVFinalLibraryPath, "3rdparty\\libs", AndroidTarget[i], ThirdLibName[j]));
		            System.Console.WriteLine("----Android -----" + Path.Combine(OpenCVFinalLibraryPath, "3rdparty\\libs", AndroidTarget[i], ThirdLibName[j]));
	            }
            }
			
            for (int j = 0; j < ThirdLibName.Length - 1; j++)
            {
	            PublicAdditionalLibraries.Add(Path.Combine(OpenCVFinalLibraryPath, "3rdparty\\libs", "mips", ThirdLibName[j]));
	            System.Console.WriteLine("----Android -----" + Path.Combine(OpenCVFinalLibraryPath, "3rdparty\\libs", "mips", ThirdLibName[j]));
            }
            for (int j = 0; j < ThirdLibName.Length - 1; j++)
            {
	            PublicAdditionalLibraries.Add(Path.Combine(OpenCVFinalLibraryPath, "3rdparty\\libs", "mips64", ThirdLibName[j]));
	            System.Console.WriteLine("----Android -----" + Path.Combine(OpenCVFinalLibraryPath, "3rdparty\\libs", "mips64", ThirdLibName[j]));
            }
          
            for (int j = 0; j < ThirdLibName_x86.Length ; j++)
            {
	            PublicAdditionalLibraries.Add(Path.Combine(OpenCVFinalLibraryPath, "3rdparty\\libs", "x86", ThirdLibName_x86[j]));
	            System.Console.WriteLine("----Android -----" + Path.Combine(OpenCVFinalLibraryPath, "3rdparty\\libs", "x86", ThirdLibName_x86[j]));
            }
            for (int j = 0; j < ThirdLibName_x86.Length; j++)
            {
	            PublicAdditionalLibraries.Add(Path.Combine(OpenCVFinalLibraryPath, "3rdparty\\libs", "x86_64", ThirdLibName_x86[j]));
	            System.Console.WriteLine("----Android -----" + Path.Combine(OpenCVFinalLibraryPath, "3rdparty\\libs", "x86_64", ThirdLibName_x86[j]));
            }
			
            //add core lib
            for (int i = 0; i < AndroidTarget.Length; i++)
            {
               
                for (int k = 0; k < OpenCVLibName.Length; k++)
                {
                 PublicAdditionalLibraries.Add(Path.Combine(OpenCVFinalLibraryPath, "libs", AndroidTarget[i], OpenCVLibName[k]));
                 System.Console.WriteLine("----Android -----" + Path.Combine(OpenCVFinalLibraryPath, "libs", AndroidTarget[i], OpenCVLibName[k]));
                }
            }
            
            //load APL.xml
            string APLXmlPath = Utils.MakePathRelativeTo(ModuleDirectory, Target.RelativeEnginePath);
            AdditionalPropertiesForReceipt.Add("AndroidPlugin", Path.Combine(APLXmlPath, "OpenCV_APL.xml"));
            System.Console.WriteLine("----APLXmlPath -----" + Path.Combine(APLXmlPath, "OpenCV_APL.xml"));
        }
        else if(Target.Platform == UnrealTargetPlatform.IOS)
        {

        }

• Android平台 遍历加载所有的.a库，最后一步需要设置OpenCV_APL.xml 用来指定将so库拷贝到android中，以及Android编译的一些相关配置

• OpenCV_APL.xml

• prebuildCopies用来配置将我们自定义的so还有java文件添加到Android工程中

• androidManifestUpdates 用来添加相机权限到Android中，没有需求可以不加

• soLoadLibrary 添加so库

• 进入Library\Android\jni 创建Android.mk

LOCAL_PATH := $(call my-dir)

PATH_TO_LIBS = libs
 


include $(CLEAR_VARS)
LOCAL_MODULE := opencv_java3
LOCAL_SRC_FILES := $(TARGET_ARCH_ABI)/libopencv_java3.so
include $(PREBUILT_SHARED_LIBRARY)


include $(CLEAR_VARS)
LOCAL_MODULE := UE4
LOCAL_SRC_FILES := $(TARGET_ARCH_ABI)/libUE4.so
include $(PREBUILT_SHARED_LIBRARY)

• 添加libopencv_java3.so

模块加载

• 针对Windows平台，需要在StartupModule中加载dll文件 ShutdownModule 释放dll

void FOpenCVModule::StartupModule()
{
	// This code will execute after your module is loaded into memory; the exact timing is specified in the .uplugin file per-module
	const FString PluginDir = IPluginManager::Get().FindPlugin(TEXT("OpenCV"))->GetBaseDir();
	FString LibraryPath;
#if PLATFORM_WINDOWS
	LibraryPath = FPaths::Combine(*PluginDir, TEXT("Library/Win64/"));
	UE_LOG(LogTemp, Warning, TEXT("opencv world LibraryPath == %s"), *(LibraryPath + TEXT("opencv_world340.dll")));
	OpenCV_World_Handler = FPlatformProcess::GetDllHandle(*(LibraryPath + TEXT("opencv_world340.dll")));
	OpenCV_FFmpeg_Handler = FPlatformProcess::GetDllHandle(*(LibraryPath + TEXT("opencv_ffmpeg340_64.dll")));
	if (!OpenCV_World_Handler || !OpenCV_FFmpeg_Handler)
	{
		UE_LOG(LogTemp, Error, TEXT("Load OpenCV dll failed!"));
	}
#elif PLATFORM_ANDROID

#elif PLATFORM_IOS

#endif
}

void FOpenCVModule::ShutdownModule()
{
	// This function may be called during shutdown to clean up your module.  For modules that support dynamic reloading,
	// we call this function before unloading the module.
#if PLATFORM_WINDOWS
	FPlatformProcess::FreeDllHandle(OpenCV_World_Handler);
	OpenCV_World_Handler = nullptr;
	FPlatformProcess::FreeDllHandle(OpenCV_FFmpeg_Handler);
	OpenCV_FFmpeg_Handler = nullptr;
#elif PLATFORM_ANDROID

#elif PLATFORM_IOS

#endif
}

• 没有什么好解释的，就是一些路径的拼接，以上基本上就是全部的OpenCV插件的接入了，如果编译有问题的话，大概率是库的路径方面有问题，一定要注意拷贝dll文件到Binaries中。

下面是具体的使用例子，其中对摄像机的使用需要用到JNI与Java交互比较麻烦

OpenCVTestActor

使用OpenCV 加载图片

主要知识点:

• 涉及到在ProjectSetting中设置不被打包成Pack的文件夹

• Android端获得Content的绝对路径

extern const FString &GetFileBasePath();
	
	FString ProjectPath = GetFileBasePath();
	FString TestImgPath = FPaths::Combine(ProjectPath, FApp::GetProjectName(), TEXT("Content/TestRes/test.jpg"));
	IPlatformFile& PlatformFile = FPlatformFileManager::Get().GetPlatformFile();
	if(!PlatformFile.FileExists(*TestImgPath))
	{
		GEngine->AddOnScreenDebugMessage(-1, 10, FColor::Red, FString::Printf(TEXT("TestImgPath == %s not exist "), *TestImgPath));
	}
	else
	{
		GEngine->AddOnScreenDebugMessage(-1, 10, FColor::Green, FString::Printf(TEXT("TestImgPath == %s exist "), *TestImgPath));
	}

	std::string temp(TCHAR_TO_UTF8(*TestImgPath));

	Mat img = imread(temp, IMREAD_UNCHANGED);
	if(img.empty())
	{	
		GEngine->AddOnScreenDebugMessage(-1, 5, FColor::Red, TEXT("Opencv open img failed!"));
		return;
	}
	else
	{
		GEngine->AddOnScreenDebugMessage(-1, 5, FColor::Green, TEXT("Opencv open img Success!"));
	}

• 使用GetFileBasePath()获得存储卡上UE4Game 路径

WebcamReader

• Windows端可以直接使用OpenCV的 VideoCapture.open(id)来获取摄像机
• Android端就麻烦一些了，需要通过JNI调用Android 的Camera进行图像获取，然后回调会C++

这里大致说明一下Android的流程
主要函数在CameraFrame中
captureCamera() == 》Java startPreview() 配置Camera相关
Java onPreviewFrame()更新Data数据 ==> FrameProcessing() 将数据从Java端传递到C++中
GetFrame*( 将经过转换过格式还有翻转方向后的图像数据返回给WebcamReader,用来更新Texture2D内容

最终效果

PS:

• 错误 C2872 “ACCESS_MASK”：不是使用using namespace cv,会与windows中的头文件起冲突
• 错误C4668 没有将“_WIN32_WINNT_WIN10_TH2”定义为预处理器宏，用“0”替换“#if/#elif”
  .build.cs中添加 bEnableUndefinedIdentifierWarnings = false;