Failed to get field handle to set library path 问题解决(jdk17 升级)--附 xgboost4j 源码分析

说明

1. @author blog.jellyfishmix.com / JellyfishMIX - github
2. LICENSE GPL-2.0

问题现象

应用程序启动失败，查看日志 grep -A 50 ‘Error creating bean’ catalina.out 发现报错: Failed to get field handle to set library path

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'machineLearningFilterRule': Invocation of init method failed; nested exception is java.lang.ExceptionInInitializerError
        at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:137)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:407)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1611)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:481)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
        ... 104 common frames omitted
Caused by: java.lang.ExceptionInInitializerError: null
        at ml.dmlc.xgboost4j.java.Booster.init(Booster.java:473)
        at ml.dmlc.xgboost4j.java.Booster.(Booster.java:48)
        at ml.dmlc.xgboost4j.java.Booster.loadModel(Booster.java:86)
        at ml.dmlc.xgboost4j.java.XGBoost.loadModel(XGBoost.java:57)
        at com.qunar.flight.algo.service.ModelService.loadModel(ModelService.java:89)
        at com.qunar.flight.algo.service.ModelService.loadModel(ModelService.java:120)
        at com.qunar.flight.algo.impl.FlightRecallPredictorImpl.init(FlightRecallPredictorImpl.java:54)
        at com.qunar.flight.algo.impl.FlightRecallPredictorImpl.(FlightRecallPredictorImpl.java:45)
        at com.qunar.flight.itts.retrench.bean.rule.MachineLearningFilterRule.init(MachineLearningFilterRule.java:88)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:366)
        at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:311)
        at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:134)
        ... 113 common frames omitted
Caused by: java.lang.RuntimeException: java.io.IOException: Failed to get field handle to set library path
        at ml.dmlc.xgboost4j.java.XGBoostJNI.(XGBoostJNI.java:37)
        ... 129 common frames omitted
Caused by: java.io.IOException: Failed to get field handle to set library path
        at ml.dmlc.xgboost4j.java.NativeLibLoader.addNativeDir(NativeLibLoader.java:185)
        at ml.dmlc.xgboost4j.java.NativeLibLoader.smartLoad(NativeLibLoader.java:146)
        at ml.dmlc.xgboost4j.java.NativeLibLoader.initXGBoost(NativeLibLoader.java:40)
        at ml.dmlc.xgboost4j.java.XGBoostJNI.(XGBoostJNI.java:34)
        ... 129 common frames omitted
    
    (后面就没有异常栈了)

分析报错原因

异常栈末尾处的类 NativeLibLoader 是 xgboost4j 的 jar 包中的类，首先排查项目依赖

mvn dependency:tree

用如上命令跑一下 maven tree, 会发现依赖树里有 xgboost4j

老版本 xgboost4j 逻辑

xgboost4j-0.80.jar

1. xgboost4j 中有一个类叫 NativeLibLoader, 0.90 之前(例如 0.80)，有一个方法叫 addNativeDir，作用是向 ClassLoader 添加一个自定义路径。
2. 然后让 ClassLoader 去自定义的路径中加载类，如果加载失败则创建一个临时路径存放类文件，让 ClassLoader 去临时路径加载类。

static synchronized void initXGBoost() throws IOException {
  if (!initialized) {
    for (String libName : libNames) {
      smartLoad(libName);
    }
    initialized = true;
  }
}

  private static void smartLoad(String libName) throws IOException {
    addNativeDir(nativePath);
    try {
      // 让 ClassLoader 去自定义的路径中加载类
      System.loadLibrary(libName);
    } catch (UnsatisfiedLinkError e) {
      try {
        String libraryFromJar = nativeResourcePath + System.mapLibraryName(libName);
        // 生成一个临时文件
        loadLibraryFromJar(libraryFromJar);
      } catch (IOException ioe) {
        logger.error("failed to load library from both native path and jar");
        throw ioe;
      }
    }
  }

ml.dmlc.xgboost4j.java.NativeLibLoader

addNativeDir 方法具体逻辑:

1. 使用反射修改 ClassLoader 的 usr_paths 字段，把自定义的 libPath 加入 ClassLoader 的 usr_paths 字段值中。ClassLoader 的 usr_paths 字段数据结构是 String[], 表示多个 libPath
2. 把 usr_paths 字段之前的值 copy 到 tmp 中，先获取了一下 usr_paths 字段之前的值。
3. 给 tmp 添加一个自定义的 libPath, 使用反射给 usr_paths 设置新值 tmp
4. try catch 把 NoSuchFieldException 的异常栈吞掉了，换成了 IOException 继续网上抛，因此 catalina.out 中异常栈最后在 IOException 处就断掉了。

  private static void addNativeDir(String libPath) throws IOException {
    try {
      // 使用反射修改 ClassLoader 的 usr_paths 字段，把自定义的 libPath 加入 ClassLoader 的 usr_paths 字段值中
      // ClassLoader 的 usr_paths 字段数据结构是 String[], 表示多个 libPath
      Field field = ClassLoader.class.getDeclaredField("usr_paths");
      field.setAccessible(true);
      // 先获取了一下 usr_paths 字段之前的值
      String[] paths = (String[]) field.get(null);
      for (String path : paths) {
        if (libPath.equals(path)) {
          return;
        }
      }
      String[] tmp = new String[paths.length + 1];
      // 把 usr_paths 字段之前的值 copy 到 tmp 中
      System.arraycopy(paths, 0, tmp, 0, paths.length);
      // 给 tmp 添加一个自定义的 libPath
      tmp[paths.length] = libPath;
      // 使用反射给 usr_paths 设置新值 tmp
      field.set(null, tmp);
    } catch (IllegalAccessException e) {
      logger.error(e.getMessage());
      throw new IOException("Failed to get permissions to set library path");
    } catch (NoSuchFieldException e) {
      logger.error(e.getMessage());
      // 这里把 NoSuchFieldException 的异常栈吞掉了，换成了 IOException 继续网上抛，因此 catalina.out 中异常栈最后在 IOException 处就断掉了
      throw new IOException("Failed to get field handle to set library path");
    }
  }

老版本 xgboost4j 报错的直接原因

ClassLoader 的 usr_paths 字段在高版本 jdk(例如 jdk17) 中没了，因此 ClassLoader.class.getDeclaredField(“usr_paths”) 会抛异常 NoSuchFieldException, try catch 把 NoSuchFieldException 的异常栈吞掉了，换成了 IOException 继续网上抛，因此 catalina.out 中异常栈最后在 IOException 处就断掉了。

新版本 xgboost4j 逻辑的修复

xgboost4j-0.90.jar

ml.dmlc.xgboost4j.java.NativeLibLoader#initXGBoost

1. 新版本 xgboost4j 直接通过 loadLibraryFromJar 方法，创建一个临时路径存放类文件，让 ClassLoader 去临时路径加载类。不会再通过 ClassLoader.class.getDeclaredField(“usr_paths”) 尝试修改 ClassLoader 的指定字段的值。

  static synchronized void initXGBoost() throws IOException {
    if (!initialized) {
      for (String libName : libNames) {
        try {
          String libraryFromJar = nativeResourcePath + System.mapLibraryName(libName);
          loadLibraryFromJar(libraryFromJar);
        } catch (IOException ioe) {
          logger.error("failed to load " + libName + " library from jar");
          throw ioe;
        }
      }
      initialized = true;
    }
  }

解决方案

dependencyManagement 中指定高版本的 xgboost4j:

		<dependencyManagement>
			
            <dependency>
                <groupId>ml.dmlcgroupId>
                <artifactId>xgboost4jartifactId>
                <version>0.90version>
            dependency>
		dependencyManagement>