[转帖]Handling native dependencies with Apache Ivy
转一个blog，关于如何使用ivy来处理native的依赖，对于有使用JNI开发的朋友应该很有价值。
原文blog地址：http://www.cooljeff.co.uk/2009/08/01/handling-native-dependencies-with-apache-ivy/
---------------------------------------------------------
Being able to handle native dependencies with Ivy has cropped up a couple of times with no best practise solution being available. This blog entry discusses the various proposals that have been put forward in order to provide a solution that:
- Works with Ivy Ant tasks in order to help populate the java.library.path.
- Works with IvyDE in order to populate the Native Library Location of a classpath entry.
- Is able to deal with different platforms.
- Is suitable for building enterprise repositories in order to download by platform.
It is important to remember that the concept of a platform constitutes a permutation of various components not limited to Operating System and Endianess. A good example would be the C libraries available from the NAG (Numeric Algorithms Group) which have distributions that take into account the compiler version to. Occasionally you even see JARs being distributed by platform (e.g. IBM’s Java MQ Series Client), which although is arguably a bad practise, does happen and Ivy needs to be able to handle these edge cases. Having said this, in this blog entry I will only take into account Windows/Linux and 32-bit/64-bit combinations for brevity.
The following proposals are explored (from least to most favourite):
- Using configurations to declare native artifacts.
- Using the extra attribute to declare a JNI path.
- Using types to declare native artifacts.
- Using types and a platform attribute to declare native artifacts.
Using configurations to declare native artifacts
This solution would look as follows:
< info organisation ="mit" module ="kerberos" revision ="1.7.0" />
< configurations >
< conf name ="ia32.linux" description ="32-bit linux native dependencies" />
< conf name ="ia64.linux" description ="64-bit linux native dependencies" />
< conf name ="ia32.windows" description ="32-bit windows native dependencies" />
</ configurations >
< publications >
< artifact name ="kerb5_lib" type ="native" ext ="so" conf ="ia32.linux" />
< artifact name ="kerb5_lib" type ="native" ext ="so" conf ="ia64.linux" />
< artifact name ="kerb5_lib" type ="native" ext ="dll" conf ="ia32.windows" />
</ publications >
</ ivy-module >
Here we are using configurations to declare artifacts by platform. Something depending on this module would then explicitly depend on a specific platform through a configuration.
This solution has the following issues:
- Switching between platforms requires updates to Ivy files.
- Switching between platforms is not really possible for a transitive dependency.
- IvyDE does not support filtering by configuration (only by types).
- Repository structuring information is now in the dependency hierarchy.
Switching between platforms is a big issue here because we are abusing what configurations are designed for. A configuration is a way of using a module, not for describing what an artifact actually is. This is why you don’t see configurations labeled: jars, javadocs or source. Whilst I have to admit that I’ve not tried implementing this solution, I cannot see how it can work when resolving native dependencies transitively. Perhaps some conf mapping trickery could be used.
The owner of a 3rd party Ivy module (i.e. one found in some repository) does not know about the environment you are going to be working in. Hence if that module itself has native dependencies, the owner of the module would not be able to say which configuration to depend on when writing the ivy module, even though it clearly has a native dependency.
Even for the end user who does have a little more control over the configurations they directly pull in, you would not be able to switch between environments without updating the configurations you depend on. Being able to switch between environments is a common use case. Many people develop on Windows and deploy on Linux. The Ivy module should remain identical for both environments because in most use cases, the logical dependency stays the same. What changes is the physical artifact.
Finally, many shared libraries have the same file name for both 32-bit and 64-bit distributions. This means that you are going to have to use the configuration as a directory name to structure your repository, which is not ideal because repository structuring information is now directly in the dependency hierarchy that a user will use.
Using the extra attribute to declare a JNI path
This solution would look as follows:
< info organisation ="mit" module ="kerberos" revision ="1.7.0"
extra:jni ="native/lib" />
</ ivy-module >
Here we are using the ability to add additional custom attributes to the info element in order to describe where the native libraries can then be found. An Ivy trigger would then need to be implemented by the Ant build infrastructure to detect module resolution in order to pick up the extra attribute and populate the java.library.path correctly.
This was proposed by Arthur Branham on a comment to his JIRA requesting Ivy to support native library path construction [IVY-600]. This is something that I’ve actually seen implemented as a quick solution to get over native library dependency issues when running integration tests as part of a build using Ant.
This solution has the following issues:
- Resolving for different platforms cannot be done using the same Ivy file.
- Resolving for a specific platform against a remote repository is not possible.
- IvyDE does not have support for extra attributes.
In order to support multiple platform resolution, you need to allow the attribute to have tokens that the resolver is capable of substituting. Even then, whilst this works for a local file system, it does not help when dealing with an external repository on the web because Ivy will need to download the artifacts into the cache for local use. With this solution there are no native artifacts, the repository structure is instead pushed into the ivy module file which ideally should remain in the ivy settings file.
Using types to declare native artifacts
This solution would look as follows:
< info organisation ="mit" module ="kerberos" revision ="1.7.0" />
< configurations >
< conf name ="runtime" description ="Core runtime dependencies" />
</ configurations >
< publications >
< artifact name ="kerb5_lib" type ="ia32.linux" ext ="so" conf ="runtime" />
< artifact name ="kerb5_lib" type ="ia64.linux" ext ="so" conf ="runtime" />
< artifact name ="kerb5_lib" type ="ia32.windows" ext ="dll" conf ="runtime" />
</ publications >
</ ivy-module >
Here we are using the type to provide the platform information. To switch between platforms, you simply need to filter by type when performing an Ivy Retrieve or Ivy Cache Path to match the platform you wish to resolve by.
This solution has the following clear advantages:
- The supported platforms are clearly stated in the Ivy module descriptor.
- Repository structuring has not leaked into the dependency information.
- Types fit in well with IvyDE which already uses types for javadocs and source.
- Types are already used by default to segregate artifacts in a cache.
This solution has the following issue:
- The type can no longer be used for other platform specific artifacts.
The concern I have with this solution is that we are not really using the type for its intended usage. This can result in blocking the use of other artifacts which are associated to the same platform but not associated to the java.library.path.
Imagine if you have jar file which is platform specific but still goes onto the classpath. As mentioned in the introduction, distributions of the IBM Java MQ Series Client have slightly different jars for each platform. For this use case you would still expect the type to be jar, since this is the type you will filter on when populating the classpath using the cachepath ant task. Similarly take a repository that has scripts or executables available for download by platform, you would expect the types to be scripts or exe respectively.
If we were to take a step back, forget what we are trying to solve and use the type classification for its intended usage, we probably would have named the type for native library dependencies to go on the java.libary.path as type=”library”.
Using types and a platform attribute to declare native artifacts
This solution would look as follows:
< info organisation ="mit" module ="kerberos" revision ="1.7.0" />
< configurations >
< conf name ="runtime" description ="Core runtime dependencies" />
</ configurations >
< publications >
< artifact name ="kerb5_lib" type ="library" ext ="so" platform ="ia32.linux"
conf ="runtime" />
< artifact name ="kerb5_lib" type ="library" ext ="so" platform ="ia64.linux"
conf ="runtime" />
< artifact name ="kerb5_lib" type ="library" ext ="dll" platform ="ia32.windows"
conf ="runtime" />
</ publications >
</ ivy-module >
This solution has the same benefits as using the type alone to define the platform information, however it does not have the same disadvantage of blocking other platform specific artifacts that are unrelated to the java.library.path from using the same type name. To integrate the above solution into Ivy the following would need to be done:
- IvyDE would need to support a Native Library Type (default to library).
- IvyDE would need to support a Platform.
- IvyDE would need to populate the Native Library Location of a classpath entry.
- Ivy would need to support platform as a filter option in retrieve and cachepath.
- Ivy would need to allow the user to set the platform easily in the Ivy settings.
When populating the Native Library Location IvyDE will need to create a directory tree to all of the artifacts and then remove all duplicate paths.
Conclusion
Handling native library dependencies is a gap in Ivy that will begin to impact enterprise users who have some native dependencies. Whilst types alone could be used to solve the problem, this potentially blocks using Ivy to resolve other platform specific dependencies that are not associated with the java.libary.path. The neatest solution would be to use type=”library” to group artifacts that are intended for the java.libary.path, along with a new attribute called platform. This solution will allow repositories to be structured well with platform specific information and also allow clients to resolve dependencies easily, regardless of the environment they happen to be currently working in.
Resources
- Official Apache Ivy website
- JNI and Endorsed directories ivy user mail thread
- Add ability to construct a native library path based on dependencies JIRA