Get Even More Visitors To Your Blog, Upgrade To A Business Listing >>

Configuring third party JAR with native library in Apache Hama

Recently, I was faced with a task to make Apache Hama work with a third party Library (CPLEX – For optimization) with native source. Initially, I made it work in Eclipse but later I had to move to a distributed environment (Apache Hama). I got the following exception. This articles shows the solution to this exception.

java.lang.UnsatisfiedLinkError: ilog.cplex.Cplex.CPXopenCPLEX([I)J
attempt_201506192258_0001_000001_0: at ilog.cplex.Cplex.CPXopenCPLEX(Native Method)
attempt_201506192258_0001_000001_0: at ilog.cplex.CplexI.init(CplexI.java:6608)
attempt_201506192258_0001_000001_0: at ilog.cplex.CplexI.<init>(CplexI.java:629)

 

Scenario:

When you install CPLEX on your machine, it will put the cplex.jar and native library code in your installation directory.  For example, on my ubuntu machine the cplex.jar can be found at

/[cplex_installation_directory]/cplex/lib/cplex.jar

and native library code can be found at

/[cplex_installation_directory]/cplex/bin/x86-64_linux/*

CPLEX Native Library code

Note: Depending on your environment you might see different extension of shared libraries.
  • Windows -> .DLL
  • Unix -> .dylib
  • Linux -> .so

Including JAR with Native Library in Eclipse

The JAR file will not work if native library source code is not properly configured. To add the JAR in eclipse do the following steps

  1. Right click your project and select Properties
  2. Select Java Build Path option
  3. Select Add External JARs option from right side and select your cplex.jar file from the installation directory.
  4. Once the JAR is added, select the small dropdown on the left side of the JAR file
  5. Native Library Location will be set to none
  6. Select Native Library Location and press Edit button on the right side
  7. Set the folder of native library of cplex
  8. Hit Apply and then Ok
  9. You are ready to use CPLEX in eclipse (enjoy !)

Including Native Library through Terminal/CLI

If you have your program and want to run it through terminal or CLI with the third party JAR and native library then follow the steps below

  1. Build a fat jar with your the third party jar (with out the native library). Check me
  2. Run your command with the -Djava switch as follows

java -Djava.library.path=/path/to/shared/library/ -jar /path/to/test/program.jar

Including JAR with Native Library in Apache Hama environment

Setting up things in eclipse were pretty straight forward but in a distributed environment things become tricky. As we have seen in the above section, for a single process we can use -Djava.library.path option to include the source but Hama can run many processes and each process should have access to the shared library. So, in Hama you need to modify the hama-site.xml file located at

$HAMA_HOME/conf/hama-site.xml

Do the following steps to have a get a third party library work with Hama

1- Edit the hama-site.xml file and include the following <property> tag. You can set other properties also like -Xmx2048m which sets the RAM allocated for each process. This property bsp.child.java.opts declares all the options that you want to assign to each java child process.

<property>
    <name>bsp.child.java.opts</name>
    <value>-Djava.library.path=/path/to/shared/library/</value>
    <description>Java opts for the groom server child processe s.The following symbol, if present, will be interpolate d: @taskid@ is replaced by current TaskID. Any other oc currences of '@' will go unchanged. For example, to enable verbose gc logging to a file named for the taskid in /tmp and to set the heap maximum to be a gigabyte, pass a 'value' of:  -Xmx1024m -verbose:gc -Xloggc:/tmp/@[email protected] The configuration variable bsp.child.ulimit can be used to control the maximum virtual memory of the child processes.
     </description>
</property>

2- Place your cplex.jar in the $HAMA_HOME/lib/cplex.jar.

3- Restart your Hama and Hadoop.

 

Special thanks to Edward J. Yoon to telling me the solution to the above problem. The thread can be found at https://mail-archives.apache.org/mod_mbox/hama-user/201506.mbox/browser

 



This post first appeared on Techroz, please read the originial post: here

Share the post

Configuring third party JAR with native library in Apache Hama

×

Subscribe to Techroz

Get updates delivered right to your inbox!

Thank you for your subscription

×