Static Java classpath hacks

July 29, 2015 7 Comments

A few days ago I encountered a situation where I needed to incorporate a JAR file into Matlab’s static Java classpath. I came across several relevant hacks that I thought could be useful for others:

The documented approach
Fixing javaclasspath.txt in run-time
Preloading Java class files in javaclasspath.txt
Loading Java class files into the static classpath in run-time (!)
Analyzing loaded Java classes in run-time

The documented approach

In most use-cases, adding Java classes (or ZIP/JAR files) to the dynamic Java classpath is good enough. This can easily be done in run-time using the javaaddpath function. In certain cases, for example where the Java code uses asynchronous events, the Java files need to be added to the static, rather than the dynamic, Java classpath. In other cases, the Java code misbehaves when loaded using Matlab’s dynamic Java classloader, rather than the system classloader (which is used for the static classpath (some additional info). One example of this are the JAR/ZIP files used to connect to various databases (each database has its own Java JDBC connector, but they all need to reside in the static Java classpath to work properly).
Adding class-file folders or ZIP/JAR files to the static Java classpath can be done in several manners: we can update the Matlab installation’s classpath.txt file, or (starting in R2012b) a user-prepared javaclasspath.txt file. Either of these files can be placed in the Matlab (or deployed application’s) startup folder, or the user’s Matlab preferences folder (prefdir). This is all documented.
There are several important drawbacks to this approach:

Both classpath.txt and javaclasspath.txt do not accept relative paths, only full path-names. In other words, we cannot specify ../../libs/mylib.jar but only C:\Yair\libs\mylib.jar or similar variants. We can use the $matlabroot, $arch, and $jre_home place-holder macros, but not user folders. We can easily do this in our personal computer where the full path is known, but this is difficult to ensure if we deploy the application onto other computers.
Matlab’s compiler only supports adding Java classes and JARs to the dynamic classpath, but not to the static one – we need to manually add a classpath.txt or javaclasspath.txt to the deployment folder, complete with the deployment target’s correct full path. Naturally this is problematic when we wish to grant the user the flexibility of installing the deployed application anywhere on their computer disk.
The javaclasspath.txt file is only supported on R2012b onward. The classpath.txt file is supported in both old and new releases, but (1) modifying it in Matlab’s installation folder may require system privileges, and (2) require update upon each and every Matlab reinstallation or upgrade. A classpath.txt file from one Matlab release will NOT be compatible with another Matlab release – if we try using an incorrect file Matlab will crash or hang. If you are not using just a single matlab release, this quickly turns into a maintenance nightmare.
The javaclasspath.txt file loads the user-specified classes after those loaded in Matlab’s pre-defined classpath.txt. If we edit The classpath.txt file, we can place our classes at the top of the file, but this again entails the maintenance nightmare described above

Luckily, there are a few hacks that can alleviate some of the pain when dealing with deployed applications that use static Java classes:

Fixing javaclasspath.txt in run-time

At the very top of our main function, we could fix (or create) the javaclasspath.txt file to include the current folder’s actual full path, and then relaunch the application and exit the current session. The newly-launched application will use this javaclasspath.txt file and everything should work well from that moment onward. Here’s a bare-bones implementation example:

function mainProgram()
   if ~exist('javaclasspath.txt','file')
      try
         fid = fopen('javaclasspath.txt', 'wt');
         fprintf(fid, [pwd '/classesFolder/']);  % add folder of *.class files
         fprintf(fid, [pwd '/subFolder/classes.jar']);  % add jar/zip file
         fclose(fid);
         system(['./' mfilename ' &']);
      catch err
         msg = sprintf('Could not create %s/javaclasspath.txt - error was: %s', pwd, err.message);
         uiwait(msgbox(msg, 'Error', 'warn'));  % need uiwait since otherwise exit below will delete the msgbox
      end
      exit
   end
   % If we reached this point, then everything is ok - proceed normally
   ...

A related solution is to recreate the classpath.txt file using the build-generation script, as described by Andrew Janke.

Preloading Java class files in javaclasspath.txt

Class folders and JAR/ZIP files can be loaded before those in classpath.txt, without having to modify the system’s classpath.txt, as exposed by Christopher Barber. All we need to do is to add a line containing the <before> string as a separate line in our javaclasspath.txt user-file. All entries beneath this line will be loaded into the front (rather than the end) of the static Java classpath. For example:

# This will be placed at the end of the static Java classpath
C:\placed\at\the\end\of\the\static\classpath\mylib1.jar
<before>
# This will be placed at the beginning of the static Java classpath
C:\placed\at\the\top\of\the\static\classpath\mylib2.jar

Loading Java class files into the static classpath in run-time (!)

The crown jewels in Java static classpath hacking belong without a doubt to power-users Andrew Janke and Amro. Based on Amro’s tip (based on a Java forum post from back in 2002), Andrew created a Matlab function that loads a Matlab class into the static classpath at run-time (i.e., without needing to modify/create either classpath.txt or javaclasspath.txt), slightly modified by me to include an optional classname input arg. Here’s the resulting javaaddpathstatic function:

function javaaddpathstatic(file, classname)
%JAVAADDPATHSTATIC Add an entry to the static classpath at run time
%
% javaaddpathstatic(file, classname)
%
% Adds the given file to the static classpath. This is in contrast to the
% regular javaaddpath, which adds a file to the dynamic classpath.
%
% Files added to the path will not show up in the output of
% javaclasspath(), but they will still actually be on there, and classes
% from it will be picked up.
%
% Caveats:
%  * This is a HACK and bound to be unsupported.
%  * You need to call this before attempting to reference any class in it,
%    or Matlab may "remember" that the symbols could not be resolved. Use
%    the optional classname input arg to let Matlab know this class exists.
%  * There is no way to remove the new path entry once it is added.
% Andrew Janke 20/3/2014 http://stackoverflow.com/questions/19625073/how-to-run-clojure-from-matlab/22524112#22524112
parms = javaArray('java.lang.Class', 1);
parms(1) = java.lang.Class.forName('java.net.URL');
loaderClass = java.lang.Class.forName('java.net.URLClassLoader');
addUrlMeth = loaderClass.getDeclaredMethod('addURL', parms);
addUrlMeth.setAccessible(1);
sysClassLoader = java.lang.ClassLoader.getSystemClassLoader();
argArray = javaArray('java.lang.Object', 1);
jFile = java.io.File(file);
argArray(1) = jFile.toURI().toURL();
addUrlMeth.invoke(sysClassLoader, argArray);
if nargin > 1
    % load the class into Matlab's memory (a no-args public constructor is expected for classname)
    eval(classname);
end

The usage is super simple: instead of using javaaddpath to load a class-files folder (or ZIP/JAR file) into Matlab’s dynamic Java classpath, we’d use javaaddpathstatic to load the same input into the static classpath – in runtime.
As Andrew notes, there are important caveats to using the new javaaddpathstatic function. One of the important caveats is that the class needs to be loaded before Matlab gets around to “discover” that it does not exist, because from then on even if we load the class into the static classpath, Matlab will report the class as missing. Therefore, we need to call javaaddpathstatic at the very top of the program (possibly even in our startup.m file). Moreover, if we directly call the class anywhere in the same m-file as our call to javaaddpathstatic, it will fail. The reason is that Matlab’s built-in interpreter processes the direct Java call when it pre-parses the file, before the javaaddpathstatic had a chance to run:

function this_Will_Fail()
   % mylib.jar contains the com.app.MyClass class
   javaaddpathstatic([pwd '/mylib.jar']);
   obj = com.app.MyClass;  % this will fail!

Instead, we need to prevent the interpreter from processing the class before run-time, by loading the class at run-time:

function this_Should_Work()
   % mylib.jar contains the com.app.MyClass class
   javaaddpathstatic([pwd '/mylib.jar']);
   obj = javaObject('com.app.MyClass');  % this should work ok

At least in one case, for a recent client project that required deploying a compiled Matlab application that used JDBC to connect to a database (and therefore relied on the static classpath), this saved the day for me.
I’m sad I wasn’t aware of this years ago, but now that I have posted it, we have less of an excuse to complain about static classpath files in Matlab. Hopefully MathWorks will incorporate this idea into Matlab so that we can more easily load classes into the static classpath in runtime, and remove the need for ugly deployment hacks (and yes, this includes MathWorks’ own toolboxes), hopefully removing the caveats listed above along the way.

Analyzing loaded Java classes in run-time

A related Matlab function, whereisjavaclassloadingfrom, was posted by Andrew Janke to report information about loaded classes, including whether they were loaded by the static (=standard Java) classloader, or the dynamic (=MathWorks’ custom) classloader, and from which location. For example:

% This is loaded into the static classpath as part of Matlab's standard classpath.txt file:
>> whereisjavaclassloadingfrom('java.util.HashMap')
Version: 8.6.0.232648 (R2015b)
Class:       java.util.HashMap
ClassLoader: sun.misc.Launcher$AppClassLoader@69365360
URL:         jar:file:/C:/Program%20Files/Matlab/R2015b/sys/java/jre/win64/jre/lib/rt.jar!/java/util/HashMap.class
 
% This is loaded into the dynamic classpath using MathWorks' custom classloader
>> javaaddpath('C:\Yair\Work\MultiClassTableModel')
>> whereisjavaclassloadingfrom('MultiClassTableModel')
Version: 8.6.0.232648 (R2015b)
Class:       MultiClassTableModel
ClassLoader: com.mathworks.jmi.CustomURLClassLoader@5270d6ad
URL:         file:/C:/Yair/Work/MultiClassTableModel/MultiClassTableModel.class
 
% This is an example of a class that is not loaded
>> whereisjavaclassloadingfrom('no.such.Class')
Error using javaArray
No class no.such.Class can be located on the Java class path
Error in whereisjavaclassloadingfrom (line 25)

Parting words

How risky is all this in terms of future compatibility? well, I’m not a prophet, but I’d say that using these hacks is pretty risky in this regard. Matlab’s custom classloader is way-deep in undocumented territory. This does not mean that the functionality will change tomorrow, but don’t be surprised if and when it does.
Have you ever used any additional undocumented hack relating to using the Java classpaths in Matlab? If so then please post a comment below.
Users who have endured my post this far, should probably also read my related article about Java class access pitfalls.

7 Responses

sebastian July 30, 2015 at 07:54 Reply
Specifying relative paths in classpath.txt may not be supported, but for me it works just fine in R2011b.

Simply add paths relative to the location of classpath.txt (i.e. relative to matlabs startup directory):
```
# matlab's original classpath entries here
# ...

# your own stuff
myjars/myjar1.jar
myjars/myjar2.jar
```
The corresponding entries in the javaclasspath within matlab are not expanded properly, but all jars are properly loaded.
Ed Yu July 30, 2015 at 09:52 Reply

Yair,

I would say with regard to loading classes through the java (root) classloader directly should not be that risky… As long as MATLAB is using Java, the root classloader should behave exactly the way it is designed. But you are right if you load classes through the MATLAB classloader, it is definitely prone to changes in the future. That’s why the utility whereisjavaclassloadingfrom from Andrew is wonderful, a necessity to tell what classloader a class comes from (in order to debug class loading issues).

Ed.
Krishnan February 4, 2016 at 14:17 Reply
Yair,

I think I have a method of adding and removing from static class path, but I am unable to make it stick on subsequent restarts of Matlab. The idea is to use java.lang.setProperty and java.lang.getProperty to set and edit the static class path.
1. Get the string from static class path and store it as a char
in_string=char(java.lang.getProperty('java.class.path');
in_string=char(java.lang.getProperty('java.class.path');
2. manipulate the string either through string concatenate (strcat), string replace (strrep)
3. put the string back into the java.class.path property
java.lang.setProperty('java.class.path',java.lang.String(in_string));
java.lang.setProperty('java.class.path',java.lang.String(in_string));
If you could help add the last bit where loaded class paths can be made permanent. This would be great.

On a secondary note thank you for sharing whereisclassloadingfrom from Andrew, it has saved so much time.

thanks
krishnan

Joonas T. Holmi April 23, 2020 at 09:58 Reply

Krishnan,
It is ‘java.lang.System.getProperty‘ and ‘java.lang.System.setProperty‘. I read that JVM uses ‘java.class.path’ content only at startup to create its SystemClassLoader and ignores it later on. I read that SystemClassLoader could be made to reinitialize by setting its sys_paths-field to null (https://stackoverflow.com/questions/15409223/adding-new-paths-for-native-libraries-at-runtime-in-java), but nothing happens unfortunately:

jC_CL = java.lang.Class.forName('java.lang.ClassLoader');
jF_sys_paths = jC_CL.getDeclaredField('sys_paths');
jF_sys_paths.setAccessible(1);
jF_sys_paths.set([], []); % Tell SCL to reinitialize on the next class load

Joonas T. Holmi April 24, 2020 at 17:31 Reply

Yair and others,
I found yet another way to add files to Java’s class path. See the function java_class_path_add_before_static(file) below. Benefit of this new method is that the classes are dynamically set before MATLAB’s static ones and its classes will be found first. This allows me to, for instance, provide newer version of *.jar of same jar libraries MATLAB uses (i.e. commons-compress.jar). It worked in at least R2019b and R2011a in short test.

I also wrote few other useful functions, available at my MATLAB toolbox repository under free BSD license:
https://gitlab.com/jtholmi/wit_io/-/tree/develop/helper/java

% Add a file to custom URL ClassLoader, which is set as a new ancestor of
% Java's System ClassLoader. In effect, its classes are found first before
% those of System ClassLoader during the class search. For instance, some
% of the MATLAB's built-in jar files can now be updated to newer version
% without conflicts. This also tries to make the file classes available
% on MATLAB side without need for external calls like importing. The
% following website has inspired developing this code:
% http://undocumentedmatlab.com/articles/static-java-classpath-hacks
function java_class_path_add_before_static(file),
    persistent jUCL cURLs;
 
    % Error if file does not exist
    jF = java.io.File(file);
    if ~jF.exists(),
        error('%s (The system cannot find the file specified)', file);
    end
 
    % Construct URL
    jURL = jF.toURI().toURL();
    cURL = char(jURL.toString());
 
    % Add if not already added or exit
    if any(strcmp(cURLs, cURL)),
        return;
    end
    cURLs{end+1} = cURL;
 
    % Create new custom URL ClassLoader if missing
    if isempty(jUCL),
        args = javaArray('java.net.URL', 1);
        args(1) = jURL;
        jUCL = java.net.URLClassLoader(args, []); % Set its parent to Bootstrap ClassLoader
        % Change Application ClassLoader ancestor to it
        jC_CL = java.lang.Class.forName('java.lang.ClassLoader');
        jF_parent = jC_CL.getDeclaredField('parent');
        jF_parent.setAccessible(1); % Set field public
        root = java.lang.ClassLoader.getSystemClassLoader(); % Application ClassLoader
        while true, % Loop until the Bootstrap ClassLoader is found
            parent_new = jF_parent.get(root); % Parent of Application ClassLoader is Extension ClassLoader. % Parent of Extension ClassLoader may be Bootstrap ClassLoader.
            if isempty(parent_new), break; % Bootstrap ClassLoader is null
            else, root = parent_new; end
        end
        jF_parent.set(root, jUCL); % Make it the highest ancestor
    else, % Otherwise, append to custom URL ClassLoader
        jC_UCL = java.lang.Class.forName('java.net.URLClassLoader');
        params = javaArray('java.lang.Class', 1);
        params(1) = java.lang.Class.forName('java.net.URL');
        jM_addURL = jC_UCL.getDeclaredMethod('addURL', params);
        jM_addURL.setAccessible(1); % Set method public
        jM_addURL.invoke(jUCL, jURL); % Append file to the java class path
    end
 
    % Try to make file class content visible on MATLAB side without need for external calls
    [~, name, ext] = fileparts(file);
    if strcmpi(ext, '.jar'),
        jJF = java.util.jar.JarFile(file); % Benefits from java.io.RandomAccessFile
        ocu_jJF = onCleanup(@() jJF.close()); % Safe cleanup of the file reading
        entries = jJF.entries();
        while entries.hasMoreElements(),
            entry = entries.nextElement();
            entry_file = char(entry.getName());
            [entry_path, entry_name, entry_ext] = fileparts(entry_file);
            if strcmpi(entry_ext, '.class'),
                % Assume that entry_path describes the package route to the entry_name.class
                entry_classname = [strrep(entry_path, '/', '.') '.' entry_name]; % JAR file separator always '/'
                try,
                    java.lang.Class.forName(entry_classname, 1, jUCL); % Try to initialize class
                    break; % On success, stop loop.
                catch,
                    % Failed. Try next.
                end
            end
        end
    elseif strcmpi(ext, '.class'),
        % Assume that bare *.class file does not belong to a package
        classname = name;
        try,
            java.lang.Class.forName(classname, 1, jUCL); % Try to initialize class
        catch,
            % Failed.
        end
    end
end

% Add a file to custom URL ClassLoader, which is set as a new ancestor of % Java's System ClassLoader. In effect, its classes are found first before % those of System ClassLoader during the class search. For instance, some % of the MATLAB's built-in jar files can now be updated to newer version % without conflicts. This also tries to make the file classes available % on MATLAB side without need for external calls like importing. The % following website has inspired developing this code: % http://undocumentedmatlab.com/articles/static-java-classpath-hacks function java_class_path_add_before_static(file), persistent jUCL cURLs; % Error if file does not exist jF = java.io.File(file); if ~jF.exists(), error('%s (The system cannot find the file specified)', file); end % Construct URL jURL = jF.toURI().toURL(); cURL = char(jURL.toString()); % Add if not already added or exit if any(strcmp(cURLs, cURL)), return; end cURLs{end+1} = cURL; % Create new custom URL ClassLoader if missing if isempty(jUCL), args = javaArray('java.net.URL', 1); args(1) = jURL; jUCL = java.net.URLClassLoader(args, []); % Set its parent to Bootstrap ClassLoader % Change Application ClassLoader ancestor to it jC_CL = java.lang.Class.forName('java.lang.ClassLoader'); jF_parent = jC_CL.getDeclaredField('parent'); jF_parent.setAccessible(1); % Set field public root = java.lang.ClassLoader.getSystemClassLoader(); % Application ClassLoader while true, % Loop until the Bootstrap ClassLoader is found parent_new = jF_parent.get(root); % Parent of Application ClassLoader is Extension ClassLoader. % Parent of Extension ClassLoader may be Bootstrap ClassLoader. if isempty(parent_new), break; % Bootstrap ClassLoader is null else, root = parent_new; end end jF_parent.set(root, jUCL); % Make it the highest ancestor else, % Otherwise, append to custom URL ClassLoader jC_UCL = java.lang.Class.forName('java.net.URLClassLoader'); params = javaArray('java.lang.Class', 1); params(1) = java.lang.Class.forName('java.net.URL'); jM_addURL = jC_UCL.getDeclaredMethod('addURL', params); jM_addURL.setAccessible(1); % Set method public jM_addURL.invoke(jUCL, jURL); % Append file to the java class path end % Try to make file class content visible on MATLAB side without need for external calls [~, name, ext] = fileparts(file); if strcmpi(ext, '.jar'), jJF = java.util.jar.JarFile(file); % Benefits from java.io.RandomAccessFile ocu_jJF = onCleanup(@() jJF.close()); % Safe cleanup of the file reading entries = jJF.entries(); while entries.hasMoreElements(), entry = entries.nextElement(); entry_file = char(entry.getName()); [entry_path, entry_name, entry_ext] = fileparts(entry_file); if strcmpi(entry_ext, '.class'), % Assume that entry_path describes the package route to the entry_name.class entry_classname = [strrep(entry_path, '/', '.') '.' entry_name]; % JAR file separator always '/' try, java.lang.Class.forName(entry_classname, 1, jUCL); % Try to initialize class break; % On success, stop loop. catch, % Failed. Try next. end end end elseif strcmpi(ext, '.class'), % Assume that bare *.class file does not belong to a package classname = name; try, java.lang.Class.forName(classname, 1, jUCL); % Try to initialize class catch, % Failed. end end end

Yair Altman April 24, 2020 at 17:42 Reply

Thanks for sharing Joonas – this seems useful indeed.

Users who will use your script are probably savvy Java programmers, who don’t need a reminder of the dangers in subverting Matlab’s shipping Java classes with different ones… 🙂

Joonas T. Holmi April 28, 2020 at 13:46 Reply

Very good comment Yair! I posted the above function too hastily… I will have to apologize for claiming above that it allows updating MATLAB’s jar files without conflicts. I finally tested it more thoroughly myself and it eventually failed in errors. I was trying to use updated version of Apache’s commons-compress.jar in my toolbox without success. Perhaps my functions in GitLab can however help someone far more savvy Java programmer to solve i.e. this obsolete jar updating issue.

HTML tags such as <b> or <i> are accepted.
Wrap code fragments inside <pre lang="matlab"> tags, like this:

<pre lang="matlab">
a = magic(3);
disp(sum(a))
</pre>

I reserve the right to edit/delete comments (read the site policies).
Not all comments will be answered. You can always email me (altmany at gmail) for private consulting.

Click here to cancel reply.