Downloads

The memcached open-source library is great for sending objects from process to process. It is typically used to cache large data objects from databases or in a distributed environment, but we can also use it as a simple distributed shared memory system, which stores data in key-value pairs (as in a hashtable).
I was able to use memcached to set and retrieve objects from two different instances of Matlab. Both the Java and .NET variants of memcached can be used for this. memcached requires two components: server and client: The server comes precompiled, while the client needs to be compiled by us.
The memcached server was compiled by Trond Norbye from Couchbase (previously Membase and Northscale) for 32-bit and 64-bit Windows OS. See below how to run it as a service using the Python PyWin32 package.
I compiled Windows binaries of memcached-client libraries for both the Java and .NET using Java-1.7 and Visual Studio 2010 and zipped them up here.
Note: additional (non-memcached) shared-memory implementations used in Matlab include Joshua Dillon’s sharedmatrix (also see here), and Kevin Stone’s SharedMemory utilities, which use POSIX shared-memory and the Boost IPC library. Rice University’s TreadMarks library is another example of a shared-memory approach that has been used with Matlab, in the MATmarks package (note that the current availability of MATmarks is unclear).

.NET

We use the Memcached server version 1.4.5 patched for 64-bit Windows, precompiled by Trond Norbye from Couchbase (previously Membase and Northscale).
For the Memcached .NET client, download the Memcached.ClientLibrary port of com.meetup.memcached from SourceForge, and then compile using Microsoft Visual Studio 2010 C#, Release/AnyCPU configuration. Since the original project files were for MSVC-8, MSVC-10 starts a conversion wizard that first backed up the previous project files and then creates a new MSVC-10 project, including a new solution file. I then build the clientlib_2.0 project from the MSVC-10 IDE GUI using “build” from the menu, and voila: I got a new DLL.
Usage in Matlab is as follows:

1. Start the memcached server:

   C:\> C:\path\to\memcached.exe -vv

2. Do this on both MATLAB instances:

   asm = NET.addAssembly('C:\full\path\to\Memcached.ClientLibrary.dll');
   pool = Memcached.ClientLibrary.SockIOPool.GetInstance();
   pool.SetServers({'127.0.0.1:11211'});
   pool.Initialize;
   mc = Memcached.ClientLibrary.MemcachedClient();
   

3. On MATLAB instance #1:

   mc.Set('name','mark')
   

4. On MATLAB instance #2:

   >> myName = mc.Get('name')
   myName =
   mark
   

Java

MATLAB and memcached also works perfectly with the original meetup.com Java version that the .NET version was ported from. The commands are exactly the same, but the Apache logger is not cleaned up for the non-bench/test sources (with isErrorEnabled(), etc. see jlog4 documentaion), so you always get this error message:

log4j:WARN No appenders could be found for logger (com.meetup.memcached.SockIOPool).
log4j:WARN Please initialize the log4j system properly.

Because there are log calls in the MemcachedClient and SockIOPool files that never have BasicConfiguration.configure() or set any jlog4. like the appender hence the error. Of course they never meant for log messages to be displayed when deployed, so wrapping with isErrorEnabled() like the .NET version does would be better.
The Java package is called com.meetup.memcached. We should build it ourselves from the Github source. The jar from the Maven repository didn’t work (attempting to start the sock I/O pool raised several Java errors due to missing includes), and differs a bit from the GitHub repo.

$ /c/Program\ Files/Java/jdk1.7.0_55/bin/javac.exe -verbose -classpath ../lib/log4j.jar -sourcepath com/meetup/memcached/ -source 1.6 -target 1.6 -d ../bin/ com/meetup/memcached/*.java
$ /c/Program\ Files/Java/jdk1.7.0_55/bin/jar.exe -cvf MeetupMemcached.jar com/

To use in Matlab, first, start the memcached server, then start sock I/O pools and memcached clients on both Matlab instances:

>> javaaddpath('C:\full\path\to\MeetupMemcached.jar');
>> pool = com.meetup.memcached.SockIOPool.getInstance();
>> pool.setServers({'127.0.0.1:11211'});
>> pool.initialize;
>> mc = com.meetup.memcached.MemcachedClient;
>> mc.set('name','mark'); % on 1st instance/process
>> myName = mc.get('name') % on 2nd instance/process
myName =
mark

Other clients

• xmemcached – Github:killme2008, Google Code project and Maven repo

The Xmemcached client works well. It is the most recently updated. There are jar files on the releases page of the GitHub repo. Here is an example from the 2.0.0 release from this April. The example is from the google.code wiki User Guide in English.

>> javaaddpath('C:\full\path\to\xmemcached-2.0.0.jar');
>> addr = net.rubyeye.xmemcached.utils.AddrUtil.getAddresses('localhost:11211')
addr =
[localhost/127.0.0.1:11211]
>> builder = net.rubyeye.xmemcached.XMemcachedClientBuilder(addr)
builder =
net.rubyeye.xmemcached.XMemcachedClientBuilder@7ef6a26
>> mc = builder.build()
log4j:WARN No appenders could be found for logger (net.rubyeye.xmemcached.XMemcachedClient).
log4j:WARN Please initialize the log4j system properly.
mc =
net.rubyeye.xmemcached.XMemcachedClient@275e6ce5
>> mc.set('name',0,'mark')
ans =
     1
>> myName = mc.get('name')
myName =
mark

• Enyim – NuGet and Github:enyim
  I couldn’t get this assembly to load as built, I always got an “Strong Name Validation Failed” error, perhaps because I am on a 64-bit machine, but using MSVC express.
• spymemcached – Github:dustin, Google Code project and Maven repo
  I also couldn’t get this to work. Even though I could make an array of Java socket objects using InetSocket the connection was always refused even though the memcached server was operating on the specified port.

memcached server as Windows service

This Gist uses Python PyWin32 win32service to install, remove, configure, start and stop the memcached server.

The post Inter-Matlab data transfer with memcached appeared first on Undocumented Matlab.

What is JSON

It’s often necessary to save objects and arrays to a file, for lots of reasons. In Matlab, you would just save the desired objects in a mat-file, but that’s a binary format that in general only Matlab can read. One reason you might want to cache objects is to pass to a non-Matlab API. True there are libraries to import mat-files (for example: JMatIO), but there are already accepted standards for passing objects such as XML and JSON (JavaScript Object Notation, http://json.org) that most APIs readily understand. Both of these methods attempt to serialize data into a text based format that limits the types of objects they can contain. Because sometimes the API is a human being who needs to read and edit the file, one of the JSON’s goals is to be “easy for humans to read and write”.
Here’s a sample of a JSON object:

{
    "students": ["Dick", "Jane"],
    "assignments": ["essay", "term paper"]
    "scores": {
        "essay": {"Dick": 86, "Jane": 88},
        "term paper":  {"Dick": 89, "Jane": 87}
    },
    "cool": {"Dick": true, "Jane": true},
    "misc": null
}

Many web services, such as Twitter, use JSON in their APIs, because another JSON goal is to be “easy for machines to parse and generate”. JSON is based on the JavaScript ECMA standard in which it is native, and is extremely well documented.

JSON Primer

The documentation on json.org’s main page includes several train-track style graphics that best explain JSON usage, but here is a short primer:

• There are two main types of JSON objects, a JSONObject and a JSONArray.
• A JSONObject is enclosed in braces and consists of a collection of unordered key-value pairs separated by a commas. Each key-value pair is separated by a colon, with the key given before the value. In the example above, “students” is a key and its value is the JSONArray [“Dick”, “Jane”]. Keys must be strings, as explained below.
• A JSONArray is enclosed in brackets and is ordered array of valid JSON types.
• Items in a JSONObject or JSONArray may be one of 7 types: string, number, JSONObject, JSONArray, true, false, or null.
• Strings are always enclosed in double-quotes and may contain backslashed escapes such as newline (\n), tab (\t), double-quotes (\”), backslash (\\) etc.
• Numbers can be integers, floats, and scientific notation.
• JSON interprets true and false as booleans (or logicals), and null as nothing.

There are several options to use JSON in Matlab, depending on what JSON parsing interface you wish to use. Matlab can use external libraries such as Java, .NET and C that allow us to use different implementations of the standard, including the original Java-based implementation by Douglas Crockford himself, the primary author of JavaScript of which JSON is a subset. A full list of JSON implementations is given on the main website along with its documentation. Here is a short list of some libraries:

• JSONlab (Matlab) (also on SourceForge)
• org.json (Java). Note: this link is now defunct; here’s an archived link
• Json.NET (.NET)
• fastJSON (.NET)

JSONlab

The most popular and mature Matlab implementation is JSONlab, which was started in 2011 by a Qianqian Fang, a medical researcher at Massachusetts General Hospital. It is available on the Matlab File Exchange, on SourceForge and via a Subversion repository. The latest version is 1.0beta that includes BSON, a variant of JSON used in MongoDB, which compresses JSON data into a binary format. JSONlab is based in part on earlier JSON-Matlab implementations that are now deprecated: JSON Parser (2009) by Joel Feenstra, another JSON Parser (2011) by François Glineur, and Highly portable JSON-input parser (2009) by Nedialko.
JSONlab converts both strings and files given by filename directly into Matlab structures and vice-versa. For example:

>> loadjson('{"this": "that", "foo": [1,2,3], "bar": ["a", "b", "c"]}')
ans =
    this: 'that'
    foo: [1 2 3]
    bar: {'a'  'b'  'c'}
>> s = struct('this', {'a', 'b'}, 'that', {1,2})
s =
1x2 struct array with fields:
    this
    that
>> j = savejson(s)
j =
{
    "s": [
        {
            "this": "a",
            "that": 1
        },
        {
            "this": "b",
            "that": 2
        }
    ]
}

JSONlab will nest structures as necessary and translates the JSON types into the appropriate Matlab types and vice-versa. JSONlab is well documented, easy and fast. It is reliable because it is well-maintained, has been around for several years and has many users. It is open source and issues and contributions are welcomed.

org.json (aka JSON-Java)

This is the Java implementation of JSON by the creator JSON and JavaScript, Douglas Crockford. This version is very well documented and under active development, but since there are no releases and only one branch, it could be considered the development branch. There are several other popular Java implementations listed on the JSON.org website.
The JSON library itself is called json.jar or org.json.jar. The easiest way to use it in Matlab is to add it to your java class path and then use it as if you were calling Java code. I put together a gist of examples and links for your reference. In particular take a look at orgJSONdemo.m which has several examples of creating JSON objects and then using them. I addition, take a look at this CSSM newsgroup thread.
How to add the jar file to your java class path depends on your version of Matlab: starting in R2012b, Matlab switched from using classpath.txt in your Matlab workspace to using javaclasspath.txt, which is much easier (however, see here). Make a file in your Matlab workspace, usually /home/MATLAB or /home/documents/matlab (e.g. on windows 7: c:\users\you\documents\matlab), called “javaclasspath.txt” and put the full path to your jar file. For example if you put the jar file in a MATLAB/JSON/ folder, then put the following line (on windows 7) in javaclasspath.txt:
the path specified in the javaclasspath.txt file should be the path to the actual jar file, so assuming your jar is called json.jar then you would put this path in your file:

C:\Users\your-user-name\Documents\MATLAB\JSON\json.jar

You must restart Matlab before this change takes effect. An alternate method is to load the jar file dynamically to Matlab’s Java classpath, using javaaddpath(‘path\to\json.jar’).
All this is pretty straight-forward in Matlab R2013b and newer, which use Java 7. json.org was implemented in Java 7 as javax.json, and so can be used with Matlab R2013b and newer. Matlab versions R2013a and earlier use Java 6, so another Java implementation must be used. You can either download a pre-compiled JAR file from my dropbox, or compile the source files on your system as follows:

1. Confirm the version of Java used in Matlab by typing

   >> version -java
   Java 1.6.0_17-b04 with Sun Microsystems Inc. Java HotSpot(TM) 64-Bit Server VM mixed mode
   

2. Download and install the corresponding Java JDK from Oracle or OpenJDK
3. Obtain the source by cloning the repository or downloading a tar/zip-ball
4. If necessary, extract the source then browse to the “src” folder
5. Compile the source into a class files from the command line:

   C:\> C:\Program Files\Java\jdk1.7.0_45\bin\javac -verbose -sourcepath org\json\ -source 1.6 -target 1.6 -d ..\bin\ org\json\*.java org\json\zip\*.java

6. Browse the created “bin” folder
7. Archive the class files into a jar file:

   C:\> C:\Program Files\Java\jdk1.7.0_45\bin\jar -cvf JSON.jar org\

8. Add the new jar file to the Matlab static Java classpath, which will depend on what version of Matlab you have:
   1. Matlab <= 2012a:
      • Find the system classpath.txt file by typing:

        >> [matlabroot '\toolbox\local\classpath.txt']
        C:\Program Files\MATLAB\R2012a\toolbox\local\classpath.txt
        

      • Copy the classpath to your working folder and append the path the jar file to the end. Use hash marks to comment lines.
         
   2. Matlab >= 2012b: create a new file called “javaclasspath.txt” in your working folder with the path of the new jar file in it. Use hash marks for comments.
       
9. Restart Matlab to enable the jar file within Matlab.

Using the Java library is the same as using any Java library in Matlab. One nice feature is that tabbing after typing “object.” gives a pop-up list of available properties and methods, including inherited ones. Here are some simple usage examples:

>> j1 = org.json.JSONObject('{"this": "that", "foo": [1,2,3], "bar": ["a", "b", "c"]}')
>> j2 = j1.put('hello', 'world')
j2 =
{"hello":"world", "foo":[1,2,3], "bar":["a","b","c"], "this":"that"}
>> w = j1.getString('hello')
ans =
world

Watch out: the output is actually a java.lang.String object; use char() to convert it into a Matlab string:

>> char(w)
ans =
world

There is no easy way to convert a JSONArray to an array, so you can either loop over it, or convert it to a string and parse it. Note that JSONArray indexes start from 0, as in Java (contrary to Matlab indexes, which start at 1)

>> f = j1.get('foo')
f =
[1,2,3]
>> for n = 1:f.length,  fa(n)=f.get(n-1);  end
>> fa
fa =
     1     2     3

Json.NET and fastJSON

These libraries require you to add add the path of the .NET dll to Matlab using NET.addAssembly. For fastJSON, since binaries are not available, you will also need to compile the .NET dll. I’ll leave it to the reader to investigate further, independently.

The post JSON-Matlab integration appeared first on Undocumented Matlab.

Introduction

Clone and Init

Clone and Init are both class methods of the Git-Porcelain API class and return a Command API class object, which is what nearly all of the Git-Porcelain methods do:

import org.eclipse.jgit.api.Git   % import the package
cloneCMD = Git.cloneRepository;   % return a CloneCommand object
URI = 'git://github.com/mikofski/JGit4Matlab.git';   % a Git repository
cloneCMD.setURI(URI)   % set the repository to clone
g = cloneCMD.call;   % clone the repository

or more compactly:

g = org.eclipse.jgit.api.Git.cloneRepository.setURI(URI).call;

The call method executes the command with the options and arguments set by the other methods. Calling the clone method also returns a Git instance. The full API for the CloneCommand class lists all the possible options and arguments for cloning. One option lets you set an optional progress monitor; there are some already included in JGit, but unfortunately, they use the carriage return (or ‘\r’) escape sequence which in Windows is the same as a newline (read here). So, I implemented a customized ProgressMonitor class for Matlab that uses backspace (or ‘\b’) instead.
Init is very similar to clone:

g = org.eclipse.jgit.api.Git.init.call;

Configuring author information

Before moving on, it is important to set the “author” and “email” fields in the Git config. JGit will use your computer’s info if you provide nothing. You probably want to set these values globally, so the place to put them is in your $HOME folder in the “.gitconfig” file.

file = java.io.File(fullfile(getenv('home'),'.gitconfig'));
gitconfig = org.eclipse.jgit.storage.file.FileBasedConfig(file, org.eclipse.jgit.util.FS.DETECTED);
gitconfig.setString('user',[], 'name','John Doe');       % place your name in here
gitconfig.setString('user',[], 'email','john@doe.com');  % place your email in here
gitconfig.save;

Hopefully your commits will all have the right name and email address in them from now on.

Typical JGit Workflow

Now that you have a Git repository, you will want to start working with it. The typical workflow is to create a feature branch in which to work. But first JGit needs a Git instance. If you saved it from cloning or initializing a new repo then good for you! But if you need to recreate it, just pass the constructor a File object that points to the .git folder inside your Git repo:

file = java.io.File(fullfile('path', 'to', 'repo', '.git'));
g = org.eclipse.jgit.api.Git.open(file);
g.checkout.setCreateBranch(true).setName('newbranch').call;

Now you are no longer on the “master” branch, which is usually reserved for buildable/deployable code. Start working. But before you do too much, commit your work to your branch:

g.add.addFilepattern('file1.m').addFilepattern('file2.m').call;
g.commit.setMessage('initial dump').call;

file = java.io.File(fullfile('path', 'to', 'repo', '.git'));
g = org.eclipse.jgit.api.Git.open(file);
r = g.getRepository;  % a Repository instance can be very useful
g.status.call;
g.log.call;
g.checkout.setName('master').call;
g.merge.include(r.resolve('newbranch')).call;
g.push.call;
g.pull.call;

Note: Repository.resolve() is one of the most powerful methods.
The JGit log window on the left can be gotten using glog – the standalone JGit command line log GUI (slightly similar to gitk) that can be downloaded as a binary for systems with /bin/sh from the eclipse site (same place as the jar download).
Some tricks can be gleaned by what little work I’ve done so far on the JGit4MATLAB wrapper on the Matlab File Exchange. I’ll be happy to answer whatever questions I can there, or comments here. I’m also happy to accept pull requests on Github, of course using JGit4MATLAB!

Authentication

You can only use an SSH paired keys to authenticate remote repositories, the keys must **not** have a pass-phrase, they must be in the openSSH format and they must be in $HOME/.ssh. PuTTYgen let’s you do all of this, so it’s really not an issue. Of course you can still use remotes that do not require authentication with no problems.
All commands that deal with remotes and that require authentication, e.g. CLONE, FETCH, PULL, PUSH, will only work

• with SSH
• with *no* passphrase
• with keys in the openSSH format
• with keys and known hosts in $HOME\.ssh

Obviously remotes that do not require authentication will work fine. e.g. public, read-only and local repositories.
SSH is very easy to set up:

1. Download puttygen (Intel x86). Disregard the “Intel x86” business, it doesn’t matter what processor you have or whether your os is 64-bit or 32-bit. Puttygen is a very mature well establish application that is used by many other applications, for example all of the TortoiseXXX scm clients use Putty, Plink and Pageant. Here is the actual download here (click to download then click to install): http://the.earth.li/~sgtatham/putty/latest/x86/puttygen.exe
2. Create a folder called .ssh in your %– USERPROFILE%. On Windows 7 this is C:\Users\ and on XP it is C:\Documents and Settings\. You can do this in Matlab: mkdir(fullfile(getenv('userprofile'), '.ssh')).
3. Create a key using puttygen, this is fairly self explanatory, and kind of fun, but do *not* set a pass-phrase; leave those fields blank!
4. Using puttygen, find the conversion menu tab and export the key in the openSSH format to the .ssh folder that you created earlier.
5. Now copy and paste your public key to your remote repositories (Github, Bitbucket, etc.) as needed. Jsch, which is the ssh client that JGit uses, should now work out of the box.

Unfortunately using SSL (HTTPS) isn’t possible with JGit4MATLAB. Sorry. but hopefully you’ll love the excitement of using ssh key pairs!

Further development

Possible future work on the Matlab-JGit integration includes assigning standard keyboard-shortcuts to common source-control tasks, integration as toolbar/toolstrip icons for easier access, and in the Matlab Editor in general. Only a few days ago, a reader on this blog has shown interest in this.
In the far future, integration with Matlab’s existing source-control system. Anyone wish to pick up the glove?
JGit is covered by the BSD 3-clause license which is very permissive, and should allow MathWorks or anyone for that matter to develop an integrated JGit application for Matlab. There are more packages in JGit than just the porcelain API, including Swing utilities and other utilities that can be used to create rich user interfaces and experiences, but the development would have to be in Java. Developers should look at the developer’s section of the JGit website and use either the Maven or Github repository to set up their forks.

Conclusion

JGit and Matlab are a natural combination. JGit is a mature and thorough open-source project. Most porcelain commands are available without any additional work. There are some arcanae, but the API documentation is sufficient to figure out most things using short high-level code. If you find yourself diving too deep, then scan the docs some more, and you will probably find a shortcut.

The post JGit-Matlab integration appeared first on Undocumented Matlab.