JSON-Matlab integration

March 26, 2014 18 Comments

I would like to once again welcome guest blogger Mark Mikofski. Mark has written here last year about JGIT-Matlab integration (p.s., in the recently-released R2014a, MathWorks added GIT support to Simulink projects, although for some reason not to Matlab projects). Today, Mark discusses how to integrate JSON with Matlab.

What is JSON

It’s often necessary to save objects and arrays to a file, for lots of reasons. In Matlab, you would just save the desired objects in a mat-file, but that’s a binary format that in general only Matlab can read. One reason you might want to cache objects is to pass to a non-Matlab API. True there are libraries to import mat-files (for example: JMatIO), but there are already accepted standards for passing objects such as XML and JSON (JavaScript Object Notation, http://json.org) that most APIs readily understand. Both of these methods attempt to serialize data into a text based format that limits the types of objects they can contain. Because sometimes the API is a human being who needs to read and edit the file, one of the JSON’s goals is to be “easy for humans to read and write”.
Here’s a sample of a JSON object:

{
    "students": ["Dick", "Jane"],
    "assignments": ["essay", "term paper"]
    "scores": {
        "essay": {"Dick": 86, "Jane": 88},
        "term paper":  {"Dick": 89, "Jane": 87}
    },
    "cool": {"Dick": true, "Jane": true},
    "misc": null
}

Many web services, such as Twitter, use JSON in their APIs, because another JSON goal is to be “easy for machines to parse and generate”. JSON is based on the JavaScript ECMA standard in which it is native, and is extremely well documented.

JSON Primer

The documentation on json.org’s main page includes several train-track style graphics that best explain JSON usage, but here is a short primer:

There are two main types of JSON objects, a JSONObject and a JSONArray.
A JSONObject is enclosed in braces and consists of a collection of unordered key-value pairs separated by a commas. Each key-value pair is separated by a colon, with the key given before the value. In the example above, “students” is a key and its value is the JSONArray [“Dick”, “Jane”]. Keys must be strings, as explained below.
A JSONArray is enclosed in brackets and is ordered array of valid JSON types.
Items in a JSONObject or JSONArray may be one of 7 types: string, number, JSONObject, JSONArray, true, false, or null.
Strings are always enclosed in double-quotes and may contain backslashed escapes such as newline (\n), tab (\t), double-quotes (\”), backslash (\\) etc.
Numbers can be integers, floats, and scientific notation.
JSON interprets true and false as booleans (or logicals), and null as nothing.

There are several options to use JSON in Matlab, depending on what JSON parsing interface you wish to use. Matlab can use external libraries such as Java, .NET and C that allow us to use different implementations of the standard, including the original Java-based implementation by Douglas Crockford himself, the primary author of JavaScript of which JSON is a subset. A full list of JSON implementations is given on the main website along with its documentation. Here is a short list of some libraries:

JSONlab (Matlab) (also on SourceForge)
org.json (Java). Note: this link is now defunct; here’s an archived link
Json.NET (.NET)
fastJSON (.NET)

JSONlab

The most popular and mature Matlab implementation is JSONlab, which was started in 2011 by a Qianqian Fang, a medical researcher at Massachusetts General Hospital. It is available on the Matlab File Exchange, on SourceForge and via a Subversion repository. The latest version is 1.0beta that includes BSON, a variant of JSON used in MongoDB, which compresses JSON data into a binary format. JSONlab is based in part on earlier JSON-Matlab implementations that are now deprecated: JSON Parser (2009) by Joel Feenstra, another JSON Parser (2011) by François Glineur, and Highly portable JSON-input parser (2009) by Nedialko.
JSONlab converts both strings and files given by filename directly into Matlab structures and vice-versa. For example:

>> loadjson('{"this": "that", "foo": [1,2,3], "bar": ["a", "b", "c"]}')
ans =
    this: 'that'
    foo: [1 2 3]
    bar: {'a'  'b'  'c'}
>> s = struct('this', {'a', 'b'}, 'that', {1,2})
s =
1x2 struct array with fields:
    this
    that
>> j = savejson(s)
j =
{
    "s": [
        {
            "this": "a",
            "that": 1
        },
        {
            "this": "b",
            "that": 2
        }
    ]
}

JSONlab will nest structures as necessary and translates the JSON types into the appropriate Matlab types and vice-versa. JSONlab is well documented, easy and fast. It is reliable because it is well-maintained, has been around for several years and has many users. It is open source and issues and contributions are welcomed.

org.json (aka JSON-Java)

This is the Java implementation of JSON by the creator JSON and JavaScript, Douglas Crockford. This version is very well documented and under active development, but since there are no releases and only one branch, it could be considered the development branch. There are several other popular Java implementations listed on the JSON.org website.
The JSON library itself is called json.jar or org.json.jar. The easiest way to use it in Matlab is to add it to your java class path and then use it as if you were calling Java code. I put together a gist of examples and links for your reference. In particular take a look at orgJSONdemo.m which has several examples of creating JSON objects and then using them. I addition, take a look at this CSSM newsgroup thread.
How to add the jar file to your java class path depends on your version of Matlab: starting in R2012b, Matlab switched from using classpath.txt in your Matlab workspace to using javaclasspath.txt, which is much easier (however, see here). Make a file in your Matlab workspace, usually /home/MATLAB or /home/documents/matlab (e.g. on windows 7: c:\users\you\documents\matlab), called “javaclasspath.txt” and put the full path to your jar file. For example if you put the jar file in a MATLAB/JSON/ folder, then put the following line (on windows 7) in javaclasspath.txt:
the path specified in the javaclasspath.txt file should be the path to the actual jar file, so assuming your jar is called json.jar then you would put this path in your file:

C:\Users\your-user-name\Documents\MATLAB\JSON\json.jar

You must restart Matlab before this change takes effect. An alternate method is to load the jar file dynamically to Matlab’s Java classpath, using javaaddpath(‘path\to\json.jar’).
All this is pretty straight-forward in Matlab R2013b and newer, which use Java 7. json.org was implemented in Java 7 as javax.json, and so can be used with Matlab R2013b and newer. Matlab versions R2013a and earlier use Java 6, so another Java implementation must be used. You can either download a pre-compiled JAR file from my dropbox, or compile the source files on your system as follows:

Confirm the version of Java used in Matlab by typing
>> version -java Java 1.6.0_17-b04 with Sun Microsystems Inc. Java HotSpot(TM) 64-Bit Server VM mixed mode
>> version -java Java 1.6.0_17-b04 with Sun Microsystems Inc. Java HotSpot(TM) 64-Bit Server VM mixed mode
Download and install the corresponding Java JDK from Oracle or OpenJDK
Obtain the source by cloning the repository or downloading a tar/zip-ball
If necessary, extract the source then browse to the “src” folder
Compile the source into a class files from the command line:
C:\> C:\Program Files\Java\jdk1.7.0_45\bin\javac -verbose -sourcepath org\json\ -source 1.6 -target 1.6 -d ..\bin\ org\json\*.java org\json\zip\*.java
C:\> C:\Program Files\Java\jdk1.7.0_45\bin\javac -verbose -sourcepath org\json\ -source 1.6 -target 1.6 -d ..\bin\ org\json\*.java org\json\zip\*.java
Browse the created “bin” folder
Archive the class files into a jar file:
C:\> C:\Program Files\Java\jdk1.7.0_45\bin\jar -cvf JSON.jar org\
C:\> C:\Program Files\Java\jdk1.7.0_45\bin\jar -cvf JSON.jar org\
Add the new jar file to the Matlab static Java classpath, which will depend on what version of Matlab you have:
1. Matlab <= 2012a:
  - Find the system classpath.txt file by typing:
    >> [matlabroot '\toolbox\local\classpath.txt'] C:\Program Files\MATLAB\R2012a\toolbox\local\classpath.txt
    >> [matlabroot '\toolbox\local\classpath.txt'] C:\Program Files\MATLAB\R2012a\toolbox\local\classpath.txt
  - Copy the classpath to your working folder and append the path the jar file to the end. Use hash marks to comment lines.
2. Matlab >= 2012b: create a new file called “javaclasspath.txt” in your working folder with the path of the new jar file in it. Use hash marks for comments.
Restart Matlab to enable the jar file within Matlab.

Using the Java library is the same as using any Java library in Matlab. One nice feature is that tabbing after typing “object.” gives a pop-up list of available properties and methods, including inherited ones. Here are some simple usage examples:

>> j1 = org.json.JSONObject('{"this": "that", "foo": [1,2,3], "bar": ["a", "b", "c"]}')
>> j2 = j1.put('hello', 'world')
j2 =
{"hello":"world", "foo":[1,2,3], "bar":["a","b","c"], "this":"that"}
>> w = j1.getString('hello')
ans =
world

Watch out: the output is actually a java.lang.String object; use char() to convert it into a Matlab string:

>> char(w)
ans =
world

There is no easy way to convert a JSONArray to an array, so you can either loop over it, or convert it to a string and parse it. Note that JSONArray indexes start from 0, as in Java (contrary to Matlab indexes, which start at 1)

>> f = j1.get('foo')
f =
[1,2,3]
>> for n = 1:f.length,  fa(n)=f.get(n-1);  end
>> fa
fa =
     1     2     3

Json.NET and fastJSON

These libraries require you to add add the path of the .NET dll to Matlab using NET.addAssembly. For fastJSON, since binaries are not available, you will also need to compile the .NET dll. I’ll leave it to the reader to investigate further, independently.

Java Mark Mikofski

18 Responses

Mark Mikofski April 29, 2014 at 22:15 Reply

FYI: scipy.io.loadmat can read a MAT file into Python.
Rune Juhl Jacobsen May 14, 2014 at 04:08 Reply

There’s also https://github.com/christianpanton/matlab-json . While I haven’t benchmarked it, it should be a lot faster than Matlab-only implementations.
Working with JSON & Matlab | Alejandro Clemotte September 6, 2014 at 02:10 Reply

[…] A good post talking about several option is this. […]
Mark Mikofski April 23, 2015 at 15:37 Reply

Just saw this matlab json client on github by christianpanton using libjson c library. there are prebuilt windows binaries in bin/

mark mikofski August 4, 2015 at 12:14 Reply

I wish I had put more detail into using Newtonsoft JSON.net. Here’s a Gist demo

% download JSON.net from https://github.com/JamesNK/Newtonsoft.Json/releases
% extract into MATLAB folder
% ====
 
% add assembly (assuming not registered in GAC)
asm = NET.addAssembly('%HOME%DocumentsMATLABJson70r1binNet35Newtonsoft.Json.dll')
 
% DESERIALIZING JSON OBJECT
% =========================
% using the following example:
% {"inputs":
%   {"module":
%     {"name": "solarpanel-435",
%      "nameplate": 435
%     }
%   },
%   {"request":
%     {"startdate": [2015,1,1,1,0,0],
%      "enddate": [2015,12,31,23,0,0]
%     }
%   }
% }
 
 
% get source using io
str_data = fileread('path/to/file.json')
 
% use Json.Convrters.DeserializeObject(str_data)
json_data = Newtonsoft.Json.JsonConvert.DeserializeObject(str_data)
 
% getting value by key
inputs = json_data.GetValue('inputs') % get keys
module = inputs.GetValue('module')
 
% getting value of Newtonsoft.Json.Linq.JValue
name = module.GetValue('name').Value() % return value of name
nameplate = module.GetValue('nameplate').Value()
 
% getting items in JArray
startdate(1) = inputs.GetValue('request').Item(0)
startdate(2) = inputs.GetValue('request').Item(1)
startdate(3) = inputs.GetValue('request').Item(2)
startdate(4) = inputs.GetValue('request').Item(3)
startdate(5) = inputs.GetValue('request').Item(4)
startdate(6) = inputs.GetValue('request').Item(5)
 
% prettyprint output
json_data.ToString()
 
% SERIALIZE LINQ JSON OBJECT
% ==========================
% create a LINQ JObject
json_obj = Newtonsoft.Json.Linq.JObject() % create object
 
% view methods
methodsview(json_obj) % view all methods for JObject
 
% add values, objects and arrays
json_obj.Add('this', Newtonsoft.Json.Linq.JValue('that'))
json_obj.Add('foo', Newtonsoft.Json.Linq.JArray([10,20,30]))
json_obj.Add('bar', Newtonsoft.Json.Linq.JArray({'x','y','z'}))
 
% prettyprint
json_obj.ToString() % pretty print
 
% serialize json object
json_str = json_obj.Add('foo', Newtonsoft.Json.Linq.JArray([10,20,30]))
fid = fopen('outputfile.json')
fwrite(fid,json_str)
fclose(fid)

آموزش متلب November 18, 2015 at 09:08 Reply

Very Nice
TNX.
ET December 11, 2015 at 08:25 Reply
I think these are semi undocumented in the latest Matlab versions (i.e. R2015a or R2015b) but they are much faster then most other solutions:
jsonString = matlab.internal.webservices.toJSON(s); matlab.internal.webservices.fromJSON(jsonString);
jsonString = matlab.internal.webservices.toJSON(s); matlab.internal.webservices.fromJSON(jsonString);

Brad Stiritz December 30, 2015 at 21:08 Reply
ET, I tried your code without success in R2015a :
>> matlab.internal.webservices.toJSON('abc') Undefined variable "matlab" or class "matlab.internal.webservices.toJSON".
>> matlab.internal.webservices.toJSON('abc') Undefined variable "matlab" or class "matlab.internal.webservices.toJSON".

Yair Altman December 31, 2015 at 00:38

@Brad – toJSON() and fromJSON() were added in R2015b

oro77 March 31, 2016 at 06:25 Reply
I am using the following command to create the JSON string.
jsonString = matlab.internal.webservices.toJSON(s);
jsonString = matlab.internal.webservices.toJSON(s);
It works but if my structure is using a real like below.
s = struct('this', {'a', 'b'}, 'that', {0.1, 2})
s = struct('this', {'a', 'b'}, 'that', {0.1, 2})
jsonString will return
[{"this":"a","that":0.10000000000000001},{"this":"b","that":2}]
[{"this":"a","that":0.10000000000000001},{"this":"b","that":2}]
Why is it 0.10000000000000001 instead of 0.1 ?
How can I correct this ?

Malcolm Lidierth March 31, 2016 at 08:12 Reply

For a long answer:
What Every Computer Scientist Should Know About Floating-Point Arithmetic, by David Goldberg, published in the March, 1991 issue of Computing Surveys. Copyright 1991, Association for Computing Machinery, Inc.

Short answer:
Just as 1/3 can not be exactly represented in decimal, 1/10 has no exact representation in binary.

The java.lang.Double class is probably underused by MATLABers: its toString() method returns a string representation of a double that uses the minimum number of digits required to exactly reproduce that double during deserialisation (i.e to get back to that member of the subset of numbers represented in IEEE double).

java.lang.Double uses the FloatingDecimal class

oro77 March 31, 2016 at 11:19

Thank you for your reply.
So the best way to keep 0.1 would be to store in my structure a string instead of a double value.
Yair Altman March 31, 2016 at 11:26

@oro77 – I would think that it would make more sense to simply round off the insignificant digits upon loading/displaying (in those rare cases where this difference in LSB makes any difference), rather than store string representations in the JSON payload.
oro77 March 31, 2016 at 11:45

@Yair Altman – Thank you for your advice. I was thinking about rounding off it but the user may want to edit the json file in a text editor and he may find awkward about the insignificant digits. It is hard to decide what is the best solution. Is there something wrong about storing string representations in the JSON payload ?
Malcolm Lidierth March 31, 2016 at 20:46
@oro77
The number of digits used depends on the JSon library and how it does the formatting. Sun/Oracle/openJDK provide a full spec for the java.lang.Double.toString() method. Remembering that MATLAB calls that method to get the text to display in the MATLAB Command Window for Java objects including Double, the following may be informative:
>> a = java.lang.Double('0.1') a = 0.1 >> b = java.lang.Double('0.10000000000000001') b = 0.1 >> a==b ans = 1 >> java.lang.Double(0.1)==java.lang.Double('0.1') ans = 1
>> a = java.lang.Double('0.1') a = 0.1 >> b = java.lang.Double('0.10000000000000001') b = 0.1 >> a==b ans = 1 >> java.lang.Double(0.1)==java.lang.Double('0.1') ans = 1

Oleg June 26, 2016 at 00:59 Reply

As of R2016a the core of matlab.internal.webservices.toJSON() and matlab.internal.webservices.fromJSON() are in mex, hence very fast.

Oliver Woodford September 19, 2016 at 01:44 Reply

Reading from a JSON still isn’t as fas as it could be. My json_read mex function is over 10x faster than fromJSON(fileread('filename.json')).

Mariusz Zaleski August 15, 2016 at 15:45 Reply

Hi,

Is there anything that would work in C# parsing JSON to MatLab native types? (MWCharArray, MWNumericalArray etc)

HTML tags such as <b> or <i> are accepted.
Wrap code fragments inside <pre lang="matlab"> tags, like this:

<pre lang="matlab">
a = magic(3);
disp(sum(a))
</pre>

I reserve the right to edit/delete comments (read the site policies).
Not all comments will be answered. You can always email me (altmany at gmail) for private consulting.

Click here to cancel reply.