Undocumented Matlab
  • SERVICES
    • Consulting
    • Development
    • Training
    • Gallery
    • Testimonials
  • PRODUCTS
    • IQML: IQFeed-Matlab connector
    • IB-Matlab: InteractiveBrokers-Matlab connector
    • EODML: EODHistoricalData-Matlab connector
    • Webinars
  • BOOKS
    • Secrets of MATLAB-Java Programming
    • Accelerating MATLAB Performance
    • MATLAB Succinctly
  • ARTICLES
  • ABOUT
    • Policies
  • CONTACT
  • SERVICES
    • Consulting
    • Development
    • Training
    • Gallery
    • Testimonials
  • PRODUCTS
    • IQML: IQFeed-Matlab connector
    • IB-Matlab: InteractiveBrokers-Matlab connector
    • EODML: EODHistoricalData-Matlab connector
    • Webinars
  • BOOKS
    • Secrets of MATLAB-Java Programming
    • Accelerating MATLAB Performance
    • MATLAB Succinctly
  • ARTICLES
  • ABOUT
    • Policies
  • CONTACT

Inter-Matlab data transfer with memcached

August 27, 2014 2 Comments

Once again I welcome guest blogger Mark Mikofski. Mark has written here last year about JGIT-Matlab integration and earlier this year on JSON-Matlab integration. Today, Mark shows how to use the well-known open-source memcached library for data transfer between separate Matlab processes. Readers are also referred to Mark’s article on his blog. I have already written about various ways to communicate between separate Matlab processes – today’s article is an important complement to these methods.

Downloads

Stringpool's memcached tutorialThe memcached open-source library is great for sending objects from process to process. It is typically used to cache large data objects from databases or in a distributed environment, but we can also use it as a simple distributed shared memory system, which stores data in key-value pairs (as in a hashtable).
I was able to use memcached to set and retrieve objects from two different instances of Matlab. Both the Java and .NET variants of memcached can be used for this. memcached requires two components: server and client: The server comes precompiled, while the client needs to be compiled by us.
The memcached server was compiled by Trond Norbye from Couchbase (previously Membase and Northscale) for 32-bit and 64-bit Windows OS. See below how to run it as a service using the Python PyWin32 package.
I compiled Windows binaries of memcached-client libraries for both the Java and .NET using Java-1.7 and Visual Studio 2010 and zipped them up here.
Note: additional (non-memcached) shared-memory implementations used in Matlab include Joshua Dillon’s sharedmatrix (also see here), and Kevin Stone’s SharedMemory utilities, which use POSIX shared-memory and the Boost IPC library. Rice University’s TreadMarks library is another example of a shared-memory approach that has been used with Matlab, in the MATmarks package (note that the current availability of MATmarks is unclear).

.NET

We use the Memcached server version 1.4.5 patched for 64-bit Windows, precompiled by Trond Norbye from Couchbase (previously Membase and Northscale).
For the Memcached .NET client, download the Memcached.ClientLibrary port of com.meetup.memcached from SourceForge, and then compile using Microsoft Visual Studio 2010 C#, Release/AnyCPU configuration. Since the original project files were for MSVC-8, MSVC-10 starts a conversion wizard that first backed up the previous project files and then creates a new MSVC-10 project, including a new solution file. I then build the clientlib_2.0 project from the MSVC-10 IDE GUI using “build” from the menu, and voila: I got a new DLL.
Usage in Matlab is as follows:

  1. Start the memcached server:
    C:\> C:\path\to\memcached.exe -vv

    C:\> C:\path\to\memcached.exe -vv

  2. Do this on both MATLAB instances:
    asm = NET.addAssembly('C:\full\path\to\Memcached.ClientLibrary.dll');
    pool = Memcached.ClientLibrary.SockIOPool.GetInstance();
    pool.SetServers({'127.0.0.1:11211'});
    pool.Initialize;
    mc = Memcached.ClientLibrary.MemcachedClient();

    asm = NET.addAssembly('C:\full\path\to\Memcached.ClientLibrary.dll'); pool = Memcached.ClientLibrary.SockIOPool.GetInstance(); pool.SetServers({'127.0.0.1:11211'}); pool.Initialize; mc = Memcached.ClientLibrary.MemcachedClient();

  3. On MATLAB instance #1:
    mc.Set('name','mark')

    mc.Set('name','mark')

  4. On MATLAB instance #2:
    >> myName = mc.Get('name')
    myName =
    mark

    >> myName = mc.Get('name') myName = mark

Java

MATLAB and memcached also works perfectly with the original meetup.com Java version that the .NET version was ported from. The commands are exactly the same, but the Apache logger is not cleaned up for the non-bench/test sources (with isErrorEnabled(), etc. see jlog4 documentaion), so you always get this error message:

log4j:WARN No appenders could be found for logger (com.meetup.memcached.SockIOPool).
log4j:WARN Please initialize the log4j system properly.

Because there are log calls in the MemcachedClient and SockIOPool files that never have BasicConfiguration.configure() or set any jlog4. like the appender hence the error. Of course they never meant for log messages to be displayed when deployed, so wrapping with isErrorEnabled() like the .NET version does would be better.
The Java package is called com.meetup.memcached. We should build it ourselves from the Github source. The jar from the Maven repository didn’t work (attempting to start the sock I/O pool raised several Java errors due to missing includes), and differs a bit from the GitHub repo.

$ /c/Program\ Files/Java/jdk1.7.0_55/bin/javac.exe -verbose -classpath ../lib/log4j.jar -sourcepath com/meetup/memcached/ -source 1.6 -target 1.6 -d ../bin/ com/meetup/memcached/*.java
$ /c/Program\ Files/Java/jdk1.7.0_55/bin/jar.exe -cvf MeetupMemcached.jar com/

$ /c/Program\ Files/Java/jdk1.7.0_55/bin/javac.exe -verbose -classpath ../lib/log4j.jar -sourcepath com/meetup/memcached/ -source 1.6 -target 1.6 -d ../bin/ com/meetup/memcached/*.java $ /c/Program\ Files/Java/jdk1.7.0_55/bin/jar.exe -cvf MeetupMemcached.jar com/

To use in Matlab, first, start the memcached server, then start sock I/O pools and memcached clients on both Matlab instances:

>> javaaddpath('C:\full\path\to\MeetupMemcached.jar');
>> pool = com.meetup.memcached.SockIOPool.getInstance();
>> pool.setServers({'127.0.0.1:11211'});
>> pool.initialize;
>> mc = com.meetup.memcached.MemcachedClient;
>> mc.set('name','mark'); % on 1st instance/process
>> myName = mc.get('name') % on 2nd instance/process
myName =
mark

>> javaaddpath('C:\full\path\to\MeetupMemcached.jar'); >> pool = com.meetup.memcached.SockIOPool.getInstance(); >> pool.setServers({'127.0.0.1:11211'}); >> pool.initialize; >> mc = com.meetup.memcached.MemcachedClient; >> mc.set('name','mark'); % on 1st instance/process >> myName = mc.get('name') % on 2nd instance/process myName = mark

Other clients

  • xmemcached – Github:killme2008, Google Code project and Maven repo
  • The Xmemcached client works well. It is the most recently updated. There are jar files on the releases page of the GitHub repo. Here is an example from the 2.0.0 release from this April. The example is from the google.code wiki User Guide in English.

    >> javaaddpath('C:\full\path\to\xmemcached-2.0.0.jar');
    >> addr = net.rubyeye.xmemcached.utils.AddrUtil.getAddresses('localhost:11211')
    addr =
    [localhost/127.0.0.1:11211]
    >> builder = net.rubyeye.xmemcached.XMemcachedClientBuilder(addr)
    builder =
    net.rubyeye.xmemcached.XMemcachedClientBuilder@7ef6a26
    >> mc = builder.build()
    log4j:WARN No appenders could be found for logger (net.rubyeye.xmemcached.XMemcachedClient).
    log4j:WARN Please initialize the log4j system properly.
    mc =
    net.rubyeye.xmemcached.XMemcachedClient@275e6ce5
    >> mc.set('name',0,'mark')
    ans =
         1
    >> myName = mc.get('name')
    myName =
    mark

    >> javaaddpath('C:\full\path\to\xmemcached-2.0.0.jar'); >> addr = net.rubyeye.xmemcached.utils.AddrUtil.getAddresses('localhost:11211') addr = [localhost/127.0.0.1:11211] >> builder = net.rubyeye.xmemcached.XMemcachedClientBuilder(addr) builder = net.rubyeye.xmemcached.XMemcachedClientBuilder@7ef6a26 >> mc = builder.build() log4j:WARN No appenders could be found for logger (net.rubyeye.xmemcached.XMemcachedClient). log4j:WARN Please initialize the log4j system properly. mc = net.rubyeye.xmemcached.XMemcachedClient@275e6ce5 >> mc.set('name',0,'mark') ans = 1 >> myName = mc.get('name') myName = mark

  • Enyim – NuGet and Github:enyim
    I couldn’t get this assembly to load as built, I always got an “Strong Name Validation Failed” error, perhaps because I am on a 64-bit machine, but using MSVC express.
  • spymemcached – Github:dustin, Google Code project and Maven repo
    I also couldn’t get this to work. Even though I could make an array of Java socket objects using InetSocket the connection was always refused even though the memcached server was operating on the specified port.

memcached server as Windows service

This Gist uses Python PyWin32 win32service to install, remove, configure, start and stop the memcached server.

Related posts:

  1. Serializing/deserializing Matlab data – Matlab does not provide a documented manner to serialize data into a byte stream, but we can do this with some undocumented functionality. ...
  2. Simulink Data Dictionary – Simulink contains undocumented public API for access to its data dictionary functionality. ...
  3. Accessing plot brushed data – Plot data brushing can be accessed programmatically using very simple pure-Matlab code...
  4. Sparse data math info – Matlab contains multiple libraries for handling sparse data. These can report very detailed internal info. ...
  5. Additional license data – Additional meta-data about installed toolboxes can be retrieved in Matlab. ...
  6. Controlling plot data-tips – Data-tips are an extremely useful plotting tool that can easily be controlled programmatically....
Mark Mikofski
Print Print
« Previous
Next »
2 Responses
  1. Mark Mikofski August 27, 2014 at 15:31 Reply

    In order to get rid of that message and output logging messages to a file configure the Apache.org log4j logger yourself by creating a FileAppender with the default layout and then configuring the BasicConfigurator.

    >> fileAppender_withDefaultLayout = org.apache.log4j.FileAppender(org.apache.log4j.PatternLayout,'memcached.log')
    fileAppender_withDefaultLayout =
    org.apache.log4j.FileAppender@60f585a2
    >> org.apache.log4j.BasicConfigurator.configure(fileAppender_withDefaultLayout)

    >> fileAppender_withDefaultLayout = org.apache.log4j.FileAppender(org.apache.log4j.PatternLayout,'memcached.log') fileAppender_withDefaultLayout = org.apache.log4j.FileAppender@60f585a2 >> org.apache.log4j.BasicConfigurator.configure(fileAppender_withDefaultLayout)

    To get rid of the logger message for xmemcached you will have to download SLF4J and add slf4j-api-X.Y.Z.jar, slf4j-simple-X.Y.Z.jar and slf4j-log4j-X.Y.Z.jar to your path first, where X.Y.Z is the SLF4J version, 1.7.7 as of 2014-08-27. Then configure log4j as above for com.meetup.memcached.

  2. Mark Mikofski August 27, 2014 at 16:23 Reply

    Turns out that the newer schooner/whalin client is even faster than the old whalin (meetup.com) client, and they say it’s faster than both spymemcached (from couchbase/membase) and xmemcached.

    The missing libraries to make this run are in the lib folder of the zip file for the Memcached-Java-Client-3.0.0 which are the release page of Greg Whalin’s Github Memcached-Java-Client repo.

    I was able to build this using jdk-7 by extracting the file, making a new bin folder at the same folder as lib and src, then navigating to src\main\java and executing the following:

    C:\> "C:\Program Files\Java\jdk1.7.0_55\bin\javac.exe" -verbose -cp ..\..\..\lib\commons-pool-1.5.6.jar;..\..\..\lib\slf4j-api-1.6.1.jar;..\..\..\lib\slf4j-simple-1.6.1.jar -sourcepath com\schooner\MemCached\ -source 1.6 -target 1.6 -d ..\..\..\bin com\whalin\MemCached\*.java com\schooner\MemCached\*.java com\schooner\MemCached\command\*.java
    

    Then back up to the new bin folder an executing this:

    c:\> "C:\Program Files\java\jdk1.7.0_55\bin\jar.exe" -cvf Memcached-Java-Client.jar com\
    

    It works almost exactly like the old meetup.com client, but the package is named “com.whalin.MemCached” instead. Also now you will also have to add “commons-pool-1.5.6.jar”, “slf4j-api-1.6.1.jar” and “slf4j-simple-1.6.1.jar” to your MATLAB Java classpath using javaaddpath().

Leave a Reply
HTML tags such as <b> or <i> are accepted.
Wrap code fragments inside <pre lang="matlab"> tags, like this:
<pre lang="matlab">
a = magic(3);
disp(sum(a))
</pre>
I reserve the right to edit/delete comments (read the site policies).
Not all comments will be answered. You can always email me (altmany at gmail) for private consulting.

Click here to cancel reply.

Useful links
  •  Email Yair Altman
  •  Subscribe to new posts (feed)
  •  Subscribe to new posts (reader)
  •  Subscribe to comments (feed)
 
Accelerating MATLAB Performance book
Recent Posts

Speeding-up builtin Matlab functions – part 3

Improving graphics interactivity

Interesting Matlab puzzle – analysis

Interesting Matlab puzzle

Undocumented plot marker types

Matlab toolstrip – part 9 (popup figures)

Matlab toolstrip – part 8 (galleries)

Matlab toolstrip – part 7 (selection controls)

Matlab toolstrip – part 6 (complex controls)

Matlab toolstrip – part 5 (icons)

Matlab toolstrip – part 4 (control customization)

Reverting axes controls in figure toolbar

Matlab toolstrip – part 3 (basic customization)

Matlab toolstrip – part 2 (ToolGroup App)

Matlab toolstrip – part 1

Categories
  • Desktop (45)
  • Figure window (59)
  • Guest bloggers (65)
  • GUI (165)
  • Handle graphics (84)
  • Hidden property (42)
  • Icons (15)
  • Java (174)
  • Listeners (22)
  • Memory (16)
  • Mex (13)
  • Presumed future risk (394)
    • High risk of breaking in future versions (100)
    • Low risk of breaking in future versions (160)
    • Medium risk of breaking in future versions (136)
  • Public presentation (6)
  • Semi-documented feature (10)
  • Semi-documented function (35)
  • Stock Matlab function (140)
  • Toolbox (10)
  • UI controls (52)
  • Uncategorized (13)
  • Undocumented feature (217)
  • Undocumented function (37)
Tags
ActiveX (6) AppDesigner (9) Callbacks (31) Compiler (10) Desktop (38) Donn Shull (10) Editor (8) Figure (19) FindJObj (27) GUI (141) GUIDE (8) Handle graphics (78) HG2 (34) Hidden property (51) HTML (26) Icons (9) Internal component (39) Java (178) JavaFrame (20) JIDE (19) JMI (8) Listener (17) Malcolm Lidierth (8) MCOS (11) Memory (13) Menubar (9) Mex (14) Optical illusion (11) Performance (78) Profiler (9) Pure Matlab (187) schema (7) schema.class (8) schema.prop (18) Semi-documented feature (6) Semi-documented function (33) Toolbar (14) Toolstrip (13) uicontrol (37) uifigure (8) UIInspect (12) uitools (20) Undocumented feature (187) Undocumented function (37) Undocumented property (20)
Recent Comments
  • Nicholas (3 days 19 hours ago): Hi Yair, Thanks for the reply. I am on Windows 10. I also forgot to mention that this all works wonderfully out of the editor. It only fails once compiled. So, yes, I have tried a...
  • Nicholas (3 days 19 hours ago): Hi Yair, Thanks for the reply. I am on Windows 10. I also forgot to mention that this all works wonderfully out of the editor. It only fails once compiled. So, yes, I have tried a...
  • Yair Altman (4 days 2 hours ago): Nicholas – yes, I used it in a compiled Windows app using R2022b (no update). You didn’t specify the Matlab code location that threw the error so I can’t help...
  • Nicholas (4 days 22 hours ago): Hi Yair, Have you attempted your displayWebPage utility (or the LightweightHelpPanel in general) within a compiled application? It appears to fail in apps derived from both R2022b...
  • João Neves (8 days 3 hours ago): I am on matlab 2021a, this still works: url = struct(struct(struct(struct(hF ig).Controller).PlatformHost). CEF).URL; but the html document is empty. Is there still a way to do...
  • Yair Altman (11 days 2 hours ago): Perhaps the class() function could assist you. Or maybe just wrap different access methods in a try-catch so that if one method fails you could access the data using another...
  • Jeroen Boschma (11 days 4 hours ago): Never mind, the new UI components have an HTML panel available. Works for me…
  • Alexandre (11 days 5 hours ago): Hi, Is there a way to test if data dictionnatry entry are signal, simulink parameters, variables … I need to access their value, but the access method depends on the data...
  • Nicholas (11 days 20 hours ago): In case anyone is looking for more info on the toolbar: I ran into some problems creating a toolbar with the lightweight panel. Previously, the Browser Panel had an addToolbar...
  • Jeroen Boschma (15 days 3 hours ago): I do not seem to get the scrollbars (horizontal…) working in Matlab 2020b. Snippets of init-code (all based on Yair’s snippets on this site) handles.text_explorer...
  • Yair Altman (43 days 5 hours ago): m_map is a mapping tool, not even created by MathWorks and not part of the basic Matlab system. I have no idea why you think that the customizations to the builtin bar function...
  • chengji chen (43 days 11 hours ago): Hi, I have tried the method, but it didn’t work. I plot figure by m_map toolbox, the xticklabel will add to the yticklabel at the left-down corner, so I want to move down...
  • Yair Altman (51 days 4 hours ago): @Alexander – this is correct. Matlab stopped including sqlite4java in R2021b (it was still included in 21a). You can download the open-source sqlite4java project from...
  • Alexander Eder (57 days 0 hours ago): Unfortunately Matlab stopped shipping sqlite4java starting with R2021(b?)
  • K (63 days 11 hours ago): Is there a way to programmatically manage which figure gets placed where? Let’s say I have 5 figures docked, and I split it into 2 x 1, I want to place 3 specific figures on the...
Contact us
Captcha image for Custom Contact Forms plugin. You must type the numbers shown in the image
Undocumented Matlab © 2009 - Yair Altman
This website and Octahedron Ltd. are not affiliated with The MathWorks Inc.; MATLAB® is a registered trademark of The MathWorks Inc.
Scroll to top