Semi-documented feature – Undocumented Matlab https://undocumentedmatlab.com Charting Matlab's unsupported hidden underbelly Fri, 20 Oct 2017 09:57:44 +0000 en-US hourly 1 https://wordpress.org/?v=4.4.1 Parsing XML stringshttps://undocumentedmatlab.com/blog/parsing-xml-strings https://undocumentedmatlab.com/blog/parsing-xml-strings#comments Wed, 01 Feb 2017 09:52:45 +0000 http://undocumentedmatlab.com/?p=6838
 
Related posts:
  1. Undocumented XML functionality Matlab's built-in XML-processing functions have several undocumented features that can be used by Java-savvy users...
  2. Matlab-Java memory leaks, performance Internal fields of Java objects may leak memory - this article explains how to avoid this without sacrificing performance. ...
  3. Types of undocumented Matlab aspects This article lists the different types of undocumented/unsupported/hidden aspects in Matlab...
  4. Pause for the better Java's thread sleep() function is much more accurate than Matlab's pause() function. ...
 
]]>
I have recently consulted in a project where data was provided in XML strings and needed to be parsed in Matlab memory in an efficient manner (in other words, as quickly as possible). Now granted, XML is rather inefficient in storing data (JSON would be much better for this, for example). But I had to work with the given situation, and that required processing the XML.

I basically had two main alternatives:

  • I could either create a dedicated string-parsing function that searches for a particular pattern within the XML string, or
  • I could use a standard XML-parsing library to create the XML model and then parse its nodes

The first alternative is quite error-prone, since it relies on the exact format of the data in the XML. Since the same data can be represented in multiple equivalent XML ways, making the string-parsing function robust as well as efficient would be challenging. I was lazy expedient, so I chose the second alternative.

Unfortunately, Matlab’s xmlread function only accepts input filenames (of *.xml files), it cannot directly parse XML strings. Yummy!

The obvious and simple solution is to simply write the XML string into a temporary *.xml file, read it with xmlread, and then delete the temp file:

% Store the XML data in a temp *.xml file
filename = [tempname '.xml'];
fid = fopen(filename,'Wt');
fwrite(fid,xmlString);
fclose(fid);
 
% Read the file into an XML model object
xmlTreeObject = xmlread(filename);
 
% Delete the temp file
delete(filename);
 
% Parse the XML model object
...

This works well and we could move on with our short lives. But cases such as this, where a built-in function seems to have a silly limitation, really fire up the investigative reporter in me. I decided to drill into xmlread to discover why it couldn’t parse XML strings directly in memory, without requiring costly file I/O. It turns out that xmlread accepts not just file names as input, but also Java object references (specifically, java.io.File, java.io.InputStream or org.xml.sax.InputSource). In fact, there are quite a few other inputs that we could use, to specify a validation parser etc. – I wrote about this briefly back in 2009 (along with other similar semi-documented input altermatives in xmlwrite and xslt).

In our case, we could simply send xmlread as input a java.io.StringBufferInputStream(xmlString) object (which is an instance of java.io.InputStream) or org.xml.sax.InputSource(java.io.StringReader(xmlString)):

% Read the xml string directly into an XML model object
inputObject = java.io.StringBufferInputStream(xmlString);                % alternative #1
inputObject = org.xml.sax.InputSource(java.io.StringReader(xmlString));  % alternative #2
 
xmlTreeObject = xmlread(inputObject);
 
% Parse the XML model object
...

If we don’t want to depend on undocumented functionality (which might break in some future release, although it has remained unchanged for at least the past decade), and in order to improve performance even further by passing xmlread‘s internal validity checks and processing, we can use xmlread‘s core functionality to parse our XML string directly. We can add a fallback to the standard (fully-documented) functionality, just in case something goes wrong (which is good practice whenever using any undocumented functionality):

try
    % The following avoids the need for file I/O:
    inputObject = java.io.StringBufferInputStream(xmlString);  % or: org.xml.sax.InputSource(java.io.StringReader(xmlString))
    try
        % Parse the input data directly using xmlread's core functionality
        parserFactory = javaMethod('newInstance','javax.xml.parsers.DocumentBuilderFactory');
        p = javaMethod('newDocumentBuilder',parserFactory);
        xmlTreeObject = p.parse(inputObject);
    catch
        % Use xmlread's semi-documented inputObject input feature
        xmlTreeObject = xmlread(inputObject);
    end
catch
    % Fallback to standard xmlread usage, using a temporary XML file:
 
    % Store the XML data in a temp *.xml file
    filename = [tempname '.xml'];
    fid = fopen(filename,'Wt');
    fwrite(fid,xmlString);
    fclose(fid);
 
    % Read the file into an XML model object
    xmlTreeObject = xmlread(filename);
 
    % Delete the temp file
    delete(filename);
end
 
% Parse the XML model object
...
]]>
https://undocumentedmatlab.com/blog/parsing-xml-strings/feed 4
Sending email/text messages from Matlabhttps://undocumentedmatlab.com/blog/sending-email-text-messages-from-matlab https://undocumentedmatlab.com/blog/sending-email-text-messages-from-matlab#comments Wed, 07 Dec 2016 21:24:03 +0000 http://undocumentedmatlab.com/?p=6765
 
Related posts:
  1. Types of undocumented Matlab aspects This article lists the different types of undocumented/unsupported/hidden aspects in Matlab...
  2. Legend ‘-DynamicLegend’ semi-documented feature The built-in Matlab legend function has a very useful semi-documented feature for automatic dynamic update, which is explained here....
  3. Undocumented XML functionality Matlab's built-in XML-processing functions have several undocumented features that can be used by Java-savvy users...
  4. Inactive Control Tooltips & Event Chaining Inactive Matlab uicontrols cannot normally display their tooltips. This article shows how to do this with a combination of undocumented Matlab and Java hacks....
 
]]>
In this day and age, applications are expected to communicate with users by sending email/text messages to alert them about applicative events (“IBM stock purchased @$99.99” or “House is on fire!”). Matlab has included the sendmail function to handle this for many years. Unfortunately, sendmail requires some tweaking to be useful on all but the most basic/insecure mail servers. Today’s post will hopefully fill the missing gaps.

None of the information I’ll present today is really new – it was all there already if you just knew what to search for online. But hopefully today’s post will concentrate all these loose ends in a single place, so it may have some value:

Using a secure mail server

All modern mail servers use end-to-end TLS/SSL encryption. The sendmail function needs extra configuration to handle such connections, since it is configured for a non-encrypted connection by default. Here’s the code that does this for gmail, using SMTP server smtp.gmail.com and default port #465 (for other SMTP servers, see here):

setpref('Internet', 'SMTP_Server',   'smtp.gmail.com');
setpref('Internet', 'SMTP_Username', username);
setpref('Internet', 'SMTP_Password', password);
 
props = java.lang.System.getProperties;
props.setProperty('mail.smtp.auth',                'true');  % Note: 'true' as a string, not a logical value!
props.setProperty('mail.smtp.starttls.enable',     'true');  % Note: 'true' as a string, not a logical value!
props.setProperty('mail.smtp.socketFactory.port',  '465');   % Note: '465'  as a string, not a numeric value!
props.setProperty('mail.smtp.socketFactory.class', 'javax.net.ssl.SSLSocketFactory');
 
sendmail(recipient, title, body, attachments);  % e.g., sendmail('recipient@gmail.com', 'Hello world', 'What a nice day!', 'C:\images\sun.jpg')

All this is not enough to enable Matlab to connect to gmail’s SMTP servers. In addition, we need to set the Google account to allow access from “less secure apps” (details, direct link). Without this, Google will not allow Matlab to relay emails. Other mail servers may require similar server-side account configurations to enable Matlab’s access.

Note: This code snippet uses a bit of Java as you can see. Under the hood, all networking code in Matlab relies on Java, and sendmail is no exception. For some reason that I don’t fully understand, MathWorks chose to label the feature of using sendmail with secure mail servers as a feature that relies on “undocumented commands” and is therefore not listed in sendmail‘s documentation. Considering the fact that all modern mail servers are secure, this seems to make sendmail rather useless without the undocumented extension. I assume that TMW are well aware of this, which is the reason they posted a partial documentation in the form of an official tech-support answer. I hope that one day MathWorks will incorporate it into sendmail as optional input args, so that using sendmail with secure servers would become fully documented and officially supported.

Emailing multiple recipients

To specify multiple email recipients, it is not enough to set sendmail‘s recipient input arg to a string with , or ; delimiters. Instead, we need to provide a cell array of individual recipient strings. For example:

sendmail({'recipient1@gmail.com','recipient2@gmail.com'}, 'Hello world', 'What a nice day!')

Note: this feature is actually fully documented in sendmail‘s doc-page, but for some reason I see that some users are not aware of it (to which it might be said: RTFM!).

Sending text messages

With modern smartphones, text (SMS) messages have become rather outdated, as most users get push notifications of incoming emails. Still, for some users text messages may still be a useful. To send such messages, all we need is to determine our mobile carrier’s email gateway for SMS messages, and send a simple text message to that email address. For example, to send a text message to T-Mobile number 123-456-7890 in the US, simply email the message to 1234567890@tmomail.net (details).

Ke Feng posted a nice Matlab File Exchange utility that wraps this messaging for a wide variety of US carriers.

User configuration panel

Many GUI programs contain configuration panels/tabs/windows. Enabling the user to set up their own email provider is a typical use-case for such a configuration. Naturally, you’d want your config panel not to display plain-text password, nor non-integer port numbers. You’d also want the user to be able to test the email connection.

Here’s a sample implementation for such a panel that I implemented for a recent project – I plan to discuss the implementation details of the password and port (spinner) controls in my next post, so stay tuned:

User configuration of emails in Matlab GUI (click to zoom-in)
User configuration of emails in Matlab GUI (click to zoom-in)

]]>
https://undocumentedmatlab.com/blog/sending-email-text-messages-from-matlab/feed 5
Online (web-based) Matlabhttps://undocumentedmatlab.com/blog/online-web-based-matlab https://undocumentedmatlab.com/blog/online-web-based-matlab#comments Mon, 04 Aug 2014 14:17:36 +0000 http://undocumentedmatlab.com/?p=4935
 
Related posts:
  1. cprintf – display formatted color text in the Command Window cprintf is a utility that utilized undocumented Matlab desktop functionalities to display color and underline-styled formatted text in the Command Window...
  2. Setting status-bar text The Matlab desktop and figure windows have a usable statusbar which can only be set using undocumented methods. This post shows how to set the status-bar text....
  3. Setting desktop tab completions The Matlab desktop's Command-Window tab-completion can be customized for user-defined functions...
  4. uisplittool & uitogglesplittool Matlab's undocumented uisplittool and uitogglesplittool are powerful controls that can easily be added to Matlab toolbars - this article explains how...
 
]]>
For many years I searched for a good solution for deploying Matlab applications in online web-pages. There used to be a solution over a decade ago but it was taken off the market. We could use the Java Builder to wrap our Matlab apps in a JAR that could be called by a Java-based application server, but we’d still need to develop the front-end web GUI as well as the middle-tier plumbing. A similar limitation exists if we wish to use the new Matlab Production Server (MSP). Developing this front-end and middle-tier is by no means a trivial exercise, as much as MathWorks presentations would like it to appear. Not to mention the fact that the Builder and the MSP are relatively costly (~$5-7K and ~$30K respectively).

I was thrilled to see the answer in the recent Matlab Computational Finance Conference, London. I presented at last year’s conference, so I was excited to review this year’s presentations when I came across a nugget in Kevin Shea’s keynote presentation on new Matlab developments. Note that there were two separate Matlab computational-finance events in 2014 – in London (June 24) and NY (April 9); the interesting piece is from London. Unlike the NY conference, the London conference proceedings do not include a video recording, only the slides (perhaps the video will be added to the proceedings page later, after all it takes some time to process). The last slide of Kevin’s presentation shows a screenshot of a Chrome browser displaying what appears to be a full-fledged Matlab desktop (Workspace, Editor, Command Window, figures) at https://matlab.mathworks.com:

Matlab Online (click to zoom)
Matlab Online (click to zoom)

The benefits of using Matlab Online are potentially tremendous: excluding some limitations (e.g., limited GUI, run time and supported toolboxes), we could deploy our Matlab programs online with minimal to no additional dev effort.

Information on Matlab Online can be found at http://mathworks.com/products/matlab/online. It even includes detailed dedicated documentation for all the supported functions and features.

For some reason, MathWorks has chosen not to link this new (and in my humble opinion, great) product in its official list of products. Perhaps the reason is that MathWorks do not wish for Matlab Online to cannibalize its main offerings. This might also explain the limitations placed on the online version (some limitations are technical, the rest are business-related). Matlab Online requires a dedicated login access, provided by MathWorks. Unfortunately, it is still not fully released to the public and is only available to users with up to date student licenses.

Still, if you are an important enough client of MathWorks, you might ask for access. Even with its current limitations, Matlab Online may well be useful for your needs. If you press strongly enough, and if your account is large enough, perhaps MathWorks might even enable a dedicated subdomain for you (as in intel.mathworks.com or matlab.apple.com).

Here’s another screenshot, in case you’re not convinced yet:

Matlab Online (click to zoom)
Matlab Online (click to zoom)

A post by MathWorks on the Answers forum earlier this year lists Matlab Online along with some additional online-computing alternatives for Matlab.

And now for some guesswork/speculation: MathWorks placed a major bet on Java technology in the early 2000’s (actually the late 90’s, since R12 took a few years to develop). Java was indeed hot at that time, but then so were some other technologies over the years. Fortunately for MathWorks, Java proved agile, mature and portable enough to enable mobile and online porting. This could explain the lack of Simulink GUI support, since Simulink GUI is still C++-based to a large extent. It might also explain the extra work done in HG2 in the graphic infrastructure (previously C++-based). After all, the basic MCOS graphic classes were available years ago, so if HG2 was only about a transition from double-value handles to MCOS, and some nice beautifications (anti-aliasing, CVD-aware colors etc.) then HG2 could be released long ago. Working on the underlying engine to make it portable could well explain HG2’s belated arrival.

So what does all this mean about Matlab’s future? Well, it appears to me that MathWorks’ apparent move towards SaaS (software as a service) and cloud-based computing is slightly belated, but quite evidently follows a general industry trend. In my eyes it heralds a move by MathWorks from desktop to online services, perhaps even to pay-per-use computing (as by Techila for example). The Matlab desktop will still be MathWorks’ bread and butter for many years to come. But the ability of Matlab programs to work either locally (on a Matlab client, either thin or thick) or online would be an enormous productivity benefit, in essence being a “killer feature” over Matlab’s competitors.

Technically, Matlab’s online integration could enable closer integration of online content in Matlab programs (esp. GUI). I’ve already shown how active web content can be displayed in Matlab GUI, but a closer integration could mean this might all become fully documented and integrated (recall Windows’ failed Active Desktop on one hand, but smart-phones’ enormously-successful active widgets on the other). It could also enable closer integration with online graphing services such as Plotly.

I for one, can’t wait to see this dream being realized. It’s not too far down the road, it would seem. I just hope that licensing and cost considerations won’t keep us from using it. MathWork’s recent Home License scheme seems to indicate that this is well understood by MathWorks, so I am highly optimistic :-)

Then again, all this might be pure baseless speculation. Time will tell.

]]>
https://undocumentedmatlab.com/blog/online-web-based-matlab/feed 5
Improving fwrite performancehttps://undocumentedmatlab.com/blog/improving-fwrite-performance https://undocumentedmatlab.com/blog/improving-fwrite-performance#comments Wed, 24 Apr 2013 18:00:47 +0000 http://undocumentedmatlab.com/?p=3731
 
Related posts:
  1. Matrix processing performance Matrix operations performance is affected by internal subscriptions in a counter-intuitive way....
  2. Matlab-Java memory leaks, performance Internal fields of Java objects may leak memory - this article explains how to avoid this without sacrificing performance. ...
  3. Explicit multi-threading in Matlab part 4 Matlab performance can be improved by employing timer objects and spawning external processes. ...
  4. Convolution performance Matlab's internal implementation of convolution can often be sped up significantly using the Convolution Theorem. ...
 
]]>
Readers of this blog are probably aware by now that I am currently writing my second book, MATLAB Performance Tuning (expected publication date: early 2014, CRC Press). During my work on this book, I encounter many surprising aspects of Matlab performance. In many cases these aspects are not un-documented per-se, but are certainly not well known in the Matlab community. So taking some artistic liberty coupled with some influence over this blog’s owner, I’ll mention some of these interesting discoveries here, even if they are not strictly-speaking undocumented.

Today’s post is about the well-known fwrite function, which is used to write binary data to file. In many cases, using fwrite provides the fastest alternative to saving data files (save(…,’-v6′) coming a close second). This function is in fact so low-level, and is used so often, that some readers may be surprised that its default speed can be improved. Today’s article applies equally to the fprintf function, which is used to save data in text format.

Apparently, there are things to be learned even with such standard low-level functions; there’s a deep moral here I guess.

Flushing and buffering

Unlike C/C++’s implementation, Matlab’s fprintf and fwrite automatically flush the output buffer whenever they are called, even when '\n' is not present in the output stream. This is not mentioned outright in the main documentation, but is stated loud and clear in the official technical support solution 1-PV371.

The only exception to this rule is when the file was fopen‘ed with the 'W' or 'A' specifiers (which for some inexplicable reason is NOT mentioned in the technical solution!), or when outputting to the MATLAB’s Command Window (more precisely, to STDOUT (fid=1) and STDERR (fid=2)). Writing data without buffering in this manner severely degrades I/O performance:

data = randi(250,1e6,1);  % 1M integer values between 1-250
 
% Standard unbuffered writing - slow
fid = fopen('test.dat', 'wb');
tic, for idx = 1:length(data), fwrite(fid,data(idx)); end, toc
fclose(fid);
  => Elapsed time is 14.006194 seconds.
 
% Buffered writing – x4 faster
fid = fopen('test.dat', 'Wb');
tic, for idx = 1:length(data), fwrite(fid,data(idx)); end, toc
fclose(fid);
  => Elapsed time is 3.471557 seconds.

If I were in a generous mood, I could say that we could infer this information from fopen‘s doc page, where it mentions using the 'W' and 'A' permission specifiers to prevent automatic flushing, although it qualifies this with the very misleading statement that these specifiers are “Used with tape drives“. So first of all, who ever uses tape drives with Matlab nowadays?! Secondly, these specifiers are very useful for regular buffered I/O on standard disks and other I/O interfaces. I really think this was a poor choice of words. At the very least some extra clarification about these specifiers could be added.

It was also (IMHO) a poor design choice by MathWorks in the first place to break consistency with the C/C++ implementation for 'w' and associate the functionality to 'W' (and similarly, 'a' vs. 'A'). C’s fopen was in widespread usage for a decade before Matlab was invented, so there is really no excuse, certainly when Matlab’s fopen was so clearly modeled after the C implementation. It would have been more reasonable (again – IMHO) to preserve consistency of 'w' and 'a' for a default of buffered I/O (which is faster!), while providing the non-buffered functionality in 'W' and 'A'.

The vast majority of Matlab users fopen their files using 'w' and not 'W'. Even Matlab’s own documentation always uses 'w' and not 'W'. So coupled with the poorly-worded qualification about the tape drives, and the unintuitive inconsistency with C’s implementation, Matlab users could well be excused for not taking advantage of this feature.

Chunking I/O

The idea of buffering, and the reason behind the speedup above, is that I/O is faster when writing full pages (typically 4KB, but this changes on different platforms) and when bunched together to remove the disk access time between adjacent writes. This idea can be extended by preparing the entire file data in memory, and then using a single fwrite to write everything at once:

fid = fopen('test.dat', 'wb');
tic, fwrite(fid,data); toc
fclose(fid);
  => Elapsed time is 0.014025 seconds.

In fact, assembling the entire data in memory, within a long numeric or char array, and then using a single fwrite to save this array to file, is almost as fast as we can expect to get (example). Further improvement lies in optimizing the array assembly (which is CPU and memory-intensive) rather than the I/O itself.

In this example, the I/O was so fast (14mS) that it makes sense to write everything at once. But for enormously large data files and slower disks (I use a local SSD; network hard disks are way slower), writing the entire file’s data in this manner might take long minutes. In such cases, it is advisable to deliberately break up the data into smaller chunks, and fwrite them separately in a loop, all the time providing feedback to the user about the I/O’s progress. This could help improve the operation’s perceived performance. Here’s a bare-bones example:

h = waitbar(0, 'Saving data...', 'Name','Saving data...');
cN = 100;  % number of steps/chunks
 
% Divide the data into chunks (last chunk is smaller than the rest)
dN = length(data);
dataIdx = [1 : round(dN/cN) : dN, dN+1];  % cN+1 chunk location indexes
 
% Save the data
fid = fopen('test.dat', 'Wb');
for chunkIdx = 0 : cN-1
   % Update the progress bar
   fraction = chunkIdx/cN;
   msg = sprintf('Saving data... (%d%% done)', round(100*fraction));
   waitbar(fraction, h, msg);
 
   % Save the next data chunk
   chunkData = data(dataIdx(chunkIdx+1) : dataIdx(chunkIdx+2)-1);
   fwrite(fid,chunkData);
end
 
fclose(fid);
close(h);

Of course, rather than using a plain-ol’ waitbar window, we could integrate a progress bar directly into our GUI. Using my statusbar utility is one way to do it, but there are of course many other possible ways to dynamically present progress:

Dynamically updating progress using the statusbar utility (click for details)

Dynamically updating progress using the statusbar utility

Note: the entire article above applies equally well to fprintf in addition to fwrite. Storing and loading data in binary format (using fwrite/fread) is often faster than text format (using fprintf/fscanf/textscan), so we should generally use text format only if the file needs to be human-readable for any reason.

Do you know of any other trick to store data efficiently? If so, please share it in a comment.

Next week: some surprising performance aspects of Matlab’s save function.

]]>
https://undocumentedmatlab.com/blog/improving-fwrite-performance/feed 10
Matlab-Java memory leaks, performancehttps://undocumentedmatlab.com/blog/matlab-java-memory-leaks-performance https://undocumentedmatlab.com/blog/matlab-java-memory-leaks-performance#comments Fri, 20 Jan 2012 00:56:10 +0000 http://undocumentedmatlab.com/?p=2665
 
Related posts:
  1. File deletion memory leaks, performance Matlab's delete function leaks memory and is also slower than the equivalent Java function. ...
  2. Undocumented XML functionality Matlab's built-in XML-processing functions have several undocumented features that can be used by Java-savvy users...
  3. Preallocation performance Preallocation is a standard Matlab speedup technique. Still, it has several undocumented aspects. ...
  4. Array resizing performance Several alternatives are explored for dynamic array growth performance in Matlab loops. ...
 
]]>
There are several ways of retrieving information from a Java object into Matlab. On the face of it, all these methods look similar. But it turns out that there are important differences between them in terms of memory leakage and performance.

The problem: “Matlab crashes” – now go figure…

A client of one of my Matlab programs recently complained that Matlab crashes after several hours of extensive use of the program. The problem looked like something that is memory related (messages such as Matlab’s out-of-memory error or Java’s heap-space error). Apparently this happens even on 64-bit systems having lots of memory, where memory should never be a problem.

Well, we know that this is only in theory, but in practice Matlab’s internal memory management has problems that occasionally lead to such crashes. This is one of the reasons, by the way, that recent Matlab releases have added the preference option of increasing the default Java heap space (the previous way to do this was a bit complex). Still, even with a high Java heap space setting and lots of RAM, Matlab crashed after using my program for several hours.

Not pleasant at all, even a bit of an embarrassment for me. I’m used to crashing Matlab, but only as a result of my playing around with the internals – I would hate it to happen to my clients.

Finding the leak

While we can do little with Matlab’s internal memory manager, I started searching for the exact location of the memory leak and then to find a way to overcome it. I’ll save readers the description about the grueling task of finding out exactly where the memory leak occurred in a program that has thousands of lines of code and where events get fired asynchronously on a constant basis. Matlab Profiler’s undocumented memory profiling option helped me quite a bit, as well as lots of intuition and trial-and-error. Detecting memory leaks is never easy, and I consider myself somewhat lucky this time to have both detected the leak source and a workaround.

It turned out that the leakage happens in a callback that gets invoked multiple times per second by a Java object (see related articles here and here). Each time the Matlab callback function is invoked, it reads the event information from the supplied Java event-data (the callback’s second input parameter). Apparently, about 1KB of memory gets leaked whenever this event-data is being read. This may appear a very small leak, but multiply this by some 50-100K callback invocations per hour and you get a leakage of 50-100MB/hour. Not a small leak at all; more of a flood you could say…

Using get()

The leakage culprit turned out to be the following code snippet:

% 160 uSecs per call, with memory leak
eventData  = get(hEventData,{'EventName','ParamNames','EventData'});
eventName  = eventData{1};
paramNames = eventData{2};
paramData  = eventData{3}.cell;

In this innocent-looking code, hEventData is a Java object that contains the EventName, ParamNames, EventData properties: EventName is a Java String, that is automatically converted by Matlab’s get() function into a Matlab string (char array); ParamNames is a Java array of Strings, that gets automatically converted into a Matlab cell-array of string; and EventData is a Java array of Objects that needs to be converted into a Matlab cell array using the built-in cell function, as described in one of my recent articles.

The code is indeed innocent, works really well and is actually extremely fast: each invocation of this code segment takes less than 0.2 millisecs. Unfortunately, because of the memory leak I needed to find a better alternative.

Using handle()

The first idea was to use the built-in handle() function, under the assumption that it would solve the memory leak, as reported here. In fact, MathWorks specifically advises to use handle() rather than to work with “naked” Java objects, when setting Java object callbacks. The official documentation of the set function says:

Do not use the set function on Java objects as it will cause a memory leak.

It stands to reason then that a similar memory leak happens with get and that a similar use of handle would solve this problem:

% 300 uSecs per call, with memory leak
s = get(handle(hEventData));
eventName  = s.EventName;
paramNames = s.ParamNames;
paramData  = cell(s.EventData);

Unfortunately, this variant, although working correctly, still leaks memory, and also performs almost twice as worse than the original version, taking some 0.3 milliseconds to execute per invocation. Looks like this is a dead end.

Using Java accessor methods

The next attempt was to use the Java object’s internal accessor methods for the requested properties. These are public methods of the form getXXX(), isXXX(), setXXX() that enable Matlab to treat XXX as a property by its get and set functions. In our case, we need to use the getter methods, as follows:

% 380 uSecs per call, no memory leak
eventName  = char(hEventData.getEventName);
paramNames = cell(hEventData.getParamNames);
paramData  = cell(hEventData.getEventData);

Here, the method getEventName() returns a Java String, that we convert into a Matlab string using the char function. In our previous two variants, the get function did this conversion for us automatically, but when we use the Java method directly we need to convert the results ourselves. Similarly, when we call getParamNames(), we need to use the cell function to convert the Java String[] array into a Matlab cell array.

This version at last doesn’t leak any memory. Unfortunately, it has an even worse performance: each invocation takes almost 0.4 milliseconds. The difference may seem insignificant. However, recall that this callback gets called dozens of times each second, so the total adds up quickly. It would be nice if there were a faster alternative that does not leak any memory.

Using struct()

Luckily, I found just such an alternative. At 0.24 millisecs per invocation, it is almost as fast as the leaky best-performance original get version. Best of all, it leaks no memory, at least none that I could detect.

The mechanism relies on the little-known fact that public fields of Java objects can be retrieved in Matlab using the built-in struct function. For example:

>> fields = struct(java.awt.Rectangle)
fields = 
             x: 0
             y: 0
         width: 0
        height: 0
      OUT_LEFT: 1
       OUT_TOP: 2
     OUT_RIGHT: 4
    OUT_BOTTOM: 8
 
>> fields = struct(java.awt.Dimension)
fields = 
     width: 0
    height: 0

Note that this useful mechanism is not mentioned in the main documentation page for accessing Java object fields, although it is indeed mentioned in another doc-page – I guess this is a documentation oversight.

In any case, I converted my Java object to use public (rather than private) fields, so that I could use this struct mechanism (Matlab can only access public fields). Yes I know that using private fields is a better programming practice and all that (I’ve programmed OOP for some 15 years…), but sometimes we need to do ugly things in the interest of performance. The latest version now looks like this:

% 240 uSecs per call, no memory leak
s = struct(hEventData);
eventName  = char(s.eventName);
paramNames = cell(s.paramNames);
paramData  = cell(s.eventData);

This solved the memory leakage issue for my client. I felt fortunate that I was not only able to detect Matlab’s memory leak but also find a working workaround without sacrificing performance or functionality.

In this particular case, I was lucky to have full control over my Java object, to be able to convert its fields to become public. Unfortunately, we do not always have similar control over the object that we use, because they were coded by a third party.

By the way, Matlab itself uses this struct mechanism in its code-base. For example, Matlab timers are implemented using Java objects (com.mathworks.timer.TimerTask). The timer callback in Matlab code converts the Java timer event data into a Matlab struct using the struct function, in %matlabroot%/toolbox/matlab/iofun/@timer/timercb.m. The users of the timer callbacks then get passed a simple Matlab EventData struct without ever knowing that the original data came from a Java object.

As an interesting corollary, this same struct mechanism can be used to detect internal properties of Matlab class objects. For example, in the timers again, we can get the underlying timer’s Java object as follows (note the highlighted warning, which I find a bit ironic given the context):

>> timerObj = timerfind
 
   Timer Object: timer-1
 
   Timer Settings
      ExecutionMode: singleShot
             Period: 1
           BusyMode: drop
            Running: off
 
   Callbacks
           TimerFcn: @myTimerFcn
           ErrorFcn: ''
           StartFcn: ''
            StopFcn: ''
 
>> timerFields = struct(timerObj)
Warning: Calling STRUCT on an object prevents the object from hiding its implementation details and should thus be avoided.Use DISP or DISPLAY to see the visible public details of an object. See 'help struct' for more information.(Type "warning off MATLAB:structOnObject" to suppress this warning.)timerFields = 
         ud: {}
    jobject: [1x1 javahandle.com.mathworks.timer.TimerTask]
]]>
https://undocumentedmatlab.com/blog/matlab-java-memory-leaks-performance/feed 34
Types of undocumented Matlab aspectshttps://undocumentedmatlab.com/blog/types-of-undocumented-matlab-aspects https://undocumentedmatlab.com/blog/types-of-undocumented-matlab-aspects#respond Thu, 24 Nov 2011 18:00:36 +0000 http://undocumentedmatlab.com/?p=2534
 
Related posts:
  1. Matlab callbacks for Java events Events raised in Java code can be caught and handled in Matlab callback functions - this article explains how...
  2. Uitable sorting Matlab's uitables can be sortable using simple undocumented features...
  3. Matlab’s HG2 mechanism HG2 is presumably the next generation of Matlab graphics. This article tries to explore its features....
  4. Minimize/maximize figure window Matlab figure windows can easily be maximized, minimized and restored using a bit of undocumented magic powder...
 
]]>
Why are there so many undocumented aspects in Matlab?

This is a great question, recently asked by a reader of this blog, so I wanted to expand on it in next week’s article. Before specifying the different reasons, let’s map the nature of undocumented aspects that we find in Matlab.

The term undocumented/unsupported (as opposed to mis-documentated or deprecated) actually refers to quite a large number of different types.
In the following list, the hyperlinks on the list-item titles lead to a list of corresponding articles on this website:

  • Undocumented functions
    Matlab functions which appears nowhere in the documentation, are usually built-in functions (do not have an m-file) and can only be inferred from online CSSM posts or usage within one of the Matlab m-functions installed with Matlab (the latter being the usual case). None of these functions is officially supported by MathWorks. MEX is an important source for such functions.

  • Semi-documented functions
    Matlab functionality which exists in Matlab m-functions installed with Matlab, but have their main comment separated from the H1 comment line, thereby hiding it from normal view (via Matlab’s help function). The H1 comment line itself is simply a standard warning that this function is not officially supported and may change in some future version. To see the actual help comment, simply edit the function (using Matlab’s edit function or any text editor) and place a comment sign (%) at the empty line between the H1 comment and the actual help section. The entire help section will then onward be visible via the help function:

            function [tree, container] = uitree(varargin)
            % WARNING: This feature is not supported in MATLAB
            % and the API and functionality may change in a future release.
    fix =>  %
            % UITREE creates a uitree component with hierarchical data in a figure window.
            %   UITREE creates an empty uitree object with default property values in
            %   a figure window.
            %...

    These functions are not documented in the full documentation (via Matlab’s doc function, or online). The odd thing is that some of these functions may appear in the category help output (for example, help(‘uitools’)), and in some cases may even have a fully-visible help section (e.g., help(‘setptr’)), but do not have any online help documentation (docsearch(‘setptr’) fails, and doc(‘setptr’) simply displays the readable help text).

    All these functions are officially unsupported by MathWorks, even when having a readable help section. The rule of thumb appears to be that a Matlab function is supported only if it has online documentation. Note, however, that in some rare cases a documentation discrepancy may be due to a MathWorks documentation error, not to unsupportability…

  • Helper functions
    Many fully-supported Matlab functions use helper functions that have a specific use in the main (documented) function(s). Often, these helper functions are tightly-coupled to their documented parents and are useless as stand-alone functions. But quite a few of them have quite useful stand-alone use, as I’ve already shown in some past articles.

  • Undocumented features and properties
    Features of otherwise-documented Matlab functions, which appear nowhere in the official documentation. You guessed it – these are also not supported by MathWorks… Like undocumented functions, you can only infer such features by the occasional CSSM post or a reference somewhere in Matlab’s m-code.

  • Semi-documented features
    Features of otherwise-documented Matlab functions, which are documented in a separate section beneath the main help section, and nowhere else (not in the full doc not the online documentation). If you did not know in advance that these features existed, you could only learn of them by manually looking at Matlab’s m-files (which is what I do in most cases…).

  • Undocumented properties
    Many Matlab objects have internal properties, which can be retrieved (via Matlab’s get function) and/or set (via the set function) programmatically. All these properties are fully documented. Many objects also possess hidden properties, some of which are very interesting and useful, but which are undocumented and (oh yes) unsupported. Like undocumented features, they can only be inferred from CSSM or existing code. In a recent article I described my getundoc utility, which lists these undocumented properties of specified objects.

  • Internal Matlab classes
    Matlab uses a vast array of specialized Java classes to handle everything from algorithms to GUI. These classes are (of course) undocumented/unsupported. They can often be accessed directly from the Matlab Command Window or user m-files. GUI classes can be inferred by inspecting the figure frame’s Java components, and non-GUI classes can often be inferred from references in Matlab’s m-files.

  • Matlab-Java integration
    Matlab’s GUI interface, as well as the Java-to-Matlab interface (JMI) is fully undocumented and unsupported. In addition to JMI, there are other mechanisms to run Matlab code from within Java (namely JMI, COM and DDE) – these are all unsupported and by-and-large undocumented.

  • Matlab’s UDD mechanism
    UDD (Unified Data Definition?) is used extensively in Matlab as the internal object-oriented mechanism for describing object properties and functionalities. We can use UDD for a wide variety of uses. UDD was described in a series of articles here in early 2011.

Next week I will list the reasons that cause MathWorks to decide whether a particular feature or property should be documented or not.

]]>
https://undocumentedmatlab.com/blog/types-of-undocumented-matlab-aspects/feed 0
Customizing uitree nodes – part 1https://undocumentedmatlab.com/blog/customizing-uitree-nodes https://undocumentedmatlab.com/blog/customizing-uitree-nodes#comments Wed, 25 Aug 2010 18:00:20 +0000 http://undocumentedmatlab.com/?p=1835
 
Related posts:
  1. Uitab colors, icons and images Matlab's semi-documented tab panels can be customized using some undocumented hacks...
  2. Figure toolbar components Matlab's toolbars can be customized using a combination of undocumented Matlab and Java hacks. This article describes how to access existing toolbar icons and how to add non-button toolbar components....
  3. Customizing uitree nodes – part 2 This article shows how Matlab GUI tree nodes can be customized with checkboxes and similar controls...
  4. Customizing uitree This article describes how to customize Matlab GUI tree controls created using the undocumented uitree function...
 
]]>
In my previous posts, I introduced the semi-documented uitree function that enables displaying data in a hierarchical (tree) control in Matlab GUI, and showed how it can be customized. Today, I will continue by describing how specific uitree nodes can be customized.

To start the discussion, let’s re-create last week’s simple uitree:

% Fruits
fruits = uitreenode('v0', 'Fruits', 'Fruits', [], false);
fruits.add(uitreenode('v0', 'Apple',  'Apple',  [], true));
fruits.add(uitreenode('v0', 'Pear',   'Pear',   [], true));
fruits.add(uitreenode('v0', 'Banana', 'Banana', [], true));
fruits.add(uitreenode('v0', 'Orange', 'Orange', [], true));
 
% Vegetables
veggies = uitreenode('v0', 'Veggies', 'Vegetables', [], false);
veggies.add(uitreenode('v0', 'Potato', 'Potato', [], true));
veggies.add(uitreenode('v0', 'Tomato', 'Tomato', [], true));
veggies.add(uitreenode('v0', 'Carrot', 'Carrot', [], true));
 
% Root node
root = uitreenode('v0', 'Food', 'Food', [], false);
root.add(veggies);
root.add(fruits);
 
% Tree
figure('pos',[300,300,150,150]);
mtree = uitree('v0', 'Root', root);

User-created tree    User-created tree

User-created tree

Labels

Node labels (descriptions) can be set using their Name property (the second uitreenode data argument). Note that the horizontal space allotted for displaying the node name will not change until the node is collapsed or expanded. So, if the new name requires more than the existing space, it will be displayed as something like “abc…”, until the node is expanded or collapsed.

Node names share the same HTML support feature as all Java Swing labels. Therefore, we can specify font size/face/color, bold, italic, underline, super-/sub-script etc.:

txt1 = '<html><b><u><i>abra</i></u>';
txt2 = '<font color="red"><sup>kadabra</html>';
node.setName([txt1,txt2]);

HTML-enriched tree nodes

HTML-enriched tree nodes

Icons

Tree-node icons can be specified during node creation, as the third data argument to uitreenode, which accepts an icon-path (a string):

iconPath = fullfile(matlabroot,'/toolbox/matlab/icons/greenarrowicon.gif');
node = uitreenode('v0',value,name,iconPath,isLeaf);

Tree node icons can also be created or modified programmatically in run-time, using Matlab’s im2java function. Icons can also be loaded from existing files as follows (real-life programs should check and possibly update jImage’s size to 16 pixels, before setting the node icon, otherwise the icon might get badly cropped; also note the tree-refreshing action):

jImage = java.awt.Toolkit.getDefaultToolkit.createImage(iconPath);
veggies.setIcon(jImage);
veggies.setIcon(im2java(imread(iconPath)));  % an alternative
 
% refresh the veggies node (and all its children)
mtree.reloadNode(veggies);

Setting node icon

Setting node icon

Behavior

Nodes can be modified from leaf (non-expandable) to parent behavior (=expandable) by setting their LeafNode property (a related property is AllowsChildren):

set(node,'LeafNode',false);  % =expandable
node.setLeafNode(0);  % an alternative

One of the questions I was asked was how to “disable” a specific tree node. One way would be to modify the tree’s ExpandFcn callback. Another way is to use a combination of HTML rendering and the node’s AllowsChildren property:

label = char(veggies.getName);
veggies.setName(['<html><font color="gray">' label]);
veggies.setAllowsChildren(false);
t.reloadNode(veggies);

Disabled node

Disabled node

Another possible behavioral customization is adding a context-menu to a uitree. We can set node-specific tooltips using similar means.

Answering a reader’s request from last week, tree nodes icons can be used to present checkboxes, radio buttons and other similar node-specific controls. This can actually be done in several ways, that will be explored in next week’s article.

There are numerous other possible customizations – if readers are interested, perhaps I will describe some of them in future articles. If you have any special request, please post a comment below.

]]>
https://undocumentedmatlab.com/blog/customizing-uitree-nodes/feed 17
Undocumented XML functionalityhttps://undocumentedmatlab.com/blog/undocumented-xml-functionality https://undocumentedmatlab.com/blog/undocumented-xml-functionality#comments Thu, 19 Nov 2009 01:12:06 +0000 http://undocumentedmatlab.com/?p=751
 
Related posts:
  1. Parsing XML strings Matlab's xmlread function cannot process XML data directly, but this can easily be overcome. ...
  2. Types of undocumented Matlab aspects This article lists the different types of undocumented/unsupported/hidden aspects in Matlab...
  3. Matlab-Java memory leaks, performance Internal fields of Java objects may leak memory - this article explains how to avoid this without sacrificing performance. ...
  4. Sending email/text messages from Matlab Sending emails and SMS (text) messages from Matlab is easy, once you know a few quirks....
 
]]>
Matlab’s built-in XML-processing functions have several undocumented features that can be used by Java-savvy users. We should note that the entire XML-support functionality in Matlab is java-based. I understand that some Matlab users have a general aversion to Java, some even going as far as to disable it using the -nojvm startup option. But if you disable Java, Matlab’s XML functions will simply not work. Matlab’s own documentation points users to Sun’s official Java website for explanations of how to use the XML functionality (the link in the Matlab docpage is dead – the correct link should probably be https://jaxp-sources.dev.java.net/nonav/docs/api/, but Sun keeps changing its website so this link could also be dead soon…).

Using the full Java XML parsing (JAXP) functionality is admittedly quite intimidating for the uninitiated, but extremely powerful once you understand how all the pieces fit together. Over the years, several interesting utilities were submitted to the Matlab File Exchange that simplify this intimidating post-processing. See for example XML parsing tools, the extremely popular XML Toolbox and xml_io_tools, the recent XML data import and perhaps a dozen other utilities.

Each of Matlab’s main built-in XML-processing functions, xmlread, xmlwrite and xslt has an internal set of undocumented and unsupported functionalities, which builds on their internal Java implementation. As far as I could tell, these unsupported functionalities were supported at least as early as Matlab 7.2 (R2006a), and possibly even on earlier releases. For the benefit of the Java and/or JAXP -speakers out there (it will probably not help any others), I list Matlab’s internal description of these unsupported functionalities, annotated with API hyperlinks. These description (sans the links) can be seen by simply editing the m file, as in (the R2008a variant is described below):

edit xmlread

xmlread

function [parseResult,p] = xmlread(fileName,varargin)
  • FILENAME can also be an InputSource, File, or InputStream object
  • DOMNODE = XMLREAD(FILENAME,…,P,…) where P is a DocumentBuilder object
  • DOMNODE = XMLREAD(FILENAME,…,’-validating’,…) will create a validating parser if one was not provided.
  • DOMNODE = XMLREAD(FILENAME,…,ER,…) where ER is an EntityResolver will set the EntityResolver before parsing
  • DOMNODE = XMLREAD(FILENAME,…,EH,…) where EH is an ErrorHandler will set the ErrorHandler before parsing
  • [DOMNODE,P] = XMLREAD(FILENAME,…) will return a parser suitable for passing back to XMLREAD for future parses.

xmlwrite

function xmlwrite(FILENAME,DOMNODE);
function str = xmlwrite(DOMNODE);
function str = xmlwrite(SOURCE);

xslt

function [xResultURI,xProcessor] = xslt(SOURCE,STYLE,DEST,varargin)
  • SOURCE can also be a XSLTInputSource
  • STYLE can also be a StylesheetRoot or XSLTInputSource
  • DEST can also be an XSLTResultTarget. Note that RESULT may be empty in this case since it may not be possible to determine a URL. If STYLE is absent or empty, the function uses the stylesheet named in the xml-stylesheet processing instruction in the SOURCE XML file. (This does not always work)
  • There is also an entirely undocumented feature: passing a ‘-tostring’ input argument transforms the inputs into a displayed text segment, rather than into a displayed URI; the transformed text is returned in the xResultURI output argument.

Note: internal comments within the Matlab code seem to indicate that XSLT is SAXON-based, so interested users might use SAXON’s documentation for accessing additional XSLT-related features/capabilities.

]]>
https://undocumentedmatlab.com/blog/undocumented-xml-functionality/feed 6
Legend ‘-DynamicLegend’ semi-documented featurehttps://undocumentedmatlab.com/blog/legend-semi-documented-feature https://undocumentedmatlab.com/blog/legend-semi-documented-feature#comments Thu, 04 Jun 2009 23:00:22 +0000 http://undocumentedmatlab.com/?p=357
 
Related posts:
  1. Context-Sensitive Help Matlab has a hidden/unsupported built-in mechanism for easy implementation of context-sensitive help...
  2. Multi-column (grid) legend This article explains how to use undocumented axes listeners for implementing multi-column plot legends...
  3. Inactive Control Tooltips & Event Chaining Inactive Matlab uicontrols cannot normally display their tooltips. This article shows how to do this with a combination of undocumented Matlab and Java hacks....
  4. UDD Events and Listeners UDD event listeners can be used to listen to property value changes and other important events of Matlab objects...
 
]]>
In one of my projects, I had to build a GUI in which users could interactively add and remove plot lines from an axes. The problem was that the legend needed to be kept in constant sync with the currently-displayed plot lines. This can of course be done programmatically, but a much simpler solution was to use legend‘s semi-documented ‘-DynamicLegend’ feature. Here’s a simple example:

x=0:.01:10;
plot(x, sin(x), 'DisplayName','sin');
legend('-DynamicLegend');
 
hold all;   % add new plot lines on top of previous ones
plot(x, cos(x), 'DisplayName','cos');

We can see how the dynamic legend automatically keeps in sync with its associated axes contents when plot lines are added/removed, even down to the zoom-box lines… The legend automatically uses the plot lines ‘DisplayName’ property where available, or a standard ‘line#’ nametag where not available:


Dynamic legend

Dynamic legend

DynamicLegend works by attaching a listener to the axes child addition/deletion callback (actually, it works on the scribe object, which is a large topic for several future posts). It is sometimes necessary to selectively disable the dynamic behavior. For example, in my GUI I needed to plot several event lines which looked alike, and so I only wanted the first line to be added to the legend. To temporarily disable the DynamicLegend listener, do the following:

% Try to disable this axes's legend plot-addition listener
legendAxListener = [];
try
   legendListeners = get(gca,'ScribeLegendListeners');
   legendAxListener = legendListeners.childadded;
   set(legendAxListener,'Enable','off');
catch
   % never mind...
end
 
% Update the axes - the legend will not be updated
...
 
% Re-enable the dynamic legend listener
set(legendAxListener,'Enable','on');

Unfortunately, this otherwise-useful DynamicLegend feature throws errors when zooming-in on bar or stairs graphs. This can be replicated by:

figure;
bar(magic(4));  %or: stairs(magic(3),magic(3));
legend('-DynamicLegend');
zoom on;
% Now zoom-in using the mouse to get the errors on the Command Window

The fix: modify %MATLABROOT%\toolbox\matlab\scribe\@scribe\@legend\init.m line #528 as follows:

%old:
str = [str(1:insertindex-1);{newstr};str(insertindex:length(str))];
 
%new:
if size(str,2)>size(str,1)
    str=[str(1:insertindex-1),{newstr},str(insertindex:length(str))];
else
    str=[str(1:insertindex-1);{newstr};str(insertindex:length(str))];
end

The origin of the bug is that bar and stairs generate hggroup plot-children, which saves the legend strings column-wise rather than the expected row-wise. My fix solves this, but I do not presume this solves all possible problems in all scenarios (please report if you find anything else).

p.s. – object visibility in the legend can be controlled on an object-by-object basis using the semi-documented hasbehavior function.

Semi-documented

The DynamicLegend feature is semi-documented. This means that the feature is explained in a comment within the function (which can be seen via the edit(‘legend’) command), that is nonetheless not part of the official help or doc sections. It is an unsupported feature originally intended only for internal Matlab use (which of course doesn’t mean we can’t use it). This feature has existed many releases back (Matlab 7.1 for sure, perhaps earlier), so while it may be discontinued in some future Matlab release, it did have a very long life span… The down side is that it is not supported: I reported the bar/stairs issue back in mid-2007 and so far this has not been addressed (perhaps it will never be). Even my reported workaround in January this year went unanswered (no hard feelings…).

DynamicLegend is a good example of a useful semi-documented feature. Some other examples, which I may cover in future posts, include text(…,‘sc’), drawnow(‘discard’), several options in pan and datacursormode etc. etc.

There are also entire semi-documented functions: many of the uitools (e.g., uitree, uiundo), as well as hgfeval and others.

Have you discovered any useful semi-documented feature or function? If so, then please share your finding in the comments section below.

]]>
https://undocumentedmatlab.com/blog/legend-semi-documented-feature/feed 21