Stock Matlab function – Undocumented Matlab https://undocumentedmatlab.com Charting Matlab's unsupported hidden underbelly Sat, 17 Mar 2018 19:03:59 +0000 en-US hourly 1 https://wordpress.org/?v=4.4.1 Customizing axes tick labelshttps://undocumentedmatlab.com/blog/customizing-axes-tick-labels https://undocumentedmatlab.com/blog/customizing-axes-tick-labels#comments Wed, 24 Jan 2018 13:38:26 +0000 https://undocumentedmatlab.com/?p=7304

Related posts:
1. HG2 update HG2 appears to be nearing release. It is now a stable mature system. ...
2. Performance: accessing handle properties Handle object property access (get/set) performance can be significantly improved using dot-notation. ...
3. Customizing axes rulers HG2 axes can be customized in numerous useful ways. This article explains how to customize the rulers. ...
4. Plot legend title Titles to plot legends are easy to achieve in HG1 (R2014a or earlier), but much more difficult in HG2 (R2014b or newer). ...

]]>
In last week’s post, I discussed various ways to customize bar/histogram plots, including customization of the tick labels. While some of the customizations that I discussed indeed rely on undocumented properties/features, many Matlab users are not aware that tick labels can be individually customized, and that this is a fully documented/supported functionality. This relies on the fact that the default axes TickLabelInterpreter property value is 'tex', which supports a wide range of font customizations, individually for each label. This includes any combination of symbols, superscript, subscript, bold, italic, slanted, face-name, font-size and color – even intermixed within a single label. Since tex is the default interpreter, we don’t need any special preparation – simply set the relevant X/Y/ZTickLabel string to include the relevant tex markup.

To illustrate this, have a look at the following excellent answer by user Ubi on Stack Overflow:

Axes with Tex-customized tick labels

plot(1:10, rand(1,10)) ax = gca;   % Simply color an XTickLabel ax.XTickLabel{3} = ['\color{red}' ax.XTickLabel{3}];   % Use TeX symbols ax.XTickLabel{4} = '\color{blue} \uparrow';   % Use multiple colors in one XTickLabel ax.XTickLabel{5} = '\color[rgb]{0,1,0}green\color{orange}?';   % Color YTickLabels with colormap nColors = numel(ax.YTickLabel); cm = jet(nColors); for i = 1:nColors ax.YTickLabel{i} = sprintf('\\color[rgb]{%f,%f,%f}%s', cm(i,:), ax.YTickLabel{i}); end

In addition to 'tex', we can also set the axes object’s TickLabelInterpreter to 'latex' for a Latex interpreter, or 'none' if we want to use no string interpretation at all.

As I showed in last week’s post, we can control the gap between the tick labels and the axle line, using the Ruler object’s undocumented TickLabelGapOffset, TickLabelGapMultiplier properties.

Also, as I explained in other posts (here and here), we can also control the display of the secondary axle label (typically exponent or units) using the Ruler’s similarly-undocumented SecondaryLabel property. Note that the related Ruler’s Exponent property is documented/supported, but simply sets a basic exponent label (e.g., '\times10^{6}' when Exponent==6) – to set a custom label string (e.g., '\it\color{gray}Millions'), or to modify its other properties (position, alignment etc.), we should use SecondaryLabel.

]]>
https://undocumentedmatlab.com/blog/customizing-axes-tick-labels/feed 3
Customizing histogram plotshttps://undocumentedmatlab.com/blog/customizing-histogram-plots https://undocumentedmatlab.com/blog/customizing-histogram-plots#respond Wed, 17 Jan 2018 20:41:15 +0000 https://undocumentedmatlab.com/?p=7292

Related posts:
1. getundoc – get undocumented object properties getundoc is a very simple utility that displays the hidden (undocumented) properties of a specified handle object....
2. Customizing axes part 3 – Backdrop Matlab HG2 axes can be customized in many different ways. This article explains some of the undocumented aspects. ...
3. Customizing axes part 4 – additional properties Matlab HG2 axes can be customized in many different ways. This article explains some of the undocumented aspects. ...
4. Plot line transparency and color gradient Static and interpolated (gradient) colors and transparency can be set for plot lines in HG2. ...

]]>
Earlier today, I was given the task of displaying a histogram plot of a list of values. In today’s post, I will walk through a few customizations that can be done to bar plots and histograms in order to achieve the desired results.

We start by binning the raw data into pre-selected bins. This can easily be done using the builtin histc (deprecated) or histcounts functions. We can then use the bar function to plot the results:

[binCounts, binEdges] = histcounts(data); hBars = bar(hAxes, binEdges(1:end-1), binCounts);

Basic histogram bar plot

Let’s improve the appearance: In my specific case, the data was financial return (percentage) values, so let’s modify the x-label format accordingly and display a title. To make the labels and title more legible, we decrease the axes FontSize to 8 and remove the axes box:

hAxes = hBar.Parent; xtickformat(hAxes, '%g%%'); title(hAxes, 'Distribution of total returns (monthly %)'); set(hAxes, 'FontSize',8, 'Box','off')

Improved histogram bar plot

So far nothing undocumented. Note that the xtickformat/ytickformat functions were only introduced in R2016b – for earlier Matlab releases see this post (which does rely on undocumented aspects).

Now, let’s use a couple of undocumented properties: to remove the excess white-space margin around the axes we’ll set the axes’ LooseInset property, and to remove the annoying white space between the tick labels and the X-axis we’ll set the XRuler‘s TickLabelGapOffset property to -2 (default: +2):

set(hAxes, 'LooseInset',[0,0,0,0]); % default = [.13,.11,.095,.075] hAxes.XRuler.TickLabelGapOffset = -2; % default = +2

Even better histogram bar plot

Note that I used the undocumented axes XRuler property instead of the axes’ documented XAxis property, because XAxis is only available since R2015b, whereas XRuler (which points to the exact same object as XAxis) exists ever since R2014b, and so is better from a backward-compatibility standpoint. In either case, the ruler’s TickLabelGapOffset property is undocumented. Note that the ruler also contains another associated and undocumented TickLabelGapMultiplier property (default: 0.2), which I have not modified in this case.

Now let’s take a look at the bin labels: The problem with the bar plot above is that it’s not intuitively clear whether the bin for “5%”, for example, includes data between 4.5-5.5 or between 5.0-6.0 (which is the correct answer). It would be nicer if the labels were matched to the actual bin edges. There are 3 basic ways to fix this:

1. We could modify the bar plot axes tick values and labels, in essence “cheating” by moving the tick labels half a bin leftward of their tick values (don’t forget to add the extra tick label on the right):
hAxes.XTick(end+1) = hAxes.XTick(end) + 1; % extra tick label on the right labels = hAxes.XTickLabels; % preserve tick labels for later use below hAxes.XTick = hAxes.XTick - 0.5; % move tick labels 1/2 bin leftward hAxes.XTickLabel = labels; % restore pre-saved tick labels hAxes.XLim = hAxes.XLim - 0.5; % ...and align the XLim

Improved labels

2. We could use the bar function’s optional 'histc' flag, in order to display the bars in histogram mode. The problem in histogram mode is that while the labels are now placed correctly, the bars touch each other – I personally find distinct bars that are separated by a small gap easier to understand.
hBars = bar(..., 'histc'); % [snip] - same customizations to hAxes as done above

Basic histogram plot

With the original bar chart we could use the built-in BarWidth to set the bar/gap width (default: 0.8 meaning a 10% gap on either side of the bar). Unfortunately, calling bar with 'hist' or 'histc' (i.e. histogram mode) results in a Patch (not Bar) object, and patches do not have a BarWidth property. However, we can modify the resulting patch vertices in order to achieve the same effect:

% Modify the patch vertices (5 vertices per bar, row-based) hBars.Vertices(:,1) = hBars.Vertices(:,1) + 0.1; hBars.Vertices(4:5:end,1) = hBars.Vertices(4:5:end,1) - 0.2; hBars.Vertices(5:5:end,1) = hBars.Vertices(5:5:end,1) - 0.2;   % Align the bars & labels at the center of the axes hAxes.XLim = hAxes.XLim + 0.5;

This now looks the same as option #1 above, except that the top-level handle is a Patch (not Bar) object. For various additional customizations, either Patch or Bar might be preferable, so you have a choice.

Improved histogram plot

3. Lastly, we could have used the builtin histogram function instead of bar. This function also displays a plot with touching bars, as above, using Quadrilateral objects (a close relative of Patch). The solution here is very similar to option #2 above, but we need to dig a bit harder to modify the patch faces, since their vertices is not exposed as a public property of the Histogram object. To modify the vertices, we first get the private Face property (explanation), and then modify its vertices, keeping in mind that in this specific case the bars have 4 vertices per bar and a different vertices matrix orientation:
hBars = histogram(data, 'FaceAlpha',1.0, 'EdgeColor','none'); % [snip] - same customizations to hAxes as done above   % Get access to *ALL* the object's properties oldWarn = warning('off','MATLAB:structOnObject'); warning off MATLAB:hg:EraseModeIgnored hBarsStruct = struct(hBars); warning(oldWarn);   % Modify the patch vertices (4 vertices per bar, column-based) drawnow; % this is important, won't work without this! hFace = hBarsStruct.Face; % a Quadrilateral object (matlab.graphics.primitive.world.Quadrilateral) hFace.VertexData(1,:) = hFace.VertexData(1,:) + 0.1; hFace.VertexData(1,3:4:end) = hFace.VertexData(1,3:4:end) - 0.2; hFace.VertexData(1,4:4:end) = hFace.VertexData(1,4:4:end) - 0.2;

In conclusion, there are many different ways to improve the appearance of charts in Matlab. Even if at first glance it may seem that some visualization function does not have the requested customization property or feature, a little digging will often find either a relevant undocumented property, or an internal object whose properties could be modified. If you need assistance with customizing your charts for improved functionality and appearance, then consider contacting me for a consulting session.

]]>
https://undocumentedmatlab.com/blog/customizing-histogram-plots/feed 0
Customizing contour plots part 2https://undocumentedmatlab.com/blog/customizing-contour-plots-part2 https://undocumentedmatlab.com/blog/customizing-contour-plots-part2#comments Sun, 12 Nov 2017 11:03:37 +0000 https://undocumentedmatlab.com/?p=7149

Related posts:
1. Draggable plot data-tips Matlab's standard plot data-tips can be customized to enable dragging, without being limitted to be adjacent to their data-point. ...
2. Customizing contour plots Contour labels, lines and fill patches can easily be customized in Matlab HG2. ...
3. Matlab’s HG2 mechanism HG2 is presumably the next generation of Matlab graphics. This article tries to explore its features....
4. getundoc – get undocumented object properties getundoc is a very simple utility that displays the hidden (undocumented) properties of a specified handle object....

]]>
A few weeks ago a user posted a question on Matlab’s Answers forum, asking whether it is possible to display contour labels in the same color as their corresponding contour lines. In today’s post I’ll provide some insight that may assist users with similar customizations in other plot types.

Matlab does not provide, for reasons that escape my limited understanding, documented access to the contour plot’s component primitives, namely its contour lines, labels and patch faces. Luckily however, these handles are accessible (in HG2, i.e. R2014b onward) via undocumented hidden properties aptly named EdgePrims, TextPrims and FacePrims, as I explained in a previous post about contour plots customization, two years ago.

[X,Y,Z] = peaks; [C,hContour] = contour(X,Y,Z, 'ShowText','on', 'LevelStep',1);

The result is the screenshot on the left:

Standard Matlab contour labels

Customized Matlab contour labels

In order to update the label colors (to get the screenshot on the right), we create a short updateContours function that updates the TextPrims color to their corresponding EdgePrims color:

### The updateContours() function

function updateContours(hContour) % Update the text label colors drawnow % very important! levels = hContour.LevelList; labels = hContour.TextPrims; % undocumented/unsupported lines = hContour.EdgePrims; % undocumented/unsupported for idx = 1 : numel(labels) labelValue = str2double(labels(idx).String); lineIdx = find(abs(levels-labelValue)<10*eps, 1); % avoid FP errors using eps labels(idx).ColorData = lines(lineIdx).ColorData; % update the label color %labels(idx).Font.Size = 8; % update the label font size end drawnow % optional end

Note that in this function we don’t directly equate the numeric label values to the contour levels’ values: this would work well for integer values but would fail with floating-point ones. Instead I used a very small 10*eps tolerance in the numeric comparison.

Also note that I was careful to call drawnow at the top of the update function, in order to ensure that EdgePrims and TextPrims are updated when the function is called (this might not be the case before the call to drawnow). The final drawnow at the end of the function is optional: it is meant to reduce the flicker caused by the changing label colors, but it can be removed to improve the rendering performance in case of rapidly-changing contour plots.

Finally, note that I added a commented line that shows we can modify other label properties (in this case, the font size from 10 to 8). Feel free to experiment with other label properties.

### Putting it all together

The final stage is to call our new updateContours function directly, immediately after creating the contour plot. We also want to call updateContours asynchronously whenever the contour is redrawn, for example, upon a zoom/pan event, or when one of the relevant contour properties (e.g., LevelStep or *Data) changes. To do this, we add a callback listener to the contour object’s [undocumented] MarkedClean event that reruns our updateContours function:

[X,Y,Z] = peaks; [C,hContour] = contour(X,Y,Z, 'ShowText','on', 'LevelStep',1);   % Update the contours immediately, and also whenever the contour is redrawn updateContours(hContour); addlistener(hContour, 'MarkedClean', @(h,e)updateContours(hContour));

### Contour level values

As noted in my comment reply below, the contour lines (hContour.EdgePrims) correspond to the contour levels (hContour.LevelList).

For example, to make all negative contour lines dotted, you can do the following:

[C,hContour] = contour(peaks, 'ShowText','on', 'LevelStep',1); drawnow set(hContour.EdgePrims(hContour.LevelList<0), 'LineStyle', 'dotted');

Customized Matlab contour lines

As I noted on my previous post on contour plot customization, I am marking this article as “High risk of breaking in future Matlab versions“, not because of the basic functionality (being important enough I don’t presume it will go away anytime soon) but because of the property names: TextPrims, EdgePrims and FacePrims don’t seem to be very user-friendly property names. So far MathWorks has been very diligent in making its object properties have meaningful names, and so I assume that when the time comes to expose these properties, they will be renamed (perhaps to TextHandles, EdgeHandles and FaceHandles, or perhaps LabelHandles, LineHandles and FillHandles). For this reason, even if you find out in some future Matlab release that TextPrims, EdgePrims and FacePrims don’t exist, perhaps they still exist and simply have different names. Note that these properties have not changed their names or functionality in the past 3 years, so while it could well happen next year, it could also remain unchanged for many years to come. The exact same thing can be said for the MarkedClean event.

### Professional assistance anyone?

As shown by this and many other posts on this site, a polished interface and functionality is often composed of small professional touches, many of which are not exposed in the official Matlab documentation for various reasons. So if you need top-quality professional appearance/functionality in your Matlab program, or maybe just a Matlab program that is dependable, robust and highly-performant, consider employing my consulting services.

]]>
https://undocumentedmatlab.com/blog/customizing-contour-plots-part2/feed 6
Runtime code instrumentationhttps://undocumentedmatlab.com/blog/runtime-code-instrumentation https://undocumentedmatlab.com/blog/runtime-code-instrumentation#comments Thu, 28 Sep 2017 13:36:17 +0000 http://undocumentedmatlab.com/?p=7063

Related posts:
1. Creating a simple UDD class This article explains how to create and test custom UDD packages, classes and objects...
2. Customizing uiundo This article describes how Matlab's undocumented uiundo undo/redo manager can be customized...
3. Introduction to UDD UDD classes underlie many of Matlab's handle-graphics objects and functionality. This article introduces these classes....
4. Getting default HG property values Matlab has documented how to modify default property values, but not how to get the full list of current defaults. This article explains how to do this. ...

]]>
I regularly follow the MathWorks Pick-of-the-Week (POTW) blog. In a recent post, Jiro Doke highlighted Per Isakson’s tracer4m utility. Per is an accomplished Matlab programmer, who has a solid reputation in the Matlab user community for many years. His utility uses temporary conditional breakpoints to enable users to trace calls to their Matlab functions and class methods. This uses a little-known trick that I wish to highlight in this post.

tracer4m utility uses conditional breakpoints that evaluate but never become live

Matlab breakpoints are documented and supported functionality, and yet their documented use is typically focused at interactive programming in the Matlab editor, or as interactive commands that are entered in the Matlab console using the set of db* functions: dbstop, dbclear, dbstatus, dbstack etc. However, nothing prevents us from using these db* functions directly within our code.

For example, the dbstack function can help us diagnose the calling tree for the current function, in order to do action A if one of the calling ancestors was FunctionX, or to do action B otherwise (for example, to avoid nested recursions).

Similarly, we could add a programmatic call to dbstop in order to stop at a certain code location downstream (for debugging), if a certain condition happens upstream.

Per extended this idea very cleverly in tracer4m: conditional breakpoints evaluate a string in run-time: if the result is true (non-zero) then the code run is stopped at that location, but if it’s false (or zero) then the code run continues normally. To instrument calls to specific functions, Per created a function tracer() that logs the function call (using dbstack) and always returns the value false. He then dynamically created a string that contains a call to this new function and used the dbstop function to create a conditional breakpoint based on this function, something similar to this:

dbstop('in', filename, 'at', location, 'if', 'tracer()');

We can use this same technique for other purposes. For example, if we want to do some action (not necessarily log – perhaps do something else) when a certain code point is reached. The benefit here is that we don’t need to modify the code at all – we’re adding ad-hoc code pieces using the conditional breakpoint mechanism without affecting the source code. This is particularly useful when we do not have access to the source code (such as when it’s compiled or write-protected). All you need to do is to ensure that the instrumentation function always returns false so that the breakpoint does not become live and for code execution to continue normally.

The tracer4m utility is quite sophisticated in the sense that it uses mlint and smart regexp to parse the code and know which functions/methods occur on which line numbers and have which type (more details). In this sense, Per used undocumented functionality. I’m certain that Jiro was not aware of the dependency on undocumented features when he posted about the utility, so please don’t take this to mean that Jiro or MathWorks officially support this or any other undocumented functionality. Undocumented aspects are often needed to achieve top functionality, and I’m happy that the POTW blog highlights utilities based on their importance and merit, even if they do happen to use some undocumented aspect.

tracer4m‘s code also contains references to the undocumented profiler option -history, but this is not in fact used by the code itself, only in comments. I use this feature in my profile_history utility, which displays the function call/timing history in an interactive GUI window. This utility complements tracer4m by providing a lot more information, but this can result in a huge amount of information for large and/or long-running programs. In addition, tracer4m has the benefit of only logging those functions/methods that the user finds useful, rather than all the function call, which enables easier debugging when the relevant code area is known. In short, I wish I had known about tracer4m when I created profile_history. Now that I know about it, maybe I’ll incorporate some of its ideas into profile_history in order to make it more useful. Perhaps another moral of this is that we should actively monitor the POTW blog, because true gems are quite often highlighted there.

Function call timeline profiling

For anyone who missed the announcement in my previous post, I’m hosting a series of live webinars on advanced Matlab topics in the upcoming 2 weeks – I’ll be happy if you would join.

]]>
https://undocumentedmatlab.com/blog/runtime-code-instrumentation/feed 3
Sending HTML emails from Matlabhttps://undocumentedmatlab.com/blog/sending-html-emails-from-matlab https://undocumentedmatlab.com/blog/sending-html-emails-from-matlab#comments Wed, 02 Aug 2017 21:19:42 +0000 http://undocumentedmatlab.com/?p=6986

Related posts:
1. Uitable sorting Matlab's uitables can be sortable using simple undocumented features...
2. Uitable customization report Matlab's uitable can be customized in many different ways. A detailed report explains how. ...
3. Types of undocumented Matlab aspects This article lists the different types of undocumented/unsupported/hidden aspects in Matlab...
4. Multi-line uitable column headers Matlab uitables can present long column headers in multiple lines, for improved readability. ...

]]>
A few months ago I wrote about various tricks for sending email/text messages from Matlab. Unfortunately, Matlab only sends text emails by default and provides no documented way to send HTML-formatted emails. Text-only emails are naturally very bland and all mail clients in the past 2 decades support HTML-formatted emails. Today I will show how we can send such HTML emails from Matlab.

A quick recap: Matlab’s sendmail function uses Java (specifically, the standard javax.mail package) to prepare and send emails. The Java classes are extremely powerful and so there is no wonder that Mathworks chose to use them rather than reinventing the wheel. However, Matlab’s sendmail function only uses part of the functionality exposed by these classes (admittedly, the most important parts that deal with the basic mail-sending mechanism), and does not expose external hooks or input args that would enable the user to take full advantage of the more advanced features, HTML formatting included.

Only two small changes are needed in sendmail.m to support HTML formatting:

1. HTML formatting required calling the message-object’s setContent() method, rather than setText().
2. We need to specify 'text/html' as part of the message’s encoding

To implement these features, change the following (lines #119-130 in the original sendmail.m file of R2017a, changed lines highlighted):

% Construct the body of the message and attachments. body = formatText(theMessage); if numel(attachments) == 0 if ~isempty(charset) msg.setText(body, charset); else msg.setText(body); end else % Add body text. messageBodyPart = MimeBodyPart; if ~isempty(charset) messageBodyPart.setText(body, charset); ...

to this (changed lines highlighted):

% Construct the body of the message and attachments. body = formatText(theMessage); isHtml = ~isempty(body) && body(1) == '<'; % msg starting with '<' indicates HTMLif isHtml if isempty(charset) charset = 'text/html; charset=utf-8'; else charset = ['text/html; charset=' charset]; endendif numel(attachments) == 0 && ~isHtml if isHtml msg.setContent(body, charset); elseif ~isempty(charset) msg.setText(body, charset); else msg.setText(body); end else % Add body text. messageBodyPart = MimeBodyPart; if isHtml messageBodyPart.setContent(body, charset); elseif ~isempty(charset) messageBodyPart.setText(body, charset); ...

In addition, I also found it useful to remove the hard-coded 75-character line-wrapping in text messages. This can be done by changing the following (line #291 in the original sendmail.m file of R2017a):

maxLineLength = 75;

to this:

maxLineLength = inf; % or some other large numeric value

### Deployment

It’s useful to note two alternatives for making these fixes:

• Making the changes directly in %matlabroot%/toolbox/matlab/iofun/sendmail.m. You will need administrator rights to edit this file. You will also need to redo the fix whenever you install Matlab, either installation on a different machine, or installing a new Matlab release. In general, I discourage changing Matlab’s internal files because it is simply not very maintainable.
• Copying %matlabroot%/toolbox/matlab/iofun/sendmail.m into a dedicated wrapper function (e.g., sendEmail.m) that has a similar function signature and exists on the Matlab path. This has the benefit of working on multiple Matlab releases, and being copied along with the rest of our m-files when we install our Matlab program on a different computer. The downside is that our wrapper function will be stuck with the version of sendmail.m that we copied into it, and we’d lose any possible improvements that Mathworks may implement in future Matlab releases.

The basic idea for the second alternative, the sendEmail.m wrapper, is something like this (the top highlighted lines are the additions made to the original sendmail.m, with everything placed in sendEmail.m on the Matlab path):

function sendEmail(to,subject,theMessage,attachments)%SENDEMAIL Send e-mail wrapper (with HTML formatting) sendmail(to,subject,theMessage,attachments);  % The rest of this file is copied from %matlabroot%/toolbox/matlab/iofun/sendmail.m (with the modifications mentioned above): function sendmail(to,subject,theMessage,attachments) %SENDMAIL Send e-mail. % SENDMAIL(TO,SUBJECT,MESSAGE,ATTACHMENTS) sends an e-mail. TO is either a % character vector specifying a single address, or a cell array of character vector ...

We would then call the wrapper function as follows:

sendEmail('abc@gmail.com', 'email subject', 'regular text message'); % will send a regular text message sendEmail('abc@gmail.com', 'email subject', '<b><font color="blue">HTML-formatted</font> <i>message'); % HTML-formatted message

In this case, the code automatically infers HTML formatting based on whether the first character in the message body is a ‘<‘ character. Instead, we could just as easily have passed an additional input argument (isHtml) to our sendEmail wrapper function.

Hopefully, in some future Matlab release Mathworks will be kind enough to enable sending 21st-century HTML-formatted emails without needing such hacks. Until then, note that sendmail.m relies on standard non-GUI Java networking classes, which are expected to be supported far into the future, well after Java-based GUI may cease to be supported in Matlab. For this reason I believe that while it seems a bit tricky, the changes that I outlined in today’s post actually have a low risk of breaking in a future Matlab release.

Do you have some other advanced email feature that you use in your Matlab program by some crafty customization to sendmail? If so, please share it in a comment below.

]]>
https://undocumentedmatlab.com/blog/sending-html-emails-from-matlab/feed 1
Matlab compilation quirks – take 2https://undocumentedmatlab.com/blog/matlab-compilation-quirks-take-2 https://undocumentedmatlab.com/blog/matlab-compilation-quirks-take-2#respond Wed, 31 May 2017 18:00:42 +0000 http://undocumentedmatlab.com/?p=6919

Related posts:
1. Matlab compiler bug and workaround Both the Matlab compiler and the publish function have errors when parsing block-comments in Matlab m-code. ...
2. UDD Properties UDD provides a very convenient way to add customizable properties to existing Matlab object handles...
3. Disabling menu entries in deployed docked figures Matlab's standard menu items can and should be removed from deployed docked figures. This article explains how. ...
4. Handle Graphics Behavior HG behaviors are an important aspect of Matlab graphics that enable custom control of handle functionality. ...

]]>
Once again I would like to welcome guest blogger Hanan Kavitz of Applied Materials. Hanan posted a couple of guest posts here over the past few years, including a post last year about quirks with Matlab-compiled DLLs. Today Hanan will follow up on that post by discussing several additional quirks that they have encountered with Matlab compilations/deployment.

### Don’t fix it, if it ain’t broke…

In Applied Materials Israel (PDC) we use Matlab code for both algorithm development and deployment (production). As part of the dev-ops build system, which builds our product software versions, we build Matlab artifacts (binaries) from the Matlab source code.

A typical software version has several hundreds Matlab artifacts that are automatically rebuilt on a daily basis, and we have many such versions – totaling many thousands of compilations each day.

This process takes a long time, so we were looking for a way to make it more efficient.

The idea that we chose to implement sounds simple – take a single binary module in any software version (Ex. foo.exe – Matlab-compiled exe) and check it: if the source code for this module has not changed since the last compilation then simply don’t compile it, just copy it from previous software version repository. Since most of our code doesn’t change daily (some of it hasn’t changed in years), we can skip the compilation time of most binaries and just copy them from some repository of previously compiled binaries.

In a broader look, avoiding lengthy compilations cycles by not compiling unchanged code is a common programming practice, implemented by all modern compilers. For example, the ‘make’ utility uses a ‘makefile’ to check the time stamps of all dependencies of every object file in order to decide which object requires recompilation. In reality, this is not always the best solution as time stamps may be incorrect, but it works well in the vast majority of cases.

Coming back to Matlab, now comes the hard part – how could our build system know that nothing has changed in module X and that something has changed in module Y? How does it even know which source files it needs to ensure didn’t change?

The credit for the idea goes to my manager, Lior Cohen, as follows: You can actually check the dependency of a given binary after compilation. The basis of the solution is that a Matlab executable is in fact a compressed (zip) file. The idea is then to:

1. Compile the binary once
2. Unzip the binary and “see” all your dependencies (source files are encrypted and resources are not, but we only need the list of file names – not their content).
3. Now build a list of all your dependency files and compute the CRC value of each from the source control. Save it for the next time you are required to compile this module.
4. In the next compilation cycle, find this dependency list, review it, dependency source file at a time and make sure CRC of the dependency hasn’t changed since last time.
5. If no dependency CRC has changed, then copy the binary from the repository of previous software version, without compiling.
6. Otherwise, recompile the binary and rebuild the CRC list of all dependencies again, in preparation for the next compilation cycle.

That’s it! That simple? Well… not really – the reality is a bit more complex since there are many other dependencies that need to be checked. Some of them are:

1. Did the requested Matlab version of the binary change since the last compilation?
2. Did the compilation instructions themselves (we have a sort of ‘makefile’) change?

Basically, I implemented a policy that if anything changed, or if the dependency check itself failed, then we don’t take any chances and just compile this binary. Keeping in mind that this dependencies check and file copying is much faster than a Matlab compilation, we save a lot of actual compilation time using this method.

Bottom line: Given a software version containing hundreds of compilation instructions to execute and assuming not much has changed in the version (which is often the case), we skip over 90% of compilations altogether and only rebuild what really changed. The result is a version build that takes about half an hour, instead of many hours. Moreover, since the compilation process is working significantly less, we get fewer failures, fewer stuck or crashed mcc processes, and [not less importantly] less maintenance required by me.

Note that in our implementation we rely on the undocumented fact that Matlab binaries are in fact compressed zip archives. If and when a future Matlab release will change the implementation such that the binaries will no longer be zip archives, another way will need to be devised in order to ensure the consistency of the target executable with its dependent source files.

### Don’t kill it, if it ain’t bad…

I want to share a very weird issue I investigated over a year ago when using Matlab compiled exe. It started with a user showed me a Matlab compiled exe that didn’t run – I’m not talking about a regular Matlab exception: the process was crashing with an MS Windows popup window popping, stating something very obscure.

It was a very weird behavior that I couldn’t explain – the compiler seemed to work well but the compiled executable process kept crashing. Compiling completely different code showed the same behavior.

This issue has to do with the system compiler configuration that is being used. As you might know, when installing the Matlab compiler, before the first compilation is ever made, the user has to state the C compiler that the Matlab compiler should use in its compilation process. This is done by command ‘mbuild –setup’. This command asks the users to choose the C compiler and saves the configuration (batch file back then, xml in the newer versions of Matlab) in the user’s prefdir folder. At the time we were using Microsoft Visual C++ compiler 9.0 SP1.

The breakthrough in the investigation came when I ran mcc command with –verbose flag, which outputs much more compilation info than I would typically ever want… I discovered that although the target executable file had been created, a post compilation step failed to execute, while issuing a very cryptic error message:

mt.exe : general error c101008d: Failed to write the updated manifest to the resource of file “…”. Access is denied.

cryptic compilation error (click to zoom)

The failure was in one of the ‘post link’ commands in the configuration batch file – something obscure such as this:

set POSTLINK_CMDS2=mt.exe -outputresource: %MBUILD_OUTPUT_FILE_NAME%;%MANIFEST_RESOURCE% -manifest "%MANIFEST_FILE_NAME%"

This line of code takes an XML manifest file and inserts it into the generated binary file (additional details).

If you open a valid R2010a (and probably other old versions as well) Matlab-generated exe in a text editor you can actually see a small XML code embedded in it, while in a non-functioning exe I could not see this XML code.

So why would this command fail?

It turned out, as funny as it sounds, to be an antivirus issue – our IT department updated its antivirus policies and this ‘post link’ command suddenly became an illegal operation. Once our IT eased the policy, this command worked well again and the compiled executables stopped crashing, to our great joy.

]]>
https://undocumentedmatlab.com/blog/matlab-compilation-quirks-take-2/feed 0

Related posts:
1. uiundo – Matlab’s undocumented undo/redo manager The built-in uiundo function provides easy yet undocumented access to Matlab's powerful undo/redo functionality. This article explains its usage....
2. Undocumented Profiler options part 3 An undocumented feature of the Matlab Profiler can report call history timeline - part 3 of series. ...
3. Undocumented Profiler options part 4 Several undocumented features of the Matlab Profiler can make it much more useful - part 4 of series. ...
4. Pinning annotations to graphs Annotation object can be programmatically set at, and pinned-to, plot axes data points. ...

]]>
Matlab’s license function returns the primary license number/ID used by Matlab, but no information about the various toolboxes that may be installed. The ver function returns a bit more information, listing the version number and installation date of installed toolboxes (even user toolboxes, such as my IB-Matlab toolbox). However, no additional useful information is provided beyond that:

>> license ans = 123456 % actual number redacted   >> ver ---------------------------------------------------------------------------------------------------- MATLAB Version: 9.1.0.441655 (R2016b) MATLAB License Number: 123456 Operating System: Microsoft Windows 7 Professional Version 6.1 (Build 7601: Service Pack 1) Java Version: Java 1.7.0_60-b19 with Oracle Corporation Java HotSpot(TM) 64-Bit Server VM mixed mode ---------------------------------------------------------------------------------------------------- MATLAB Version 9.1 (R2016b) Curve Fitting Toolbox Version 3.5.4 (R2016b) Database Toolbox Version 7.0 (R2016b) Datafeed Toolbox Version 5.4 (R2016b) Financial Instruments Toolbox Version 2.4 (R2016b) Financial Toolbox Version 5.8 (R2016b) GUI Layout Toolbox Version 2.2.1 (R2015b) Global Optimization Toolbox Version 3.4.1 (R2016b) IB-Matlab - Matlab connector to InteractiveBrokers Version 1.89 Expires: 1-Apr-2018 Image Processing Toolbox Version 9.5 (R2016b) MATLAB Coder Version 3.2 (R2016b) MATLAB Report Generator Version 5.1 (R2016b) Optimization Toolbox Version 7.5 (R2016b) Parallel Computing Toolbox Version 6.9 (R2016b) Statistical Graphics Toolbox Version 1.2 Statistics and Machine Learning Toolbox Version 11.0 (R2016b)   >> v = ver v = 1×16 struct array with fields: Name Version Release Date   >> v(1) ans = struct with fields:   Name: 'Curve Fitting Toolbox' Version: '3.5.4' Release: '(R2016b)' Date: '25-Aug-2016'   >> v(8) ans = struct with fields:   Name: 'IB-Matlab - Matlab connector to InteractiveBrokers' Version: '1.89' Release: 'Expires: 1-Apr-2018' Date: '02-Feb-2017'

It is sometimes useful to know which license number “owns” which product/toolbox, and the expiration date is associated with each of them. Unfortunately, there is no documented way to retrieve this information in Matlab – the only documented way is to go to your account section on the MathWorks website and check there.

Luckily, there is a simpler way that can be used to retrieve additional information, from right inside Matlab, using matlab.internal.licensing.getFeatureInfo:

>> all_data = matlab.internal.licensing.getFeatureInfo all_data = 23×1 struct array with fields: feature expdate keys license_number entitlement_id   >> all_data(20) ans = struct with fields:   feature: 'optimization_toolbox' expdate: '31-mar-2018' keys: 0 license_number: '123456' entitlement_id: '1409891'   >> all_data(21) ans = struct with fields:   feature: 'optimization_toolbox' expdate: '07-mar-2017' keys: 0 license_number: 'DEMO' entitlement_id: '3749959'

As can be seen in this example, I have the Optimization toolbox licensed under my main Matlab license (123456 [actual number redacted]) until 31-mar-2018, and also licensed under a trial (DEMO) license that expires in 3 weeks. As long as a toolbox has any future expiration date, it will continue to function, so in this case I’m covered until March 2018.

We can also request information about a specific toolbox (“feature”):

>> data = matlab.internal.licensing.getFeatureInfo('matlab') data = 3×1 struct array with fields: feature expdate keys license_number entitlement_id   >> data(1) data = struct with fields:   feature: 'matlab' expdate: '31-mar-2018' keys: 0 license_number: '123456' entitlement_id: '1409891'

The drawback of this functionality is that it only provides information about MathWorks’ toolbox, not any user-provided toolboxes (such as my IB-Matlab connector, or MathWorks’ own GUI Layout toolbox). Also, some of the toolbox names may be difficult to understand (“gads_toolbox” apparently stands for the Global Optimization Toolbox, for example):

>> {all_data.feature} ans = 1×23 cell array Columns 1 through 4 'curve_fitting_toolbox' 'database_toolbox' 'datafeed_toolbox' 'distrib_computing_toolbox' Columns 5 through 8 'distrib_computing_toolbox' 'excel_link' 'fin_instruments_toolbox' 'financial_toolbox' Columns 9 through 15 'gads_toolbox' 'gads_toolbox' 'image_toolbox' 'image_toolbox' 'matlab' 'matlab' 'matlab' Columns 16 through 20 'matlab_coder' 'matlab_coder' 'matlab_report_gen' 'matlab_report_gen' 'optimization_toolbox' Columns 21 through 23 'optimization_toolbox' 'optimization_toolbox' 'statistics_toolbox'

A related undocumented builtin function is matlab.internal.licensing.getLicInfo:

% Information on a single toolbox/product: >> matlab.internal.licensing.getLicInfo('matlab') ans = struct with fields:   license_number: {'123456' 'Prerelease' 'T3749959'} expiration_date: {'31-mar-2018' '30-sep-2016' '07-mar-2017'}   % Information on multiple toolboxes/products: >> matlab.internal.licensing.getLicInfo({'matlab', 'image_toolbox'}) % cell array of toolbox/feature names ans = 1×2 struct array with fields: license_number expiration_date   % The full case-insensitive names of the toolboxes can also be used: >> matlab.internal.licensing.getLicInfo({'Matlab', 'Image Processing toolbox'}) ans = 1×2 struct array with fields: license_number expiration_date   % And here's how to get the full list (MathWorks products only): >> v=ver; data=matlab.internal.licensing.getLicInfo({v.Name}) data = 1×16 struct array with fields: license_number expiration_date

I have [still] not found any way to associate a user toolbox/product (such as my IB-Matlab) in a way that will report it in a unified manner with the MathWorks products. If anyone finds a way to do this, please do let me know.

p.s. – don’t even think of asking questions or posting comments on this website related to illegal uses or hacks of the Matlab license…

]]>
Parsing XML stringshttps://undocumentedmatlab.com/blog/parsing-xml-strings https://undocumentedmatlab.com/blog/parsing-xml-strings#comments Wed, 01 Feb 2017 09:52:45 +0000 http://undocumentedmatlab.com/?p=6838

Related posts:
1. Undocumented XML functionality Matlab's built-in XML-processing functions have several undocumented features that can be used by Java-savvy users...
2. Matlab-Java memory leaks, performance Internal fields of Java objects may leak memory - this article explains how to avoid this without sacrificing performance. ...
3. Types of undocumented Matlab aspects This article lists the different types of undocumented/unsupported/hidden aspects in Matlab...
4. Pause for the better Java's thread sleep() function is much more accurate than Matlab's pause() function. ...

]]>
I have recently consulted in a project where data was provided in XML strings and needed to be parsed in Matlab memory in an efficient manner (in other words, as quickly as possible). Now granted, XML is rather inefficient in storing data (JSON would be much better for this, for example). But I had to work with the given situation, and that required processing the XML.

I basically had two main alternatives:

• I could either create a dedicated string-parsing function that searches for a particular pattern within the XML string, or
• I could use a standard XML-parsing library to create the XML model and then parse its nodes

The first alternative is quite error-prone, since it relies on the exact format of the data in the XML. Since the same data can be represented in multiple equivalent XML ways, making the string-parsing function robust as well as efficient would be challenging. I was lazy expedient, so I chose the second alternative.

Unfortunately, Matlab’s xmlread function only accepts input filenames (of *.xml files), it cannot directly parse XML strings. Yummy!

The obvious and simple solution is to simply write the XML string into a temporary *.xml file, read it with xmlread, and then delete the temp file:

% Store the XML data in a temp *.xml file filename = [tempname '.xml']; fid = fopen(filename,'Wt'); fwrite(fid,xmlString); fclose(fid);   % Read the file into an XML model object xmlTreeObject = xmlread(filename);   % Delete the temp file delete(filename);   % Parse the XML model object ...

This works well and we could move on with our short lives. But cases such as this, where a built-in function seems to have a silly limitation, really fire up the investigative reporter in me. I decided to drill into xmlread to discover why it couldn’t parse XML strings directly in memory, without requiring costly file I/O. It turns out that xmlread accepts not just file names as input, but also Java object references (specifically, java.io.File, java.io.InputStream or org.xml.sax.InputSource). In fact, there are quite a few other inputs that we could use, to specify a validation parser etc. – I wrote about this briefly back in 2009 (along with other similar semi-documented input altermatives in xmlwrite and xslt).

In our case, we could simply send xmlread as input a java.io.StringBufferInputStream(xmlString) object (which is an instance of java.io.InputStream) or org.xml.sax.InputSource(java.io.StringReader(xmlString)):

% Read the xml string directly into an XML model object inputObject = java.io.StringBufferInputStream(xmlString); % alternative #1 inputObject = org.xml.sax.InputSource(java.io.StringReader(xmlString)); % alternative #2   xmlTreeObject = xmlread(inputObject);   % Parse the XML model object ...

If we don’t want to depend on undocumented functionality (which might break in some future release, although it has remained unchanged for at least the past decade), and in order to improve performance even further by passing xmlread‘s internal validity checks and processing, we can use xmlread‘s core functionality to parse our XML string directly. We can add a fallback to the standard (fully-documented) functionality, just in case something goes wrong (which is good practice whenever using any undocumented functionality):

try % The following avoids the need for file I/O: inputObject = java.io.StringBufferInputStream(xmlString); % or: org.xml.sax.InputSource(java.io.StringReader(xmlString)) try % Parse the input data directly using xmlread's core functionality parserFactory = javaMethod('newInstance','javax.xml.parsers.DocumentBuilderFactory'); p = javaMethod('newDocumentBuilder',parserFactory); xmlTreeObject = p.parse(inputObject); catch % Use xmlread's semi-documented inputObject input feature xmlTreeObject = xmlread(inputObject); end catch % Fallback to standard xmlread usage, using a temporary XML file:   % Store the XML data in a temp *.xml file filename = [tempname '.xml']; fid = fopen(filename,'Wt'); fwrite(fid,xmlString); fclose(fid);   % Read the file into an XML model object xmlTreeObject = xmlread(filename);   % Delete the temp file delete(filename); end   % Parse the XML model object ...
]]>
https://undocumentedmatlab.com/blog/parsing-xml-strings/feed 6
Quirks with parfor vs. forhttps://undocumentedmatlab.com/blog/quirks-with-parfor-vs-for https://undocumentedmatlab.com/blog/quirks-with-parfor-vs-for#comments Thu, 05 Jan 2017 17:15:48 +0000 http://undocumentedmatlab.com/?p=6821

Related posts:
1. Matlab mex in-place editing Editing Matlab arrays in-place can be an important technique for optimizing calculations. This article shows how to do it using Mex. ...
2. Preallocation performance Preallocation is a standard Matlab speedup technique. Still, it has several undocumented aspects. ...
3. Array resizing performance Several alternatives are explored for dynamic array growth performance in Matlab loops. ...
4. Matlab’s internal memory representation Matlab's internal memory structure is explored and discussed. ...

]]>
A few months ago, I discussed several tips regarding Matlab’s parfor command, which is used by the Parallel Computing Toolbox (PCT) for parallelizing loops. Today I wish to extend that post with some unexplained oddities when using parfor, compared to a standard for loop.

### Data serialization quirks

Dimitri Shvorob may not appear at first glance to be a prolific contributor on Matlab Central, but from the little he has posted over the years I regard him to be a Matlab power-user. So when Dimitri reports something, I take it seriously. Such was the case several months ago, when he contacted me regarding very odd behavior that he saw in his code: the for loop worked well, but the parfor version returned different (incorrect) results. Eventually, Dimitry traced the problem to something originally reported by Dan Austin on his Fluffy Nuke It blog.

The core issue is that if we have a class object that is used within a for loop, Matlab can access the object directly in memory. But with a parfor loop, the object needs to be serialized in order to be sent over to the parallel workers, and deserialized within each worker. If this serialization/deserialization process involves internal class methods, the workers might see a different version of the class object than the one seen in the serial for loop. This could happen, for example, if the serialization/deserialization method croaks on an error, or depends on some dynamic (or random) conditions to create data.

In other words, when we use data objects in a parfor loop, the data object is not necessarily sent “as-is”: additional processing may be involved under the hood that modify the data in a way that may be invisible to the user (or the loop code), resulting in different processing results of the parallel (parfor) vs. serial (for) loops.

For additional aspects of Matlab serialization/deserialization, see my article from 2 years ago (and its interesting feedback comments).

### Data precision quirks

The following section was contributed by guest blogger Lior Perlmuter-Shoshany, head algorithmician at a private equity fund.

In my work, I had to work with matrixes in the order of 109 cells. To reduce the memory footprint (and hopefully also improve performance), I decided to work with data of type single instead of Matlab’s default double. Furthermore, in order to speed up the calculation I use parfor rather than for in the main calculation. In the end of the run I am running a mini for-loop to see the best results.

What I discovered to my surprise is that the results from the parfor and for loop variants is not the same!

The following simplified code snippet illustrate the problem by calculating a simple standard-deviation (std) over the same data, in both single– and double-precision. Note that the loops are ran with only a single iteration, to illustrate the fact that the problem is with the parallelization mechanism (probably the serialization/deserialization parts once again), not with the distribution of iterations among the workers.

clear rng('shuffle','twister');   % Prepare the data in both double and single precision arr_double = rand(1,100000000); arr_single = single(arr_double);   % No loop - direct computation std_single0 = std(arr_single); std_double0 = std(arr_double);   % Loop #1 - serial for loop std_single = 0; std_double = 0; for i=1 std_single(i) = std(arr_single); std_double(i) = std(arr_double); end   % Loop #2 - parallel parfor loop par_std_single = 0; par_std_double = 0; parfor i=1 par_std_single(i) = std(arr_single); par_std_double(i) = std(arr_double); end   % Compare results of for loop vs. non-looped computation isForSingleOk = isequal(std_single, std_single0) isForDoubleOk = isequal(std_double, std_double0)   % Compare results of single-precision data (for vs. parfor) isParforSingleOk = isequal(std_single, par_std_single) parforSingleAccuracy = std_single / par_std_single   % Compare results of double-precision data (for vs. parfor) isParforDoubleOk = isequal(std_double, par_std_double) parforDoubleAccuracy = std_double / par_std_double

Output example :

isForSingleOk = 1 % <= true (of course!) isForDoubleOk = 1 % <= true (of course!)   isParforSingleOk = 0 % <= false (odd!) parforSingleAccuracy = 0.73895227413361 % <= single-precision results are radically different in parfor vs. for   isParforDoubleOk = 0 % <= false (odd!) parforDoubleAccuracy = 1.00000000000021 % <= double-precision results are almost [but not exactly] the same in parfor vs. for

From my testing, the larger the data array, the bigger the difference is between the results of single-precision data when running in for vs. parfor.

In other words, my experience has been that if you have a huge data matrix, it’s better to parallelize it in double-precision if you wish to get [nearly] accurate results. But even so, I find it deeply disconcerting that the results are not exactly identical (at least on R2015a-R2016b on which I tested) even for the native double-precision .

Hmmm… bug?

### Upcoming travels – Zürich & Geneva

I will shortly be traveling to clients in Zürich and Geneva, Switzerland. If you are in the area and wish to meet me to discuss how I could bring value to your work with some advanced Matlab consulting or training, then please email me (altmany at gmail):

• Zürich: January 15-17
• Geneva: January 18-21

Happy new year everybody!

]]>
https://undocumentedmatlab.com/blog/quirks-with-parfor-vs-for/feed 7
Sending email/text messages from Matlabhttps://undocumentedmatlab.com/blog/sending-email-text-messages-from-matlab https://undocumentedmatlab.com/blog/sending-email-text-messages-from-matlab#comments Wed, 07 Dec 2016 21:24:03 +0000 http://undocumentedmatlab.com/?p=6765

Related posts:
1. Types of undocumented Matlab aspects This article lists the different types of undocumented/unsupported/hidden aspects in Matlab...
2. Legend ‘-DynamicLegend’ semi-documented feature The built-in Matlab legend function has a very useful semi-documented feature for automatic dynamic update, which is explained here....
3. Undocumented XML functionality Matlab's built-in XML-processing functions have several undocumented features that can be used by Java-savvy users...
4. Inactive Control Tooltips & Event Chaining Inactive Matlab uicontrols cannot normally display their tooltips. This article shows how to do this with a combination of undocumented Matlab and Java hacks....

]]>
In this day and age, applications are expected to communicate with users by sending email/text messages to alert them about applicative events (“IBM stock purchased @\$99.99” or “House is on fire!”). Matlab has included the sendmail function to handle this for many years. Unfortunately, sendmail requires some tweaking to be useful on all but the most basic/insecure mail servers. Today’s post will hopefully fill the missing gaps.

None of the information I’ll present today is really new – it was all there already if you just knew what to search for online. But hopefully today’s post will concentrate all these loose ends in a single place, so it may have some value:

### Using a secure mail server

All modern mail servers use end-to-end TLS/SSL encryption. The sendmail function needs extra configuration to handle such connections, since it is configured for a non-encrypted connection by default. Here’s the code that does this for gmail, using SMTP server smtp.gmail.com and default port #465 (for other SMTP servers, see here):

setpref('Internet', 'SMTP_Server', 'smtp.gmail.com'); setpref('Internet', 'SMTP_Username', username); setpref('Internet', 'SMTP_Password', password);   props = java.lang.System.getProperties; props.setProperty('mail.smtp.auth', 'true'); % Note: 'true' as a string, not a logical value! props.setProperty('mail.smtp.starttls.enable', 'true'); % Note: 'true' as a string, not a logical value! props.setProperty('mail.smtp.socketFactory.port', '465'); % Note: '465' as a string, not a numeric value! props.setProperty('mail.smtp.socketFactory.class', 'javax.net.ssl.SSLSocketFactory');   sendmail(recipient, title, body, attachments); % e.g., sendmail('recipient@gmail.com', 'Hello world', 'What a nice day!', 'C:\images\sun.jpg')

All this is not enough to enable Matlab to connect to gmail’s SMTP servers. In addition, we need to set the Google account to allow access from “less secure apps” (details, direct link). Without this, Google will not allow Matlab to relay emails. Other mail servers may require similar server-side account configurations to enable Matlab’s access.

Note: This code snippet uses a bit of Java as you can see. Under the hood, all networking code in Matlab relies on Java, and sendmail is no exception. For some reason that I don’t fully understand, MathWorks chose to label the feature of using sendmail with secure mail servers as a feature that relies on “undocumented commands” and is therefore not listed in sendmail‘s documentation. Considering the fact that all modern mail servers are secure, this seems to make sendmail rather useless without the undocumented extension. I assume that TMW are well aware of this, which is the reason they posted a partial documentation in the form of an official tech-support answer. I hope that one day MathWorks will incorporate it into sendmail as optional input args, so that using sendmail with secure servers would become fully documented and officially supported.

### Emailing multiple recipients

To specify multiple email recipients, it is not enough to set sendmail‘s recipient input arg to a string with , or ; delimiters. Instead, we need to provide a cell array of individual recipient strings. For example:

sendmail({'recipient1@gmail.com','recipient2@gmail.com'}, 'Hello world', 'What a nice day!')

Note: this feature is actually fully documented in sendmail‘s doc-page, but for some reason I see that some users are not aware of it (to which it might be said: RTFM!).

### Sending text messages

With modern smartphones, text (SMS) messages have become rather outdated, as most users get push notifications of incoming emails. Still, for some users text messages may still be a useful. To send such messages, all we need is to determine our mobile carrier’s email gateway for SMS messages, and send a simple text message to that email address. For example, to send a text message to T-Mobile number 123-456-7890 in the US, simply email the message to 1234567890@tmomail.net (details).

Ke Feng posted a nice Matlab File Exchange utility that wraps this messaging for a wide variety of US carriers.

### User configuration panel

Many GUI programs contain configuration panels/tabs/windows. Enabling the user to set up their own email provider is a typical use-case for such a configuration. Naturally, you’d want your config panel not to display plain-text password, nor non-integer port numbers. You’d also want the user to be able to test the email connection.

Here’s a sample implementation for such a panel that I implemented for a recent project – I plan to discuss the implementation details of the password and port (spinner) controls in my next post, so stay tuned:

User configuration of emails in Matlab GUI (click to zoom-in)

]]>
https://undocumentedmatlab.com/blog/sending-email-text-messages-from-matlab/feed 5
Afterthoughts on implicit expansionhttps://undocumentedmatlab.com/blog/afterthoughts-on-implicit-expansion https://undocumentedmatlab.com/blog/afterthoughts-on-implicit-expansion#comments Wed, 30 Nov 2016 20:28:44 +0000 http://undocumentedmatlab.com/?p=6750

Related posts:
1. tic / toc – undocumented option Matlab's built-in tic/toc functions have an undocumented option enabling multiple nested clockings...
2. Plot LimInclude properties The plot objects' XLimInclude, YLimInclude, ZLimInclude, ALimInclude and CLimInclude properties are an important feature, that has both functional and performance implications....
3. Matrix processing performance Matrix operations performance is affected by internal subscriptions in a counter-intuitive way....
4. Performance: accessing handle properties Handle object property access (get/set) performance can be significantly improved using dot-notation. ...

]]>
Matlab release R2016b introduced implicit arithmetic expansion, which is a great and long-awaited natural expansion of Matlab’s arithmetic syntax (if you are still unaware of this or what it means, now would be a good time to read about it). This is a well-documented new feature. The reason for today’s post is that this new feature contains an undocumented aspect that should very well have been documented and even highlighted.

The undocumented aspect that I’m referring to is the fact that code that until R2016a produced an error, in R2016b produces a valid result:

% R2016a >> [1:5] + [1:3]' Error using + Matrix dimensions must agree.   % R2016b >> [1:5] + [1:3]' ans = 2 3 4 5 6 3 4 5 6 7 4 5 6 7 8 

This incompatibility is indeed documented, but not where it matters most (read on).

I first discovered this feature by chance when trying to track down a very strange phenomenon with client code that produced different numeric results on R2015b and earlier, compared to R2016a Pre-release. After some debugging the problem was traced to a code snippet in the client’s code that looked something like this (simplified):

% Ensure compatible input data try dataA + dataB; % this will (?) error if dataA, dataB are incompatible catch dataB = dataB'; end

The code snippet relied on the fact that incompatible data (row vs. col) would error when combined, as it did up to R2015b. But in R2016a Pre-release it just gave a valid numeric matrix, which caused numerically incorrect results downstream in the code. The program never crashed, so everything appeared to be in order, it just gave different numeric results. I looked at the release notes and none of the mentioned release incompatibilities appeared relevant. It took me quite some time, using side-by-side step-by-step debugging on two separate instances of Matlab (R2015b and R2016aPR) to trace the problem to this new feature.

This implicit expansion feature was removed from the official R2016a release for performance reasons. This was apparently fixed in time for R2016b’s release.

I’m totally in favor of this great new feature, don’t get me wrong. I’ve been an ardent user of bsxfun for many years and (unlike many) have even grown fond of it, but I still find the new feature to be better. I use it wherever there is no significant performance penalty, a need to support older Matlab releases, or a possibility of incorrect results due to dimensional mismatch.

### So what’s my point?

What I am concerned about is that I have not seen the new feature highlighted as a potential backward compatibility issue in the documentation or the release notes. Issues of far lesser importance are clearly marked for their backward incompatibility in the release notes, but not this important major change. A simple marking of the new feature with the warning icon () and in the “Functionality being removed or changed” section would have saved my client and me a lot of time and frustration.

MathWorks are definitely aware of the potential problems that the new feature might cause in rare use cases such as this. As Steve Eddins recently noted, there were plenty of internal discussions about this very thing. MathWorks were careful to ensure that the feature’s benefits far outweigh its risks (and I concur). But this also highlights the fact that MathWorks were fully aware that in some rare cases it might indeed break existing code. For those cases, I believe that they should have clearly marked the incompatibility implications in the release notes and elsewhere.

I have several clients who scour Matlab’s release notes before each release, trying to determine the operational risk of a Matlab upgrade. Having a program that returns different results in R2016b compared to R2016a, without being aware of this risk, is simply unacceptable to them, and leaves users with a disinclination to upgrade Matlab, to MathWorks’ detriment.

MathWorks in general are taking a very serious methodical approach to compatibility issues, and are clearly investing a lot of energy in this (a recent example). It’s too bad that sometimes this chain is broken. I find it a pity, and think that this can still be corrected in the online doc pages. If and when this is fixed, I’ll be happy to post an addendum here.

In my humble opinion from the backbenches, increasing the transparency on compatibility issues and open bugs will increase user confidence and result in greater adoption and upgrades of Matlab. Just my 2 cents…

Today MathWorks added the following compatibility warning to the release notes (R2016b, Mathematics section, first item) – thanks for listening MathWorks

]]>
https://undocumentedmatlab.com/blog/afterthoughts-on-implicit-expansion/feed 12
Customizing axes part 5 – origin crossover and labelshttps://undocumentedmatlab.com/blog/customizing-axes-part-5-origin-crossover-and-labels https://undocumentedmatlab.com/blog/customizing-axes-part-5-origin-crossover-and-labels#comments Wed, 27 Jul 2016 17:00:02 +0000 http://undocumentedmatlab.com/?p=6564

Related posts:
1. Customizing axes rulers HG2 axes can be customized in numerous useful ways. This article explains how to customize the rulers. ...
2. Customizing axes part 2 Matlab HG2 axes can be customized in many different ways. This article explains some of the undocumented aspects. ...
3. Undocumented scatter plot jitter Matlab's scatter plot can automatically jitter data to enable better visualization of distribution density. ...
4. HG2 update HG2 appears to be nearing release. It is now a stable mature system. ...

]]>
When HG2 graphics was finally released in R2014b, I posted a series of articles about various undocumented ways by which we can customize Matlab’s new graphic axes: rulers (axles), baseline, box-frame, grid, back-drop, and other aspects. Today I extend this series by showing how we can customize the axes rulers’ crossover location.

Non-default axes crossover location

### The documented/supported stuff

Until R2015b, we could only specify the axes’ YAxisLocation as 'left' (default) or 'right', and XAxisLocation as 'bottom' (default) or 'top'. For example:

x = -2*pi : .01 : 2*pi; plot(x, sin(x)); hAxis = gca; hAxis.YAxisLocation = 'left'; % 'left' (default) or 'right' hAxis.XAxisLocation = 'bottom'; % 'bottom' (default) or 'top'

Default axis locations: axes crossover is non-fixed

The crossover location is non-fixed in the sense that if we zoom or pan the plot, the axes crossover will remain at the bottom-left corner, which changes its coordinates depending on the X and Y axes limits.

Since R2016a, we can also specify 'origin' for either of these properties, such that the X and/or Y axes pass through the chart origin (0,0) location. For example, move the YAxisLocation to the origin:

hAxis.YAxisLocation = 'origin';

Y-axis location at origin: axes crossover at 0 (fixed), -1 (non-fixed)

And similarly also for XAxisLocation:

hAxis.XAxisLocation = 'origin';

X and Y-axis location at origin: axes crossover fixed at (0,0)

The axes crossover location is now fixed at the origin (0,0), so as we move or pan the plot, the crossover location changes its position in the chart area, without changing its coordinates. This functionality has existed in other graphic packages (outside Matlab) for a long time and until now required quite a bit of coding to emulate in Matlab, so I’m glad that we now have it in Matlab by simply updating a single property value. MathWorks did a very nice job here of dynamically updating the axles, ticks and labels as we pan (drag) the plot towards the edges – try it out!

### The undocumented juicy stuff

So far for the documented stuff. The undocumented aspect is that we are not limited to using the (0,0) origin point as the fixed axes crossover location. We can use any x,y crossover location, using the FirstCrossoverValue property of the axes’ hidden XRuler and YRuler properties. In fact, we could do this since R2014b, when the new HG2 graphics engine was released, not just starting in R2016a!

% Set a fixed crossover location of (pi/2,-0.4) hAxis.YRuler.FirstCrossoverValue = pi/2; hAxis.XRuler.FirstCrossoverValue = -0.4;

Custom fixed axes crossover location at (π/2,-0.4)

For some reason (bug?), setting XAxisLocation/YAxisLocation to ‘origin’ has no visible effect in 3D plots, nor is there any corresponding ZAxisLocation property. Luckily, we can set the axes crossover location(s) in 3D plots using FirstCrossoverValue just as easily as for 2D plots. The rulers also have a SecondCrossoverValue property (default = -inf) that controls the Z-axis crossover, as Yaroslav pointed out in a comment below. For example:

N = 49; x = linspace(-10,10,N); M = peaks(N); mesh(x,x,M);

Default crossover locations at (-10,±10,-10)

hAxis.XRuler.FirstCrossoverValue = 0; % X crossover with Y axis hAxis.YRuler.FirstCrossoverValue = 0; % Y crossover with X axis hAxis.ZRuler.FirstCrossoverValue = 0; % Z crossover with X axis hAxis.ZRuler.SecondCrossoverValue = 0; % Z crossover with Y axis

Custom fixed axes crossover location at (0,0,-10)

hAxis.XRuler.SecondCrossoverValue = 0; % X crossover with Z axis hAxis.YRuler.SecondCrossoverValue = 0; % Y crossover with Z axis

Custom fixed axes crossover location at (0,0,0)

### Labels

Users will encounter the following unexpected behavior (bug?) when using either the documented *AxisLocation or the undocumented FirstCrossoverValue properties: when setting an x-label (using the xlabel function, or the internal axes properties), the label moves from the center of the axes (as happens when XAxisLocation=’top’ or ‘bottom’) to the right side of the axes, where the secondary label (e.g., exponent) usually appears, whereas the secondary label is moved to the left side of the axis:

Unexpected label positions

In such cases, we would expect the labels locations to be reversed, with the main label on the left and the secondary label in its customary location on the right. The exact same situation occurs with the Y labels, where the main label unexpectedly appears at the top and the secondary at the bottom. Hopefully MathWorks will fix this in the next release (it is probably too late to make it into R2016b, but hopefully R2017a). Until then, we can simply switch the strings of the main and secondary label to make them appear at the expected locations:

% Switch the Y-axes labels: ylabel(hAxis, '\times10^{3}'); % display secondary ylabel (x10^3) at top set(hAxis.YRuler.SecondaryLabel, 'Visible','on', 'String','main y-label'); % main label at bottom   % Switch the X-axes labels: xlabel(hAxis, '2^{nd} label'); % display secondary xlabel at right set(hAxis.XRuler.SecondaryLabel, 'Visible','on', 'String','xlabel'); % main label at left

As can be seen from the screenshot, there’s an additional nuisance: the main label appears a bit larger than the axes font size (the secondary label uses the correct font size). This is because by default Matlab uses a 110% font-size for the main axes label, ostensibly to make them stand out. We can modify this default factor using the rulers’ hidden LabelFontSizeMultiplier property (default=1.1). For example:

hAxis.YRuler.LabelFontSizeMultiplier = 1; % use 100% font-size (same as tick labels) hAxis.XRuler.LabelFontSizeMultiplier = 0.8; % use 80% (smaller than standard) font-size

Note: I described the ruler objects in my first article of the axes series. Feel free to read it for more ideas on customizing the axes rulers.

]]>
https://undocumentedmatlab.com/blog/customizing-axes-part-5-origin-crossover-and-labels/feed 5
rmfield performancehttps://undocumentedmatlab.com/blog/rmfield-performance https://undocumentedmatlab.com/blog/rmfield-performance#comments Wed, 25 May 2016 07:00:48 +0000 http://undocumentedmatlab.com/?p=6427

Related posts:
1. tic / toc – undocumented option Matlab's built-in tic/toc functions have an undocumented option enabling multiple nested clockings...
2. Solving a MATLAB bug by subclassing Matlab's Image Processing Toolbox's impoint function contains an annoying bug that can be fixed using some undocumented properties....
3. Plot LimInclude properties The plot objects' XLimInclude, YLimInclude, ZLimInclude, ALimInclude and CLimInclude properties are an important feature, that has both functional and performance implications....
4. Class object creation performance Performance aspects of Matlab class object creation are discussed, with specific suggestions. ...

]]>
Once again I would like to introduce guest blogger Hanan Kavitz of Applied Materials. Several months ago Hanan discussed some quirks with compiled Matlab DLLs. Today Hanan will discuss how they overcame a performance bottleneck with Matlab’s builtin rmfield function, exemplifying the general idea that we can sometimes improve performance by profiling the core functionality that causes a performance hotspot and optimizing it, even when it is part of a builtin Matlab function. For additional ideas of improving Matlab peformance, search this blog for “Performance” articles, and/or get the book “Accelerating MATLAB Performance“.

I’ve been using Matlab for many years now and from time to time I need to profile low-throughput code. When I profile this code sometimes I realize that a computational ‘bottleneck’ is due to a builtin Matlab function (part of the core language). I can often find ways to accelerate such builtin functions and get significant speedup in my code.

I recently found Matlab’s builtin rmfield function being too slow for my needs. It works great when one needs to remove a few fields from a small structure, but in our case we needed to remove thousands of fields from a structure containing about 5000 fields – and this is executed in a function that is called many times inside an external loop. The program was significantly sluggish.

It started when a co-worker asked me to look at a code that looked just slightly more intelligent than this:

for i = 1:5000 myStruct = rmfield(myStruct,fieldNames{i}); end

Running this code within a tic/toc pair yielded the following results:

>> tic; myFunc(); t1 = toc t1 = 25.7713

In my opinion 25.77 secs for such a simple functionality seems like an eternity…

The obvious thing was to change the code to the documented faster (vectorized) version:

>> tic; myStruct = rmfield(myStruct,fieldNames); t2 = toc t2 = 0.6097

This is obviously much better but since rmfield is called many times in my application, I needed something even better. So I profiled rmfield and was not happy with the result.

The original code of rmfield (%matlabroot%/toolbox/matlab/datatypes/rmfield.m) looks something like this (I deleted some non-essential code for brevity):

function t = rmfield(s,field)   % get fieldnames of struct f = fieldnames(s);   % Determine which fieldnames to delete. idxremove = []; for i=1:length(field) j = find(strcmp(field{i},f) == true); idxremove = [idxremove;j]; end   % set indices of fields to keep idxkeep = 1:length(f); idxkeep(idxremove) = [];   % remove the specified fieldnames from the list of fieldnames. f(idxremove,:) = [];   % convert struct to cell array c = struct2cell(s);   % find size of cell array sizeofarray = size(c); newsizeofarray = sizeofarray;   % adjust size for fields to be removed newsizeofarray(1) = sizeofarray(1) - length(idxremove);   % rebuild struct t = cell2struct(reshape(c(idxkeep,:),newsizeofarray),f);

When I profiled the code, the highlighted row was the bottleneck I was looking for.

First, I noticed the string comparison equals to true part – while '==true' is not the cause of the bottleneck, it does leave an impression of bad coding style Perhaps this code was created as some apprentice project, which might also explain its suboptimal performance.

The real performance problem here is that for each field that we wish to remove, rmfield compares it to all existing fields to find its location in a cell array of field names. This is algorithmically inefficient and makes the code hard to understand (just try – it took me hard, long minutes).

So, I created a variant of rmfield.m called fast_rmfield.m, as follows (again, omitting some non-essential code):

function t = fast_rmfield(s,field)   % get fieldnames of struct f = fieldnames(s); [f,ia] = setdiff(f,field,'R2012a');   % convert struct to cell array c = squeeze(struct2cell(s));   % rebuild struct t = cell2struct(c(ia,:),f)';

This code is much shorter, easier to explain and maintain, but also (and most importantly) much faster:

>> tic; myStruct = fast_rmfield(myStruct,fieldNames); t3 = toc t3 = 0.0302   >> t2/t3 ans = 20.1893

This resulted in a speedup of ~850x compared to the original version (of 25.77 secs), and ~20x compared to the vectorized version. A nice improvement in my humble opinion…

The point in all this is that we can and should rewrite Matlab builtin functions when they are too slow for our needs, whether it is found to be an algorithmic flaw (as in this case), extraneous sanity checks (as in the case of ismember or datenum), bad default parameters (as in the case of fopen/fwrite or scatter), or merely slow implementation (as in the case of save, cellfun, or the conv family of functions).

A good pattern is to save such code pieces in file names that hint to the original code. In our case, I used fast_rmfield to suggest that it is a faster alternative to rmfield.

Do you know of any other example of a slow implementation in a built-in Matlab function that can be optimized? If so, please leave a comment below.

]]>
https://undocumentedmatlab.com/blog/rmfield-performance/feed 5
Viewing saved profiling resultshttps://undocumentedmatlab.com/blog/viewing-saved-profiling-results https://undocumentedmatlab.com/blog/viewing-saved-profiling-results#respond Wed, 18 May 2016 18:00:37 +0000 http://undocumentedmatlab.com/?p=6421

Related posts:
1. Plot performance Undocumented inner plot mechanisms can significantly improve plotting performance ...
2. Undocumented Profiler options part 2 Several undocumented features of the Matlab Profiler can make it much more useful - part 2 of series. ...
3. Undocumented Profiler options part 3 An undocumented feature of the Matlab Profiler can report call history timeline - part 3 of series. ...
4. Undocumented Profiler options part 4 Several undocumented features of the Matlab Profiler can make it much more useful - part 4 of series. ...

]]>
Many Matlab users know and utilize Matlab’s built-in Profiler tool to identify performance bottlenecks and code-coverage issues. Unfortunately, not many are aware of the Profiler’s programmatic interface. In past articles as well as my performance book I explained how we can use this programmatic interface to save profiling results and analyze it offline. In fact, I took this idea further and even created a utility (profile_history) that displays the function call timeline in a standalone Matlab GUI, something that is a sorely missed feature in the built-in profiler:

Function call timeline profiling (click for full-size image)

Today I will discuss a related undocumented feature of the Profiler: loading and viewing pre-saved profiling results.

Matlab’s syntax for returning the detailed profiling results in a data struct is clearly documented in the profile function’s doc page. Although the documentation does not explain the resulting struct and sub-struct fields, they have meaningful names and we can relatively easily infer what each of them means (I added a few annotation comments for clarity):

>> profile('on','-history') >> surf(peaks); drawnow >> profile('off') >> profData = profile('info') profData = FunctionTable: [26x1 struct] FunctionHistory: [2x56 double] ClockPrecision: 4.10517962829241e-07 ClockSpeed: 2501000000 Name: 'MATLAB' Overhead: 0   >> profData.FunctionTable(1) ans = CompleteName: 'C:\Program Files\Matlab\R2016a\toolbox\matlab\specgraph\peaks.m>peaks' FunctionName: 'peaks' FileName: 'C:\Program Files\Matlab\R2016a\toolbox\matlab\specgraph\peaks.m' Type: 'M-function' Children: [1x1 struct] Parents: [0x1 struct] ExecutedLines: [9x3 double] IsRecursive: 0 TotalRecursiveTime: 0 PartialData: 0 NumCalls: 1 TotalTime: 0.0191679078068094   >> profData.FunctionTable(1).Children ans = Index: 2 % index in profData.FunctionTable array NumCalls: 1 TotalTime: 0.00136415141013509   >> profData.FunctionTable(1).ExecutedLines % line number, number of calls, duration in secs ans = 43 1 0.000160102031282782 44 1 2.29890096200918e-05 45 1 0.00647592190637408 56 1 0.0017093970724654 57 1 0.00145036019621044 58 1 0.000304193859437286 60 1 4.39254290955326e-05 62 1 3.44835144301377e-05 63 1 0.000138755093778411   >> profData.FunctionHistory(:,1:5) ans = 0 0 1 1 0 % 0=enter, 1=exit 1 2 2 1 6 % index in profData.FunctionHistory array

As we can see, this is pretty intuitive so far.

If we wish to save these results results in a file and later load and display them in the Profiler’s visualization browser, then we need to venture deeper into undocumented territory. It seems that while retrieving the profiling results (via profile(‘info’)) is fully documented, doing the natural complementary action (namely, loading this data into the viewer) is not. For the life of me I cannot understand the logic behind this decision, but that’s the way it is.

Luckily, the semi-documented built-in function profview does exactly what we need: profview accepts 2 input args (function name and the profData struct) and displays the resulting profiling info. The first input arg (function name) accepts either a string (e.g., 'peaks' or 'view>isAxesHandle'), or the numeric value 0 which signifies the home (top-level) page:

profView(0, profData); % display profiling home (top-level) page profview('peaks', profData); % display a specific profiling page

I use the 0 input value much more frequently than the string inputs, because I often don’t know which functions exactly were profiled, and starting at the home page enables me to easily drill-down the profiling results interactively.

Things get slightly complicated if we try to load saved profiling results from a different computer. If the other computer has exactly the same folder structure as our computer, and all our Matlab functions reside in exactly the same disk folders/path, then everything will work out of the box. The problem is that in general the other computer will have the functions in different folders. When we then try to load the profData on our computer, it will not find the associated Matlab functinos in order to display the line-by-line profiling results. We will only see the profiling data at the function level, not line level. This significantly reduces the usefulness of the profiling data. The Profiler page will display the following error message:

This file was modified during or after profiling. Function listing disabled.

We can solve this problem in either of two ways:

1. Modify our profData to use the correct folder path on the local computer, rather than the other computer’s path (which is invalid on the local computer). For example:
% Save the profData on computer #1: profData = profile('info'); save('profData.mat', 'profData');   % Load the profData on computer #2: fileData = load('profData.mat'); profData = fileData.profData; path1 = 'N:\Users\Juan\programs\myProgram'; path2 = 'C:\Yair\consulting\clients\Intel\code'; for idx = 1 : numel(profData.FunctionTable) funcData = profData.FunctionTable(idx); funcData.FileName = strrep(funcData.FileName, path1, path2); funcData.CompleteName = strrep(funcData.CompleteName, path1, path2); profData.FunctionTable(idx) = funcData; end % note: this loop can be vectorized if you wish
2. As an alternative, we can modify Matlab’s profview.m function (%matlabroot%/toolbox/matlab/codetools/profview.m) to search for the function’s source code in the current Matlab path, if the specified direct path is not found (note that changing profview.m may require administrator priviledges). For example, the following is the code from R2016a’s profview.m file, line #506:
 % g894021 - Make sure the MATLAB code file still exists if ~exist(fullName, 'file') [~,fname,fext] = fileparts(fullName); % Yair fname = which([fname fext]); % Yair if isempty(fname) % Yair mFileFlag = 0; else % Yair fullName = fname; % Yair end % Yair end

These two workarounds complement each other: the first workaround does not require changing any installed Matlab code, and so is platform- and release-independent, but would require rerunning the code snippet for each and every profiling data file that we receive from external computers. On the other hand, the second workaround is a one-time operation that should work for multiple saved profiling results, although we would need to redo it whenever we install Matlab.

Modifying the profview.m function can be used for different improvements as well.

For example, several years ago I explained how this function can be modified to display 1 ms timing resolutions, rather than the default 10 mS.

Another customization that I often do after I install Matlab is to change the default setting of truncating function lines longer than 40 characters – I typically modify this to 60 or 80 (depending on the computer monitor’s size…). All we need to do is to update the truncateDisplayName sub-function within profview.m as follows (taken from R2016a again, line #1762):

function shortFileName = truncateDisplayName(longFileName,maxNameLen) %TRUNCATEDISPLAYNAME Truncate the name if it gets too long maxNameLen = max(60,maxNameLen); % YairshortFileName = escapeHtml(longFileName); if length(longFileName) > maxNameLen, shortFileName = char(com.mathworks.util.FileUtils.truncatePathname( ... shortFileName, maxNameLen)); end

You can see additional undocumented profiling features in the “Related posts” section below, as well as in Chapter 2 of my book “Accelerating MATLAB Performance“.

Do you have any other customization to the profiling results? If so, please share it in a comment.

]]>
https://undocumentedmatlab.com/blog/viewing-saved-profiling-results/feed 0
Convolution performancehttps://undocumentedmatlab.com/blog/convolution-performance https://undocumentedmatlab.com/blog/convolution-performance#comments Wed, 03 Feb 2016 19:00:35 +0000 http://undocumentedmatlab.com/?p=6256

Related posts:
1. tic / toc – undocumented option Matlab's built-in tic/toc functions have an undocumented option enabling multiple nested clockings...
2. Plot LimInclude properties The plot objects' XLimInclude, YLimInclude, ZLimInclude, ALimInclude and CLimInclude properties are an important feature, that has both functional and performance implications....
3. Matrix processing performance Matrix operations performance is affected by internal subscriptions in a counter-intuitive way....
4. Preallocation performance Preallocation is a standard Matlab speedup technique. Still, it has several undocumented aspects. ...

]]>
MathWorks’ latest MATLAB Digest (January 2016) featured my book “Accelerating MATLAB Performance“. I am deeply honored and appreciative of MathWorks for this.

I would like to dedicate today’s post to a not-well-known performance trick from my book, that could significantly improve the speed when computing the convolution of two data arrays. Matlab’s internal implementation of convolution (conv, conv2 and convn) appears to rely on a sliding window approach, using implicit (internal) multithreading for speed.

However, this can often be sped up significantly if we use the Convolution Theorem, which states in essence that conv(a,b) = ifft(fft(a,N) .* fft(b,N)), an idea proposed by Bruno Luong. In the following usage example we need to remember to zero-pad the data to get comparable results:

% Prepare the input vectors (1M elements each) x = rand(1e6,1); y = rand(1e6,1);   % Compute the convolution using the builtin conv() tic, z1 = conv(x,y); toc => Elapsed time is 360.521187 seconds.   % Now compute the convolution using fft/ifft: 780x faster! n = length(x) + length(y) - 1; % we need to zero-pad tic, z2 = ifft(fft(x,n) .* fft(y,n)); toc => Elapsed time is 0.463169 seconds.   % Compare the relative accuracy (the results are nearly identical) disp(max(abs(z1-z2)./abs(z1))) => 2.75200348450538e-10

This latest result shows that the results are nearly identical, up to a tiny difference, which is certainly acceptable in most cases when considering the enormous performance speedup (780x in this specific case). Bruno’s implementation (convnfft) is made even more efficient by using MEX in-place data multiplications, power-of-2 FFTs, and use of GPU/Jacket where available.

It should be noted that the builtin Matlab functions can still be faster for relatively small data arrays, or if your machine has a large number of CPU cores and free memory that Matlab’s builtin conv* functions can utilize, and of course also depending on the Matlab release. So, your mileage might well vary. But given the significant speedup potential, I contend that you should give it a try and see how well it performs on your specific system and data.

]]>
https://undocumentedmatlab.com/blog/convolution-performance/feed 6
Graphic sizing in Matlab R2015bhttps://undocumentedmatlab.com/blog/graphic-sizing-in-matlab-r2015b https://undocumentedmatlab.com/blog/graphic-sizing-in-matlab-r2015b#comments Wed, 20 Jan 2016 18:00:31 +0000 http://undocumentedmatlab.com/?p=6244

Related posts:
1. HG2 update HG2 appears to be nearing release. It is now a stable mature system. ...
2. Modifying default toolbar/menubar actions The default Matlab figure toolbar and menu actions can easily be modified using simple pure-Matlab code. This article explains how....
3. FIG files format FIG files are actually MAT files in disguise. This article explains how this can be useful in Matlab applications....
4. A couple of internal Matlab bugs and workarounds A couple of undocumented Matlab bugs have simple workarounds. ...

]]>
I would like to introduce Daniel Dolan of Sandia National Laboratories. Dan works on a variety of data analysis projects in Matlab, and is an active lurker on MATLAB Central. Dan has a habit of finding interesting bugs for the Mac version of Matlab. Today he will discuss graphic sizing in Matlab and important changes that occurred in release R2015b.

Matlab-generated graphics are often not displayed at their requested size. This problem has been known for some time and has a well-known solution: setting the root object’s ScreenPixelsPerInch property to the display’s actual DPI (dots per inch) value. Release R2015b no longer supports this solution, creating problems for publication graphics and general readability.

Physical sizing in R2015a vs. R2015b (click for full-size)

### Physical sizing

Matlab supports graphic sizing in various physical units: inches, centimeters, and points. For example:

figure; axes('Box','on', 'Units','inches','Position',[0.3 0.3 4 4]);

requests to display an axes having square sizes measuring exactly 4″ (101.6 mm) each. It is evident, however, that the displayed axes is smaller than 4″. The mismatch between requested and physical size depends on the display and operating system — go ahead, try it on your system. The problem is particularly severe on Mac laptops, presumably even worse for those with Retina displays.

The problem is that Matlab cannot determine pixel size, which varies from one display to the other. Generating a figure spanning a particular number of pixels (e.g., 1024 x 768) is easy, but absolute physical units requires a conversion factor called ScreenPixelsPerInch, which is a root property (see related post on setting/getting default graphics property values):

DPI = 110; % dots per inch for my 27" Apple Cinema Display set(0, 'ScreenPixelsPerInch',DPI); % all releases prior to R2015b set(groot,'ScreenPixelsPerInch',DPI); % R2014b through R2015a

DPI values tend to be higher for laptops, usually in the 120-130 range. Retina displays are supposed to be >300 DPI, but I have not been able to test that myself.

There are several ways to determine the correct DPI setting for a particular display. It may be available in the hardware specifications, and it can be calculated from the diagonal size and the number of pixels. Unfortunately these methods are not always reliable. If you really care about physical sizing, the best approach is to actually calibrate your display. There are tools for doing this at Matlab Central, but it’s not hard to do manually:

• Create a figure.
• Manually resize the figure to match a convenient width. I often use a piece of US letter paper as 8.5″ guide on the display.
• Determine the width of the figure in pixels:
set(gcf,'Units','pixels'); pos = get(gcf,'Position'); width = 8.5; % inches DPI = pos(3) / width;

I usually apply the DPI settings in my startup file so that Matlab begins with a calibrated display.

### What changed in 2015b?

ScreenPixelsPerInch is a read-only property in R2015b, so display calibration no longer works. The following sequence of commands:

figure('Units','inches', 'PaperPositionMode','auto', 'Position',[0 0 4 4]); set(gcf, 'MenuBar','none', 'ToolBar','none', 'DockControls','off', 'NumberTitle','off'); axes('FontUnits','points', 'FontSize',10); image

now renders differently in R2015b than does for a calibrated display in R2015a. Differences between the two outputs are shown in the screenshot at the top of this post. The grid behind the figures was rendered at 8.5″ x 8.5″ inches on my display; if your browser’s zoom level isn’t 100%, it may appear larger or smaller.

A side effect of improper graphic sizing is that text is difficult to read — the uncalibrated axes labels are clearly smaller than 10 points. These examples were rendered on ~110 DPI display. Matlab assumes that Macs use 72 DPI (96 DPI on Windows), so graphics appear at 65% of the request size.

The loss of ScreenPixelsPerInch as an adjustable setting strongly affects anyone using Matlab for publication graphics. Scientific and engineering journals are extremly strict about figure widths. With a calibrated screen, figure appear exactly as they will when printed to a file (usually EPS or PDF). Figures are often made as small as possible to and densely packed to save journal space, and accurate sized display helps the author determine legibility. Displaying accurately sized graphics is very difficult in R2015b, which is unfortunate given the many enhancements in this release.

Developers who create graphical interfaces for other users should also care about this change. A common complaint I get is that text and control labels is too small to easily read. Screen calibration deals with this problem, but this option is no longer available.

### Where do we go from here?

I reported the above issues to the Mathworks several months ago. It does not appear as a formal bug, but technical support is aware of the problem. The change is part of the “DPI aware” nature of release R2015b. So far I have found no evidence this release is any more aware of pixel size than previous releases, but my experience is limited to non-Retina Macs. I welcome input from users on other operating systems, particularly those with high-resolution displays.

To be fair, correct physical sizing is not an easy across the many platforms that Matlab runs on. Display resolution is particularly tricky when it changes during a Matlab session, such as when computer is connector to projector/television or a laptop is connected to a docking station.

Thankfully, printed graphic sizes are rendered correctly when a figure’s PaperPositionMode property is 'auto'. Many users can (and will) ignore the display problem if they aren’t dealing with strict size requirements and text legibility isn’t too bad. Some users may be willing to periodically print publication figures to externally verify sizing, but this breaks the interactive nature of Matlab figures.

A potential work around is the creating of a new figure class that oversizes figures (as needed) to account for a particular display. I started working on such a class, but the problem is more complicated than one might think:

• Child objects (axes, uicontrols, etc.) also must be resized if they are based on physical units.
• Resized objects must be temporarily restored to their original size for printing, and new objects must be tracked whenever they are added.
• Figure resolution may need to be changed when moving to different computer systems.

These capabilities are quite possible to implement, but this is a complicated solution to problem that was once easy to fix.

Retina displays don’t suffer as badly as one might think from the DPI mismatch. Even though the display specification may be greater than 200 DPI, OS X and/or Matlab must perform some intermediate size transformations. The effective DPI in R2015a is 110-120 for 13-15″ MacBook Pro laptops (at the default resolution). Objected sized with physical units still appear smaller than they should (~72/110), but not as small as I expected (<72/200).

Effect pixel size can also be changed by switching between different monitor scalings. This isn’t entirely surprising, but it can lead to some interesting results because Matlab only reads these settings at startup. Changing the display scaling during a session can cause square figures to appear rectangular. Also, the effective DPI changes for setting: I could reach values of ~60-110 DPI on an Apple Cinema Display.

So where does this leave us? Display calibration was always a finicky matter, but at least in principle one could make graphics appear exactly the same size on two different displays. Now it seems that sizing is completely variable between operation systems, displays, and display settings. For publication graphics, there will almost always be a disconnect between figure size on the screen and the printed output; some iteration may be needed to ensure everything looks right in the finished output. For graphical interfaces, font sizes may need to generated in normalized units and then converted to pixels (to avoid resizing).

Physical accuracy may not be important for non-publication figures, but the issue of text legibility remains. Some text objects–such as axes and tick labels–can easily be resized because the parent axes automatically adjusts itself as needed. Free floating text objects and uincontrols are much more difficult to deal with. Controls are often sized around the extent of their text label, so changing font sizes may require changes to the control position; adjacent controls may overlap after resizing for text clarity. Normalized units partially solve this problem, but their effect on uicontrols is not always desirable: do you really want push buttons to get larger/smaller when the figure is resized?

Can you think of a better workaround to this problem? If so, then please post a comment below. I will be very happy to hear your ideas, as I’m sure others who have high resolution displays would as well.

(cross-reference: CSSM newsgroup post)

Addendum Dec 31, 2016: Dan Dolan just posted a partial workaround on the MathWorks File Exchange. Also see the related recent article on working with non-standard DPI values.

]]>
https://undocumentedmatlab.com/blog/graphic-sizing-in-matlab-r2015b/feed 8
Customizing contour plotshttps://undocumentedmatlab.com/blog/customizing-contour-plots https://undocumentedmatlab.com/blog/customizing-contour-plots#comments Wed, 18 Nov 2015 18:00:55 +0000 http://undocumentedmatlab.com/?p=6075

Related posts:
1. Draggable plot data-tips Matlab's standard plot data-tips can be customized to enable dragging, without being limitted to be adjacent to their data-point. ...
2. Customizing contour plots part 2 Matlab contour labels' color and font can easily be customized. ...
3. Matlab’s HG2 mechanism HG2 is presumably the next generation of Matlab graphics. This article tries to explore its features....
4. getundoc – get undocumented object properties getundoc is a very simple utility that displays the hidden (undocumented) properties of a specified handle object....

]]>
One of my clients asked me last week whether it is possible to access and customize individual contour lines and labels in HG2 (Matlab’s new graphics system, R2014+). Today’s post will discuss how this could indeed be done.

Matlab contour plot

In HG1 (R2014a and earlier), contour handles were simple hggroup objects that incorporated text and patch child handles. The contour labels, lines and fill patches could easily be accessed via these child handles (contour lines and fills use the same patch object: the lines are simply the patch edges; fills are their faces). The lines could then be customized, the label strings changed, and the patch faces (fills) recolored:

[X,Y,Z] = peaks; [C,hContour] = contour(X,Y,Z,20, 'ShowText','on'); hChildren = get(hContour, 'Children'); set(hChildren(1), 'String','Yair', 'Color','b'); % 1st text (contour label) set(hChildren(end), 'EdgeColor',[0,1,1]); % last patch (contour line)

The problem is that in HG2 (R2014b onward), contour (and its sibling functions, contourf etc.) return a graphic object that has no accessible children. In other words, hContour.Children returns an empty array:

>> hContour.Children ans = 0x0 empty GraphicsPlaceholder array. >> allchild(hContour) ans = 0x0 empty GraphicsPlaceholder array. >> isempty(hContour.Children) ans = 1

So how then can we access the internal contour patches and labels?

### HG2’s contour object’s hidden properties

Skipping several fruitless dead-ends, it turns out that in HG2 the text labels, lines and fills are stored in undocumented hidden properties called TextPrims, EdgePrims and (surprise, surprise) FacePrims, which hold corresponding arrays of matlab.graphics.primitive.world.Text, matlab.graphics.primitive.world.LineStrip and matlab.graphics.primitive.world.TriangleStrip object handles (the drawnow part is also apparently very important, otherwise you might get errors due to the Prim objects not being ready by the time the code is reached):

>> drawnow; % very important! >> hContour.TextPrims % row array of Text objects ans = 1x41 Text array:   Columns 1 through 14 Text Text Text Text Text Text Text Text Text Text Text Text Text Text Columns 15 through 28 Text Text Text Text Text Text Text Text Text Text Text Text Text Text Columns 29 through 41 Text Text Text Text Text Text Text Text Text Text Text Text   >> hContour.EdgePrims % column array of LineStrip objects ans = 20x1 LineStrip array:   ineStrip LineStrip LineStrip ...   >> hContour.FacePrims % column array of TriangleStrip objects (empty if no fill) ans = 0x0 empty TriangleStrip array.

We can now access and customize the individual contour lines, labels and fills:

hContour.TextPrims(4).String = 'Dani'; hContour.TextPrims(7).Visible = 'off'; hContour.TextPrims(9).VertexData = single([-1.3; 0.5; 0]); % Label location in data units   hContour.EdgePrims(2).ColorData = uint8([0;255;255;255]); % opaque cyan hContour.EdgePrims(5).Visible = 'off';

Note that the LineStrip objects here are the same as those used for the axes Axles, which I described a few months ago. Any customization that we could do to the axle LineStrips can also be applied to contour LineStrips, and vice versa.

For example, to achieve the appearance of a topographic map, we might want to modify some contour lines to use dotted LineStyle and other lines to appear bold by having larger LineWidth. Similarly, we may wish to hide some labels (by setting their Visible property to ‘off’) and make other labels bold (by setting their Font.Weight property to ‘bold’). There are really numerous customization possibilities here.

Here is a listing of the standard (non-hidden) properties exposed by these objects:

>> get(hContour.TextPrims(1)) BackgroundColor: [] ColorData: [] EdgeColor: [] Font: [1x1 matlab.graphics.general.Font] FontSmoothing: 'on' HandleVisibility: 'on' HitTest: 'off' HorizontalAlignment: 'center' Interpreter: 'none' Layer: 'middle' LineStyle: 'solid' LineWidth: 1 Margin: 1 Parent: [1x1 Contour] PickableParts: 'visible' Rotation: 7.24591082075548 String: '-5.1541' StringBinding: 'copy' VertexData: [3x1 single] VerticalAlignment: 'middle' Visible: 'on'   >> get(hContour.EdgePrims(1)) AlignVertexCenters: 'off' AmbientStrength: 0.3 ColorBinding: 'object' ColorData: [4x1 uint8] ColorType: 'truecolor' DiffuseStrength: 0.6 HandleVisibility: 'on' HitTest: 'off' Layer: 'middle' LineCap: 'none' LineJoin: 'round' LineStyle: 'solid' LineWidth: 0.5 NormalBinding: 'none' NormalData: [] Parent: [1x1 Contour] PickableParts: 'visible' SpecularColorReflectance: 1 SpecularExponent: 10 SpecularStrength: 0.9 StripData: [1 18] Texture: [0x0 GraphicsPlaceholder] VertexData: [3x17 single] VertexIndices: [] Visible: 'on' WideLineRenderingHint: 'software'   >> get(hContour.FacePrims(1)) AmbientStrength: 0.3 BackFaceCulling: 'none' ColorBinding: 'object' ColorData: [4x1 uint8] ColorType: 'truecolor' DiffuseStrength: 0.6 HandleVisibility: 'on' HitTest: 'off' Layer: 'middle' NormalBinding: 'none' NormalData: [] Parent: [1x1 Contour] PickableParts: 'visible' SpecularColorReflectance: 1 SpecularExponent: 10 SpecularStrength: 0.9 StripData: [1 4 13 16 33 37 41 44 51 54 61 64 71 74 87 91 94 103] Texture: [0x0 GraphicsPlaceholder] TwoSidedLighting: 'off' VertexData: [3x102 single] VertexIndices: [] Visible: 'on'

But how did I know these properties existed? The easiest way in this case would be to use my getundoc utility, but we could also use my uiinspect utility or even the plain-ol’ struct function.

p.s. – there’s an alternative way, using the Java bean adapter that is associated with each Matlab graphics object: java(hContour). Specifically, this object apparent has the public method browseableChildren(java(hContour)) which returns the list of all children (in our case, 41 text labels [bean adapters], 20 lines, and a single object holding a ListOfPointsHighlight that corresponds to the regular hidden SelectionHandle property). However, I generally dislike working with the bean adapters, especially when there’s a much “cleaner” way to get these objects, in this case using the regular EdgePrims, FacePrims, TextPrims and SelectionHandle properties. Readers who are interested in Matlab internals can explore the bean adapters using a combination of my getundoc and uiinspect utilities.

So far for the easy part. Now for some more challenging questions:

### Customizing the color

First, can we modify the contour fill to have a semi- (or fully-) transparent fill color? – indeed we can:

[~, hContour] = contourf(peaks(20), 10); drawnow; % this is important, to ensure that FacePrims is ready in the next line! hFills = hContour.FacePrims; % array of TriangleStrip objects [hFills.ColorType] = deal('truecoloralpha'); % default = 'truecolor' for idx = 1 : numel(hFills) hFills(idx).ColorData(4) = 150; % default=255 end

Contour plot in HG2, with and without transparency

Similar transparency effects can also be applied to the LineStrip and Text objects. A discussion of the various combinations of acceptable color properties can be found here.

### Mouse clicks

Next, how can we set a custom context-menu for individual labels and contour lines?

Unfortunately, Text, LineStrip and TriangleStrip objects do not posses a ButtonDownFcn or UIContextMenu property, not even hidden. I tried searching in the internal/undocumented properties, but nothing came up.

##### Mouse click solution #1

So the next logical step would be to trap the mouse-click event at the contour object level. We cannot simply click the contour and check the clicked object because that would just give us the hContour object handle rather than the individual Text or LineStrip. So the idea would be to set hContour.HitTest='off', in the hope that the mouse click would be registered on the graphic object directly beneath the mouse cursor, namely the label or contour line. It turns out that the labels’ and lines’ HitTest property is ‘off’ by default, so, we also need to set them all to ‘on’:

hContour.HitTest = 'off'; [hContour.TextPrims.HitTest] = deal('on'); [hContour.EdgePrims.HitTest] = deal('on'); [hContour.FacePrims.HitTest] = deal('on'); hContour.ButtonDownFcn = @(h,e)disp(struct(e));

This seemed simple enough, but failed spectacularly: it turns out that because hContour.HitTest='off', mouse clicks are not registered on this objects, and on the other hand we cannot set the ButtonDownFcn on the primitive objects because they don’t have a ButtonDownFcn property!

Who said life is easy?

One workaround is to set the figure’s WindowButtonDownFcn property:

set(gcf, 'WindowButtonDownFcn', @myMouseClickCallback);

Now, inside your myMouseClickCallback function you can check the clicked object. We could use the undocumented builtin hittest(hFig) function to see which object was clicked. Alternatively, we could use the callback eventData‘s undocumented HitObject/HitPrimitive properties (this variant does not require the HitTest property modifications above):

function myMouseClickCallback(hFig, eventData) hitPrimitive = hittest(hFig); % undocumented function   hitObject = eventData.HitObject; % undocumented property => returns a Contour object (=hContour) hitPrimitive = eventData.HitPrimitive; % undocumented property => returns a Text or LineStrip object hitPoint = eventData.Point; % undocumented property => returns [x,y] pixels from figure's bottom-left corner   if strcmpi(hFig.SelectionType,'alt') % right-click if isa(hitPrimitive, 'matlab.graphics.primitive.world.Text') % label displayTextContextMenu(hitPrimitive, hitPoint) elseif isa(hitPrimitive, 'matlab.graphics.primitive.world.LineStrip') % contour line displayLineContextMenu(hitPrimitive, hitPoint) elseif isa(hitPrimitive, 'matlab.graphics.primitive.world.TriangleStrip') % contour fill displayFillContextMenu(hitPrimitive, hitPoint) else ... end end end
##### Mouse click solution #2

A totally different solution is to keep the default hContour.HitTest='on' (and the primitives’ as ‘off’) and simply query the contour object’s ButtonDownFcn callback’s eventData‘s undocumented Primitive property:

hContour.ButtonDownFcn = @myMouseClickCallback;

And in the callback function:

function myMouseClickCallback(hContour, eventData) hitPrimitive = eventData.Primitive; % undocumented property => returns a Text or LineStrip object hitPoint = eventData.IntersectionPoint; % [x,y,z] in data units   hFig = ancestor(hContour, 'figure'); if strcmpi(hFig.SelectionType,'alt') % right-click if isa(hitPrimitive, 'matlab.graphics.primitive.world.Text') % label displayTextContextMenu(hitPrimitive, hitPoint) elseif isa(hitPrimitive, 'matlab.graphics.primitive.world.LineStrip') % contour line displayLineContextMenu(hitPrimitive, hitPoint) elseif isa(hitPrimitive, 'matlab.graphics.primitive.world.TriangleStrip') % contour fill displayFillContextMenu(hitPrimitive, hitPoint) else ... end end end

This article should be a good start in how to code the displayTextContextMenu etc. functions to display a context menu.

### Customizations reset

Finally, there are apparently numerous things that cause our customized labels and lines to reset to their default appearance: resizing, updating contour properties etc. To update the labels in all these cases in one place, simply listen to the undocumented MarkedClean event:

addlistener(hContour, 'MarkedClean', @updateLabels);

Where updateLabels is a function were you set all the new labels.

I am marking this article as “High risk of breaking in future Matlab versions“, not because of the basic functionality (being important enough I don’t presume it will go away anytime soon) but because of the property names: TextPrims, EdgePrims and FacePrims don’t seem to be very user-friendly property names. So far MathWorks has been very diligent in making its object properties have meaningful names, and so I assume that when the time comes to expose these properties, they will be renamed (perhaps to TextHandles, EdgeHandles and FaceHandles, or perhaps LabelHandles, LineHandles and FillHandles). For this reason, even if you find out in some future Matlab release that TextPrims, EdgePrims and FacePrims don’t exist, perhaps they still exist and simply have different names.

Addendum November 11, 2017: The TextPrims, EdgePrims and FacePrims properties have still not changed their names and functionality. I explained a nice use for them in a followup post, explaining how we can modify the contour labels to have different font sizes and the same colors as their corresponding contour lines.

]]>
https://undocumentedmatlab.com/blog/customizing-contour-plots/feed 8

Related posts:
1. New information on HG2 More information on Matlab's new HG2 object-oriented handle-graphics system...
2. Performance: accessing handle properties Handle object property access (get/set) performance can be significantly improved using dot-notation. ...
3. uiundo – Matlab’s undocumented undo/redo manager The built-in uiundo function provides easy yet undocumented access to Matlab's powerful undo/redo functionality. This article explains its usage....
4. Matlab’s HG2 mechanism HG2 is presumably the next generation of Matlab graphics. This article tries to explore its features....

]]>
A client recently asked me to extend one of Matlab’s built-in graphic containers (uiflowcontainer in this specific case) with automatic scrollbars that would enable the container to act as a scroll-panel. The basic idea would be to dynamically monitor the container’s contents and when it is determined that they overflow the container’s boundaries, then attach horizontal/vertical scrollbars to enable scrolling the contents into view:

Scrollable Matlab container

This may sound simple, but there are actually quite a few undocumented hacks that make this possible, including listening to ObjectChildAdded/ObjectChildRemoved events, location/size/visibility events, layout changes etc. Maybe I’ll blog about it in some future article.

Today’s post is focused on a specific aspect of this project, attaching dynamic properties to the builtin uiflowcontainer, that would enable users to modify the container’s properties directly, as well as control aspects of the scrolling using the new properties: handles to the parent container, as well as the horizontal and vertical scrollbars, and even a new refresh() method.

The “textbook” approach to this would naturally be to create a new class that extends (inherits) uiflowcontainer and includes these new properties and methods. Unfortunately, for some reason that escapes my understanding, MathWorks saw fit to make all of its end-use graphic object classes Sealed, such that they cannot be extended by users. I did ask for this to be changed long ago, but the powers that be apparently decided that it’s better this way.

So the fallback would be to create our own dedicated class having all the new properties as well as those of the original container, and ensure that all the property values are synchronized in both directions. This is probably achievable, if you have a spare few days and a masochistic state of mind. Being the lazy bum and authority-rebel that I am, I decided to take an alternate approach that would simply add my new properties to the built-in container handle. The secret lies in the undocumented function schema.prop (for HG1, R2014a and older) and the fully-documented addprop function (for HG2, R2014b and newer).

In the examples below I use a panel, but this mechanism works equally well on any Matlab HG object: axes, lines, uicontrols, figures, etc.

The addprop function is actually a public method of the dynamicprops class. Both the dynamicprops class as well as its addprop function are fully documented. What is NOT documented, as far as I could tell, is that all of Matlab’s builtin handle graphics objects indirectly inherit dynamicprops, via matlab.graphics.Graphics, which is a high-level superclass for all HG objects. The bottom line is that we can dynamically add run-time properties to any HG object, without affecting any other object. In other words, the new properties will only be added to the handles that we specifically request, and not to any others. This suits me just fine:

hProp = addprop(hPanel, 'hHorizontalScrollBar'); hPanel.hHorizontalScrollBar = hMyScrollbar; hProp.SetAccess = 'private'; % make this property read-only

The new property hHorizontalScrollBar is now added to the hPanel handle, and can be accessed just like any other read-only property. For example:

>> get(hPanel, 'hHorizontalScrollBar') ans = JavaWrapper >> hPanel.hHorizontalScrollBar ans = JavaWrapper >> hPanel.hHorizontalScrollBar = 123 You cannot set the read-only property 'hHorizontalScrollBar' of UIFlowContainer.

Adding new methods is more tricky, since we do not have a corresponding addmethod function. The trick I used was to create a new property having the requested new method’s name, and set its read-only value to a handle of the requested function. For example:

hProp = addprop(hPanel, 'refresh'); hPanel.refresh = @myRefreshFunc; hProp.SetAccess = 'private'; % make this property read-only

We can then invoke the new refresh “method” using the familiar dot-notation:

hPanel.refresh();

Note: if you ever need to modify the initial value in your code, you should revert the property’s SetAccess meta-property to 'public' before Matlab will enable you to modify the value:

try % This will raise an exception if the property already exists hProp = addprop(hPanel, propName); catch % Property already exists - find it and set its access to public hProp = findprop(hPanel, propName); hProp.SetAccess = 'public'; end hPanel.(propName) = newValue;

### HG1 – schema.prop function

In HG1 (R2014a and earlier), we can use the undocumented schema.prop function to add a new property to any HG handle (which is a numeric value in HG1). Donn Shull wrote about schema.prop back in 2011, as part of his series of articles on UDD (Unified Data Dictionary, MCOS’s precursor). In fact, schema.prop is so useful that it has its own blog tag here and appears in no less than 15 separate articles (excluding today). With HG2’s debut 2 years ago, MathWorks tried very hard to rid the Matlab code corpus of all the legacy schema-based, replacing most major functionalities with MCOS-based HG2 code. But so far it has proven impossible to get rid of schema completely, and so schema code is still used extensively in Matlab to this day (R2015b). Search your Matlab path for “schema.prop” and see for yourself.

Anyway, the basic syntax is this:

hProp = schema.prop(hPanel, propName, 'mxArray');

The 'mxArray' specifies that the new property can accept any data type. We can limit the property to only accept certain types of data by specifying a less-generic data type, among those recognized by UDD (details).

Note that the meta-properties of the returned hProp are somewhat different from those of HG2’s hProp. Taking this into account, here is a unified function that adds/updates a new property (with optional initial value) to any HG1/HG2 object:

function addProp(hObject, propName, initialValue, isReadOnly) try hProp = addprop(hObject, propName); % HG2 catch try hProp = schema.prop(hObject, propName, 'mxArray'); % HG1 catch hProp = findprop(hObject, propName); end end if nargin > 2 try hProp.SetAccess = 'public'; % HG2 catch hProp.AccessFlags.PublicSet = 'on'; % HG1 end hObject.(propName) = initialValue; end if nargin > 3 && isReadOnly try % Set the property as read-only hProp.SetAccess = 'private'; % HG2 catch hProp.AccessFlags.PublicSet = 'off'; % HG1 end end end
]]>
Solving an mput (FTP) hang problemhttps://undocumentedmatlab.com/blog/solving-an-mput-ftp-hang-problem https://undocumentedmatlab.com/blog/solving-an-mput-ftp-hang-problem#comments Wed, 19 Aug 2015 20:10:17 +0000 http://undocumentedmatlab.com/?p=5974

Related posts:
1. More undocumented timing features There are several undocumented ways in Matlab to get CPU and clock data...
2. Uitable customization report Matlab's uitable can be customized in many different ways. A detailed report explains how. ...
3. Waiting for asynchronous events The Matlab waitfor function can be used to wait for asynchronous Java/ActiveX events, as well as with timeouts. ...
4. uisplittool & uitogglesplittool callbacks Matlab's undocumented uisplittool and uitogglesplittool are powerful toolbar controls - this article explains how to customize their behavior...

]]>
Matlab includes a variety of builtin utility functions that enable easy access to FTP. These functions are basically methods of an FTP object using Matlab’s old class system (for example, the mput function is implemented as mput.m in the %matlabroot%\toolbox\matlab\iofun\@ftp\ folder). These are pretty old files that haven’t changed much in years.

I recently needed to upload files from Matlab onto an FTP server, and discovered that calling mput simply hang Matlab to the point that I needed to kill and restart the Matlab process. The problem was not in the FTP server, since it could be accessed normally using FTP clients such as FileZilla and WinSCP. So it had to be something internal to Matlab. My initial workaround was to create an automation script (using WinSCP in my case) for the file upload. But this kludge is both non-robust as well as slow. A fix to the Matlab problem would be much better.

Some online research yielded others who have complained about similar issues over the years, but I saw no concrete answer. I saw many references online to problems that relate to the combination of passive FTP with Windows 7 / firewall / Java 7, that suggested several fixes (example 1, example 2, example 3, example 4). However, none of them solved the problem: calling mput (or dir etc.) continued to freeze Matlab, forcing either a hard process kill or waiting several minutes for the timeout.

Today, Malcolm Lidierth, author of the Waterloo graphics package, told me that he also saw a similar situation on Mac. Fortunately, Malcolm discovered a much better solution to the problem than my external scripts workaround. It seems that simply setting the Java object underlying Matlab’s FTP object to use passive mode fixes the problem (possibly in addition to the other fixes mentioned above).

This can be done in two manners:

The easiest is to add the following highlighted line in mput.m, on or about line 69 (the exact line may change based on your Matlab release):

... % Upload this file. fis = java.io.FileInputStream(fileObject); h.jobject.enterLocalPassiveMode(); % Use passive modestatus = h.jobject.storeFile(name,fis); fis.close; ...

This works but has several limitations:

1. it does not solve the problems of other passive FTP commands such as dir, although they can be solved in a similar manner
2. it creates a counter-problem when connecting to non-passive FTP servers – users might wish to make this a parameter of the ftp constructor function
3. and of course it requires changing Matlab’s installation files which is problematic in many aspects, as well as non-portable if you ever use your program on another machine or Matlab release.

Here’s a much simpler, portable and flexible solution: simply set the underlying Java object’s passive mode after connecting to the server (using the ftp constructor function) but before using mput or dir or any other command that uses passive FTP mode:

f = ftp('myftpserver.com',username,password); cd(f); sf=struct(f); sf.jobject.enterLocalPassiveMode();mput(f,filename); dir(f); close(f);

Note that we need to connect first, then get the underlying Java reference using the struct hack (since direct access to f.jobject is prevented), then we enter local passive mode, and only then we call mput to upload the file. If we omit the highlighted line in the script above, then mput (and dir etc.) will hang.

This way, we can programmatically control when to use passive mode, and no changes to the Matlab install files is required: our Matlab script should now work on all platforms and Matlab releases.

Note the seemingly unnecessary call to cd(f) – this is done to ensure a valid connection before setting the passive mode. We should ensure to do this before each call to mput/dir etc., since otherwise, if the connection drops for any reason (e.g., timeout or some other disconnection), then the mput/dir command would reconnect without passive mode (causing a hang). By calling cd(f) we ensure that the connection is done and passive mode is re-entered before mput/dir are called.

As an interesting related note, ftp accepts an undocumented set of optional input parameters following its first 3 documented inputs (hostname, username and password). We can pass one or more of the following parameters in P-V (param name, param value) pairs format: System, LenientFutureDates, DefaultDateFormatStr, RecentDateFormatStr, ServerLanguageCode, ServerTimeZoneId, ShortMonthNames. All of these parameters expect string (char) values, except LenientFutureDates that expects a logical value (true/false). All the parameters have a default value of empty. An explanation of these parameters can be found in Apache’s documentation of the FTPClientConfig class (Matlab’s ftp uses a plain Apache FTPClient object, where you can also find an explanation of the enterLocalPassiveMode method that was used above).

### Italy visit, Aug 26 – Sep 1, 2015

I will be traveling to north Italy between Aug 26 – Sep 1, 2015. If you happen to be in the area at that time, I will be happy to meet you to discuss how I could bring value to your work. Please email me (altmany at gmail) if you are interested.

Due to my travel, this blog will take a short summer vacation, and will return in early September. Stay tuned!

]]>
https://undocumentedmatlab.com/blog/solving-an-mput-ftp-hang-problem/feed 2

Related posts:
1. Multi-column (grid) legend This article explains how to use undocumented axes listeners for implementing multi-column plot legends...
2. Plot LimInclude properties The plot objects' XLimInclude, YLimInclude, ZLimInclude, ALimInclude and CLimInclude properties are an important feature, that has both functional and performance implications....
3. FIG files format FIG files are actually MAT files in disguise. This article explains how this can be useful in Matlab applications....
4. Handle Graphics Behavior HG behaviors are an important aspect of Matlab graphics that enable custom control of handle functionality. ...

]]>
One of my clients recently asked me to solve a very peculiar problem: He had several axes and was using Matlab’s builtin linkaxes function to link their axis limits. However, it didn’t behave quite the way that he expected. His axes were laid out as 2×2 subplots, and he wanted the two columns to be independently linked in the X axis, and the two rows to be independently linked in the Y axis:

% Prepare the axes ax(1,1) = subplot(2,2,1); ax(1,2) = subplot(2,2,2); ax(2,1) = subplot(2,2,3); ax(2,2) = subplot(2,2,4);   % Plot something x = 0 : 0.01 : 10; line(x, sin(x), 'Parent',ax(1,1)); line(x, sin(2*x), 'Parent',ax(1,2)); line(x, cos(x), 'Parent',ax(2,1)); line(x, cos(5*x), 'Parent',ax(2,2));   % Link the relevant axes linkaxes(ax(:,1),'x'); % left column linkaxes(ax(:,2),'x'); % right column linkaxes(ax(1,:),'y'); % top row linkaxes(ax(2,:),'y'); % bottom row

The problem was that the plots didn’t behave as expected: when zooming in on the bottom-left axes, for example, only the bottom-right axes was updated (Y-limits synced), whereas the top-left axes’ X-limits remained unchanged:

Apparently, the second set of two linkaxes commands (to sync the rows’ Y-limits) overrode the first set of two linkaxes commands (to sync the columns’ X-limits).

### Analysis

The reason for this unexpected behavior is that under the hood, linkaxes attaches property-change listeners to the corresponding axes, and stores these listeners in the axes’ hidden ApplicationData property (which is typically accessible via the getappdata / setappdata / isappdata / rmappdata set of functions). Specifically, up to a certain Matlab release (R2013b?), the listeners were placed in a field called ‘listener__’, and since then in a field called ‘graphics_linkaxes’. In either case, the field name was constant.

Therefore, when we placed the first set of linkaxes commands, the axes were correctly synced vertically (ax(1,1) with ax(2,1) in their X-limits, and similarly ax(1,2) with ax(2,2)). But when we placed the second set of linkaxes commands, the internal field in the axes’ ApplicationData property was overriden with the new listeners (that synced the rows’ Y-limits).

It so happens that Matlab listeners have a very nasty feature of being deleted when they are no longer referenced anywhere (within a workspace variable or object property). So when we overrode the first set of listener handles, we effectively deleted them, as if they were never set in the first place.

Some people may possibly complain about both issues at this point:

• That Matlab listeners get deleted so easily without so much as a console warning, and certainly against naive intuition.
• That repeated calls to linkaxes should override (rather than complement) each other.

As a side note, the addlistener function creates a listener and then persists it in the object’s hidden AutoListeners__ property. However, unlike the linkaxes behavior, addlistener‘s listener handles are always appended to AutoListeners__‘s contents, rather than replacing it. This ensures that all listeners are accessible and active until their container is deleted or they are specifically modified/removed. I wish that linkaxes used this mechanism, rather than its current ApplicationData one.

Luckily, there is a very easy and simple workaround, namely to use linkprop rather than linkaxes. The linkprop function is a lower-level function that creates a property-change listener that syncs corresponding properties in any specified array of object handles. In fact, linkaxes uses linkprop in order to create the necessary listeners. In our case, we can use linkprop directly, to independently attach such listeners to the axes’ XLim and YLim properties. We just need to ensure that all these listeners remain accessible to Matlab throughout the corresponding objects’ life-cycle. This is easily done using ApplicationData, as is done by linkaxes.m but in a smarter manner that does not override the previous values. The benefit of this is that when the axes are deleted, then so are the listeners; as long as the axes are accessible then so are the listeners. We just need to ensure that we don’t override these listener values:

setappdata(ax(1,1), 'YLim_listeners', linkprop(ax(1,:),'YLim')); setappdata(ax(2,1), 'YLim_listeners', linkprop(ax(2,:),'YLim')); setappdata(ax(1,1), 'XLim_listeners', linkprop(ax(:,1),'XLim')); setappdata(ax(1,2), 'XLim_listeners', linkprop(ax(:,2),'XLim'));

This results in the expected behavior:

### Conclusions

The design decision by MathWorks to automatically delete Matlab listeners as soon as their reference count is zeroed and they get garbage-collected, causes a myriad of unexpected run-time behaviors, one of which is exemplified in today’s post on linkaxes. This would still have not caused any problem had the developers of linkaxes been aware of this listener feature and taken care to store the linked listener handles in an accumulating repository (e.g., adding the listener handle to an array of existing handles, rather than replacing a scalar handle).

Luckily, now that we know how Matlab listeners behave, we can easily identify abnormal behavior that results from listener handles going out of scope, and can easily take steps to persist the handles somewhere, so that they will remain active.

I wish to stress here that the listeners’ limited scope is fully documented in several places in the documentation (e.g., here as well as the linkprop doc page). The non-additive behavior of linkaxes is also documented, albeit in an almost-invisible footnote on its doc-page.

However, I humbly contend that the fact that these behaviors are documented doesn’t meant that they are correct. After all, figure windows or timers aren’t deleted when their handle goes out of scope, are they? At the very least, I hope that MathWorks will improve the relevant doc pages, to highlight these non-intuitive behaviors, and in the case of linkaxes to present a linkprop usage example as a workaround.

If you are interested in the topic of Matlab listeners, note that I’ve written quite a few listener-related posts over the years (about property-change listeners as well as event listeners). I urge you to take a look at the list of related articles presented below, or to use the search box at the top of the page.

]]>