Pure Matlab – Undocumented Matlab https://undocumentedmatlab.com Charting Matlab's unsupported hidden underbelly Fri, 20 Oct 2017 09:57:44 +0000 en-US hourly 1 https://wordpress.org/?v=4.4.1 Tips for accelerating Matlab performancehttps://undocumentedmatlab.com/blog/tips-for-accelerating-matlab-performance https://undocumentedmatlab.com/blog/tips-for-accelerating-matlab-performance#respond Thu, 05 Oct 2017 18:25:06 +0000 http://undocumentedmatlab.com/?p=7099

Related posts:
1. Performance: scatter vs. line In many circumstances, the line function can generate visually-identical plots as the scatter function, much faster...
2. Plot LimInclude properties The plot objects' XLimInclude, YLimInclude, ZLimInclude, ALimInclude and CLimInclude properties are an important feature, that has both functional and performance implications....
3. Plot performance Undocumented inner plot mechanisms can significantly improve plotting performance ...
4. Performance: accessing handle properties Handle object property access (get/set) performance can be significantly improved using dot-notation. ...

]]>
I’m proud to report that MathWorks has recently posted my article “Tips for Accelerating MATLAB Performance” in their latest newsletter digest (September 2017). This article is an updated and expanded version of my post about consulting work that I did for the Crustal Dynamics Research Group at Harvard University, where I helped speed-up a complex Matlab-based GUI by a factor of 50-500 (depending on the specific feature).

Crustal dynamics visualization GUI

Featuring an article on the official newsletter by a non-MathWorker is rare. Doing this with someone like myself who has a reputation for undocumented aspects, and a consultancy business that potentially competes with theirs, is certainly not obvious. I take this to be a sign that despite the possible drawbacks of publishing my article, MathWorks felt that it provided enough value to the Matlab user community to merit the risk. I applaud MathWorks for this, and thank them for the opportunity of being featured in their official newsletter and conferences. I do not take it for granted in the least.

The newsletter article provides multiple ideas of improving the run-time performance for file I/O and graphics. Many additional techniques for improving Matlab’s performance can be found under the Performance tag in this blog, as well as in my book “Accelerating MATLAB Performance” (CRC Press, 2014, ISBN 978-1482211290).

Next week I will present live online webinars about various ways to improve Matlab’s run-time performance:

These live webinars will be 3.5 hours long, starting at 10am EDT (7am PDT, 3pm UK, 4pm CET, 7:30pm IST, time in your local timezone), with a short break in the middle. The presentations content will be based on onsite training courses that I presented at multiple client locations (details). A recording of the webinars will be available for anyone who cannot join the live events.

Email me if you would like additional information on the webinars or my consulting, or to inquire regarding an onsite training course.

]]>
https://undocumentedmatlab.com/blog/tips-for-accelerating-matlab-performance/feed 0
Faster csvwrite/dlmwritehttps://undocumentedmatlab.com/blog/faster-csvwrite-dlmwrite https://undocumentedmatlab.com/blog/faster-csvwrite-dlmwrite#comments Tue, 03 Oct 2017 15:00:05 +0000 http://undocumentedmatlab.com/?p=7080

Related posts:
1. Parsing mlint (Code Analyzer) output The Matlab Code Analyzer (mlint) has a lot of undocumented functionality just waiting to be used. ...
2. Explicit multi-threading in Matlab part 3 Matlab performance can be improved by employing POSIX threads in C/C++ code. ...
3. Plot LimInclude properties The plot objects' XLimInclude, YLimInclude, ZLimInclude, ALimInclude and CLimInclude properties are an important feature, that has both functional and performance implications....
4. Preallocation performance Preallocation is a standard Matlab speedup technique. Still, it has several undocumented aspects. ...

]]>
Matlab’s builtin functions for exporting (saving) data to output files are quite sub-optimal (as in slowwwwww…). I wrote a few posts about this in the past (how to improve fwrite performance, and save performance). Today I extend the series by showing how we can improve the performance of delimited text output, for example comma-separated (CSV) or tab-separated (TSV/TXT) files.

The basic problem is that Matlab’s dlmwrite function, which can either be used directly, or via the csvwrite function which calls it internally, is extremely inefficient: It processes each input data value separately, in a non-vectorized loop. In the general (completely non-vectorized) case, each data value is separately converted into a string, and is separately sent to disk (using fprintf). In the specific case of real data values with simple delimiters and formatting, row values are vectorized, but in any case the rows are processed in a non-vectorized loop: A newline character is separately exported at the end of each row, using a separate fprintf call, and this has the effect of flushing the I/O to disk each and every row separately, which is of course disastrous for performance. The output file is indeed originally opened in buffered mode (as I explained in my fprintf performance post), but this only helps for outputs done within the row – the newline output at the end of each row forces an I/O flush regardless of how the file was opened. In general, when you read the short source-code of dlmwrite.m you’ll get the distinct feeling that it was written for correctness and maintainability, and some focus on performance (e.g., the vectorization edge-case). But much more could be done for performance it would seem.

This is where Alex Nazarovsky comes to the rescue.

Alex was so bothered by the slow performance of csvwrite and dlmwrite that he created a C++ (MEX) version that runs about enormously faster (30 times faster on my system). He explains the idea in his blog, and posted it as an open-source utility (mex-writematrix) on GitHub.

Usage of Alex’s utility is very easy:

mex_WriteMatrix(filename, dataMatrix, textFormat, delimiter, writeMode);

where the input arguments are:

• filename – full path name for file to export
• dataMatrix – matrix of numeric values to be exported
• textFormat – format of output text (sprintf format), e.g. '%10.6f'
• delimiter – delimiter, for example ',' or ';' or char(9) (=tab)
• writeMode – 'w+' for rewriting file; 'a+' for appending (note the lowercase: uppercase will crash Matlab!)

Here is a sample run on my system, writing a simple CSV file containing 1K-by-1K data values (1M elements, ~12MB text files):

>> data = rand(1000, 1000); % 1M data values, 8MB in memory, ~12MB on disk   >> tic, dlmwrite('temp1.csv', data, 'delimiter',',', 'precision','%10.10f'); toc Elapsed time is 28.724937 seconds.   >> tic, mex_WriteMatrix('temp2.csv', data, '%10.10f', ',', 'w+'); toc % 30 times faster! Elapsed time is 0.957256 seconds.

Alex’s mex_WriteMatrix function is faster even in the edge case of simple formatting where dlmwrite uses vectorized mode (in that case, the file is exported in ~1.2 secs by dlmwrite and ~0.9 secs by mex_WriteMatrix, on my system).

### Trapping Ctrl-C interrupts

Alex’s mex_WriteMatrix code includes another undocumented trick that could help anyone else who uses a long-running MEX function, namely the ability to stop the MEX execution using Ctrl-C. Using Ctrl-C is normally ignored in MEX code, but Wotao Yin showed how we can use the undocumented utIsInterruptPending() MEX function to monitor for user interrupts using Ctrl-C. For easy reference, here is a copy of Wotao Yin’s usage example (read his webpage for additional details):

/* A demo of Ctrl-C detection in mex-file by Wotao Yin. Jan 29, 2010. */   #include "mex.h"   #if defined (_WIN32) #include <windows.h> #elif defined (__linux__) #include <unistd.h> #endif   #ifdef __cplusplus extern "C" bool utIsInterruptPending(); #else extern bool utIsInterruptPending(); #endif   void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[]) { int count = 0; while(1) { #if defined(_WIN32) Sleep(1000); /* Sleep one second */ #elif defined(__linux__) usleep(1000*1000); /* Sleep one second */ #endif   mexPrintf("Count = %d\n", count++); /* print count and increase it by 1 */ mexEvalString("drawnow;"); /* flush screen output */   if (utIsInterruptPending()) { /* check for a Ctrl-C event */ mexPrintf("Ctrl-C Detected. END\n\n"); return; } if (count == 10) { mexPrintf("Count Reached 10. END\n\n"); return; } } }

### Matlab performance webinars

Next week I will present live online webinars about numerous other ways to improve Matlab’s run-time performance:

These live webinars will be 3.5 hours long, starting at 10am EDT (7am PDT, 3pm UK, 4pm CET, 7:30pm IST, time in your local timezone), with a short break in the middle. The presentations content will be based on onsite training courses that I presented at multiple client locations (details). A recording of the webinars will be available for anyone who cannot join the live events.

Additional Matlab performance tips can be found under the Performance tag in this blog, as well as in my book “Accelerating MATLAB Performance“.

Email me if you would like additional information on the webinars, or an onsite training course, or about my consulting.

]]>
https://undocumentedmatlab.com/blog/faster-csvwrite-dlmwrite/feed 5
Runtime code instrumentationhttps://undocumentedmatlab.com/blog/runtime-code-instrumentation https://undocumentedmatlab.com/blog/runtime-code-instrumentation#comments Thu, 28 Sep 2017 13:36:17 +0000 http://undocumentedmatlab.com/?p=7063

Related posts:
1. Creating a simple UDD class This article explains how to create and test custom UDD packages, classes and objects...
2. Customizing print setup Matlab figures print-setup can be customized to automatically prepare the figure for printing in a specific configuration...
3. UDD Events and Listeners UDD event listeners can be used to listen to property value changes and other important events of Matlab objects...
4. Trapping warnings efficiently warnings can be trapped in a try-catch block just like regular errors. This articles explains how. ...

]]>
I regularly follow the MathWorks Pick-of-the-Week (POTW) blog. In a recent post, Jiro Doke highlighted Per Isakson’s tracer4m utility. Per is an accomplished Matlab programmer, who has a solid reputation in the Matlab user community for many years. His utility uses temporary conditional breakpoints to enable users to trace calls to their Matlab functions and class methods. This uses a little-known trick that I wish to highlight in this post.

tracer4m utility uses conditional breakpoints that evaluate but never become live

Matlab breakpoints are documented and supported functionality, and yet their documented use is typically focused at interactive programming in the Matlab editor, or as interactive commands that are entered in the Matlab console using the set of db* functions: dbstop, dbclear, dbstatus, dbstack etc. However, nothing prevents us from using these db* functions directly within our code.

For example, the dbstack function can help us diagnose the calling tree for the current function, in order to do action A if one of the calling ancestors was FunctionX, or to do action B otherwise (for example, to avoid nested recursions).

Similarly, we could add a programmatic call to dbstop in order to stop at a certain code location downstream (for debugging), if a certain condition happens upstream.

Per extended this idea very cleverly in tracer4m: conditional breakpoints evaluate a string in run-time: if the result is true (non-zero) then the code run is stopped at that location, but if it’s false (or zero) then the code run continues normally. To instrument calls to specific functions, Per created a function tracer() that logs the function call (using dbstack) and always returns the value false. He then dynamically created a string that contains a call to this new function and used the dbstop function to create a conditional breakpoint based on this function, something similar to this:

dbstop('in', filename, 'at', location, 'if', 'tracer()');

We can use this same technique for other purposes. For example, if we want to do some action (not necessarily log – perhaps do something else) when a certain code point is reached. The benefit here is that we don’t need to modify the code at all – we’re adding ad-hoc code pieces using the conditional breakpoint mechanism without affecting the source code. This is particularly useful when we do not have access to the source code (such as when it’s compiled or write-protected). All you need to do is to ensure that the instrumentation function always returns false so that the breakpoint does not become live and for code execution to continue normally.

The tracer4m utility is quite sophisticated in the sense that it uses mlint and smart regexp to parse the code and know which functions/methods occur on which line numbers and have which type (more details). In this sense, Per used undocumented functionality. I’m certain that Jiro was not aware of the dependency on undocumented features when he posted about the utility, so please don’t take this to mean that Jiro or MathWorks officially support this or any other undocumented functionality. Undocumented aspects are often needed to achieve top functionality, and I’m happy that the POTW blog highlights utilities based on their importance and merit, even if they do happen to use some undocumented aspect.

tracer4m‘s code also contains references to the undocumented profiler option -history, but this is not in fact used by the code itself, only in comments. I use this feature in my profile_history utility, which displays the function call/timing history in an interactive GUI window. This utility complements tracer4m by providing a lot more information, but this can result in a huge amount of information for large and/or long-running programs. In addition, tracer4m has the benefit of only logging those functions/methods that the user finds useful, rather than all the function call, which enables easier debugging when the relevant code area is known. In short, I wish I had known about tracer4m when I created profile_history. Now that I know about it, maybe I’ll incorporate some of its ideas into profile_history in order to make it more useful. Perhaps another moral of this is that we should actively monitor the POTW blog, because true gems are quite often highlighted there.

Function call timeline profiling

For anyone who missed the announcement in my previous post, I’m hosting a series of live webinars on advanced Matlab topics in the upcoming 2 weeks – I’ll be happy if you would join.

]]>
https://undocumentedmatlab.com/blog/runtime-code-instrumentation/feed 3
User-defined tab completions – take 2https://undocumentedmatlab.com/blog/user-defined-tab-completions-take-2 https://undocumentedmatlab.com/blog/user-defined-tab-completions-take-2#comments Wed, 12 Jul 2017 13:00:30 +0000 http://undocumentedmatlab.com/?p=6961

Related posts:
1. Setting desktop tab completions The Matlab desktop's Command-Window tab-completion can be customized for user-defined functions...
2. Class object tab completion & improper field names Tab completions and property access can be customized for user-created Matlab classes. ...
3. Recovering previous editor state Recovering the previous state of the Matlab editor and its loaded documents is possible using a built-in backup config file. ...
4. setPrompt – Setting the Matlab Desktop prompt The Matlab Desktop's Command-Window prompt can easily be modified using some undocumented features...

]]>
Back in 2010, I posted about Matlab’s undocumented mechanism for setting Matlab desktop tab-completions. That mechanism used a couple of internal files (TC.xml and TC.xsd) to describe the various possible options that are auto-completed (or displayed in a small tooltip window) when the user clicks the <Tab> key on partially-entered function input parameters.

Using TabComplete for user-defined functions

Unfortunately, this mechanism apparently broke in R2016a and was replaced with a new mechanism, as explained below.

The new mechanism relies on a file called functionSignatures.json which exists in every single folder that contains Matlab files that have functions whose input parameters ought to be tab-completable.

The new mechanism offers far greater versatility and flexability in defining the input types and inter-relationsships compared to the old TC.*-based mechanism. Another important benefit is that we can now add custom user-defined functionSignatures.json files to our user folders, next to our m-files, without having to modify any Matlab system file.

Note that you may need to restart Matlab the first time that you create a functionSignatures.json file. But once it’s created, you can modify it within a Matlab session and the changes take effect immediately.

Note: Credit for first posting about this new mechanism goes to Nicholas Mati. I’ve known about this new mechanism for over a year, but I never found the time to write about it until now, so Nicholas gets credit for breaking the scoop. The discussion below uses and expands Nicholas’ original post.

### Syntax

The functionSignatures.json file has the general form:

{ "FunctionName1": { "key1":"val1", "key2":"val2", "keyn":"valn" }, "FunctionName2": { "key1":"val1", "key2":"val2", "keyn":"valn" } }

A number of keys are supported including “platform“, “setsAns“, “inputs“, and “outputs“, although inputs and outputs are by far the most common (and presumably only inputs are relevant for tab-completion). These keys take an array of (or a single) object value(s). The objects typically take one of the following forms:

{"name":"variable_name", "kind":"kind_option", "type":"string_or_array_of_type"} {"mutuallyExclusiveGroup": [ ... ] } {"name":"varargin", "kind":"optional", "multiplicity":"append"}

The value for “kind” can be “required”, “optional”, “positional”, “flag”, “namevalue” or “platform” (and perhaps a few other lesser-used kinds):

• required” means that the specified input is mandatory
• optional” means that it can be added or omitted
• positional” means that it’s an optional input but if it is specified then it must appear at the specified position relative to the previous (earlier) inputs
• flag” means that it’s an optional input flag, from a predefined list of one or more single-token strings. For example, in regexp(s1,s2,'once') the last input arg ('once') is such a flag.
• namevalue” means that it follows Matlab’s standard practice of using P-V pairs (parameter name followed by its value). For example, func('propName',propValue)
• platform” indicates that this input is only available on the specified platform(s)

These “kind”s are all explained below.

The value for “type” can be a string such as “char” or “numeric” or “filepath”, or a more complicated JSON array (see below).

In addition to “name”, “kind” and “type”, we can also define a “default” value (e.g. "default":"false") and a “display” string. While these are [currently] not used by Desktop tab-completion, they might be used by other components such as the JIT compiler or the Editor, if not today then perhaps in a future release.

Note that while pure JSON format does not accept comments, Matlab’s functionSignatures.json does accept C++-style comments, as discovered by Heiko in a comment below. To add a comment, simply add // comment text at the end of any line, or /* comment text */ anywhere within a line.

### Usage examples

Multiple examples of functionSignatures.json files can be found in subfolders of %matlabroot%/toolbox/matlab. For example, here’s the tab-completion definition for the visdiff function, which displays a visual comparison between two files, and resides in %matlabroot%/toolbox/shared/comparisons/functionSignatures.json:

{ "visdiff": { "inputs": [ {"name":"filename1", "kind":"required", "type":"filepath"}, {"name":"filename2", "kind":"required", "type":"filepath"}, {"name":"type", "kind":"positional", "type":"choices={'text', 'binary'}"} ] } }

As can be seen in this example, the first and second inputs are expected to be a filename, whereas the third input is one of the two predefined strings ‘text’ or ‘binary’. This third input has “kind”:”positional”, meaning that it is optional, but if it is provided then it must be in the 3rd position and cannot appear sooner. Moreover, if the user specifies any input argument to the “right” of a positional input, then the positional argument becomes required, not optional.

Whereas a “positional” parameter has a specific position in the args list (#3 in the case of visdiff above), an “optional” parameter may appear anywhere in the list of inputs.

Here’s a more complex example, for the built-in regexprep function (in %matlabroot%/toolbox/matlab/strfun/functionSignatures.json). This example shows how to limit the input to certain data types and how to specify optional input flags with pre-defined choices:

"regexprep": { "inputs": [ {"name":"str", "kind":"required", "type":[["char"], ["cell"], ["string"]]}, {"name":"expression", "kind":"required", "type":[["char"], ["cell"], ["string"]]}, {"name":"replace", "kind":"required", "type":[["char"], ["cell"], ["string"]]}, {"name":"optMatch", "kind":"flag", "display":"", "type":[["char", "choices={'all','once'}"], ["numeric", "scalar"]], "default":"'all'"}, {"name":"optWarnings", "kind":"flag", "display":"", "type":["char", "choices={'nowarnings','warnings'}"], "default":"'nowarnings'"}, {"name":"optCase", "kind":"flag", "display":"", "type":["char", "choices={'matchcase','ignorecase','preservecase'}"], "default":"'matchcase'"}, {"name":"optEmptyMatch", "kind":"flag", "display":"", "type":["char", "choices={'noemptymatch','emptymatch'}"], "default":"'noemptymatch'"}, {"name":"optDotAll", "kind":"flag", "display":"", "type":["char", "choices={'dotall','dotexceptnewline'}"], "default":"'dotall'"}, {"name":"optStringAnchors", "kind":"flag", "display":"", "type":["char", "choices={'stringanchors','lineanchors'}"], "default":"'stringanchors'"}, {"name":"optSpacing", "kind":"flag", "display":"", "type":["char", "choices={'literalspacing','freespacing'}"], "default":"'literalspacing'"} ], "outputs": [ {"name":"newStr", "type":[["char"], ["cell"], ["string"]]} ] },

Here’s an even more complex example, this time for the codegen function (in %matlabroot%/toolbox/coder/matlabcoder/functionSignatures.json, part of the Matlab Coder toolbox). This example shows how to limit the filenames to certain extensions and how to specify name-value input pairs:

"codegen": { "inputs": [ {"name":"compile_only", "kind":"flag", "type":"choices={'-c'}"}, {"name":"config_flag", "kind":"flag", "type":"choices={'-config:mex','-config:lib','-config:dll','-config:exe','-config:hdl'}"}, {"name":"debug", "kind":"flag", "type":"choices={'-g'}"}, {"name":"report", "kind":"flag", "type":"choices={'-report'}"}, {"name":"launchreport", "kind":"flag", "type":"choices={'-launchreport'}"}, {"name":"file", "kind":"flag", "type":"filepath=*.m,*.mlx,*.c,*.cpp,*.h,*.o,*.obj,*.a,*.so,*.lib,*.tmf", "multiplicity":"append"}, {"name":"-d", "kind":"namevalue", "type":"folderpath"}, {"name":"-I", "kind":"namevalue", "type":"folderpath"}, {"name":"-globals", "kind":"namevalue"}, {"name":"-o", "kind":"namevalue", "type":[["char"], ["filepath"]]}, {"name":"-O", "kind":"namevalue", "type":"choices={'enable:inline','disable:inline','enable:blas','disable:blas','enable:openmp','disable:openmp'}"}, {"name":"-args", "kind":"namevalue", "type":[["identifier=variable"], ["char"]]}, {"name":"-config", "kind":"namevalue", "type":[["identifier=variable"], ["char"]]}, {"name":"verbose", "kind":"flag", "type":"choices={'-v'}"}, {"name":"singleC", "kind":"flag", "type":"choices={'-singleC'}"}, {"name":"-test", "kind":"namevalue", "type":"identifier=function"} ] },

### Argument types

As noted above, we use "type":... to specify the expected data type of each parameter. This can be a simple string such as “char”, “cellstr”, “numeric”, “table”, “categorical”, “filepath”, “folderpath”, “matlabpath”, class name, or a more complicated JSON array. For example:

• "type":["numeric","scalar"]
• "type":["numeric","numel=3",">=4"]
• "type":[["char"], ["cellstr"], ["numeric"], ["logical","vector"]]
• "type":[["char", "choices={'-ascii'}"]]
• "type":[["filepath"], ["matlabpath=*.m,*.mlx"], ["char"]]
• "type":"identifier=variable,function,localfunction,package,classdef"
• "type":"matlab.graphics.axis.Axes"
• "type":"choices={'yes','no','maybe'}"

We can even specify on-the-fly Matlab computation that returns a cell-array of values, for example a list of available fonts via "type":"choices=listfonts". A more complex example is the definition of the rmfield function, where the possible input choices for the second input arg (highlighted) depend on the struct that is provided in the first input arg (by running the fieldnames function on it):

"rmfield": { "inputs": [ {"name":"s", "kind":"required", "type":"struct"}, {"name":"field", "kind":"required", "type":"choices=fieldnames(s)"} ], "outputs": [ {"name":"s", "type":"struct"} ] },

### Alternative inputs

Multiple alternative inputs can be specified in the functionSignatures.json file. The easiest way to do so is to simply create multiple different definitions for the same function, one beneath the other. Matlab’s tab-completion parser is smart enough to combine those definitions and proceed with the most appropriate one based on the user-entered inputs.

For example, in the same Coder file above we find 6 alternative definitions. If (for example) we start typing coder('-ecoder', and click <Tab>, Matlab would automatically auto-complete the second input to “false”, and then another <Tab> click would set the third input to the required ‘-new’ parameter (see highlighted lines below):

... "coder": { "inputs": [ {"name":"projectname", "kind":"required", "type":"filepath=*.prj"} ] }, "coder": { "inputs": [ {"name":"-open", "kind":"namevalue", "type":"filepath=*.prj"} ] }, "coder": { "inputs": [ {"name":"-build", "kind":"namevalue", "type":"filepath=*.prj"} ] }, "coder": { "inputs": [ {"name":"-new", "kind":"namevalue", "type":[["filepath=*.prj"], ["char"]]} ] }, "coder": { "inputs": [ {"name":"ecoderFlag", "kind":"required", "type":"choices={'-ecoder'}"}, {"name":"ecoderValue", "kind":"required", "type":[["logical"], ["choices={'false'}"]]}, {"name":"newFlag", "kind":"required", "type":"choices={'-new'}"}, {"name":"newvalue", "kind":"required", "type":[["filepath=*.prj"], ["char"]]} ] }, "coder": { "inputs": [ {"name":"tocodeFlag", "kind":"required", "type":"choices={'-tocode'}"}, {"name":"tocodevalue", "kind":"required", "type":"filepath=*.prj"}, {"mutuallyExclusiveGroup": [ [], [ {"name":"scriptFlag", "kind":"required", "type":"choices={'-script'}"}, {"name":"scriptname", "kind":"required", "type":[["filepath=*.m"], ["char"]]} ] ] } ] }

This example also shows, in the last definition for the coder function, another mechanism for specifying alternative inputs, using “mutuallyExclusiveGroup” (aka “MEGs”). A MEG is defined using an array of options, enclosed in square brackets ([]). Each of the MEG options is exclusive to each of the others, meaning that we can only work with one of them and not the others. This is equivalent to duplicating the definition as we saw above, and saves us some copy-paste (in some cases a lot of copy-pastes, especially with multiple and/or nested MEGs). However, MEGs have a major drawback of reduced readability. I believe that in most cases we only have a single MEG and few input args, and in such cases it makes more sense to use repeated function defs rather than a MEG. The Matlab signature files contain numerous usage examples for either of these two mechanisms.

### Platform dependencies

If a specific function (or a specific signature variant) depends on the running platform, this can be specified via the “platform” directive. For example, the winopen function only works on Windows, but not on Linux/Mac. Its corresponding signature definition is:

"winopen": { "platform":"win32,win64", "inputs": [ {"name":"filename", "kind":"required", "type":"filepath"}, {"name":"varargin", "kind":"optional", "multiplicity":"append"} ] }

Platform dependence could also be specified at the parameter level, not just the entire function level. For example, in the xlsread function (defined in %matlabroot%/toolbox/matlab/iofun/functionSignatures.json), we see that the usage variant xlsread(filename,-1) is only available on Windows (note that the numeric value is defined as "<=-1", not necessarily -1), and so is the “functionHandle” input (which is called processFcn in the documentation – for some reason that escapes me the names of many input args do not match in the documentation and functionSignature):

"xlsread": { "inputs": [ {"name":"filename", "kind":"required", "type":"filepath=*.xls,*.xlsx,*.xlsb,*.csv"}, {"mutuallyExclusiveGroup": [ {"name":"openExcel", "kind":"required", "display":"", "type":["numeric", "<=-1"], "platform":"win64,win32"}, {"name":"xlRange", "kind":"required", "type":["char", "@(x) isempty(x) || ~isempty(strfind(x, ':'))"], "default":"''"}, [ {"name":"sheet", "kind":"positional", "type":[["char", "choices=matlab.internal.language.introspective.tabcompletion.xlsread_vsheet(filename)"], ["numeric", ">=1"]], "default":"1"}, {"name":"xlRange", "kind":"positional", "type":"char", "default":"''"}, {"name":"basic", "kind":"positional", "display":"", "type":["char", "choices={'basic',''}"]}, {"name":"functionHandle", "kind":"positional", "type":"function_handle", "platform":"win64,win32"} ] ] } ], ...

### Parsing errors

The new mechanism is not very user-friendly when you get something wrong. In the best case, it issues a cryptic error message (see below), and in the worst case it simply ignores the changes and the user has no idea why the new custom tab-completion is not working as intended.

The most likely causes of such problems are:

• The most common problem is that you placed the functionSignatures.json file in a different folder than the Matlab function. For example, if the myFunction() function is defined in myFunction.m, then the tab-completion of this function MUST be located in a functionSignatures.json file that resides in the same folder, not anywhere else on the Matlab path. In other words, the Matlab path is NOT relevant for tab-completion.
• Your functionSignatures.json file does not follow the [extremely strict] syntax rules above, to the letter. For example, forgetting the top or final curly braces, forgetting a comma or adding an extra one, or not closing all brackets/braces properly.
• You mistyped one or more of the input parameters, types or options.

In case of a parsing error, you’d see a red error message on the Matlab console the next time that you try to use tab-completion:

Error parsing JSON data; Boost reports "(189): expected ',' or ']'".

Unfortunately the error message only tells us the problematic line location within the functionSignatures.json file, but not the file’s location, so if we haven’t recently edited this file we’d need to find it in the relevant folder. For example:

edit(fullfile(fileparts(which('myFunction')), 'functionSignatures.json')

Moreover, when a JSON syntax error (such as the one above) occurs, the entire file is not parsed, not just the definition that caused the error.

Another limitation of tab-completion is that it does not work while the main Matlab thread is working (e.g., during a uiwait or waitfor). This may be somewhat misleading since most editor/debugging actions do work.

Arguably, this new tab-completion mechanism could be made more programmer-friendly. Perhaps this will improve in a future Matlab release.

For a related mechanism, see my article on tab-completion for class properties and methods from 2014, which is apparently still relevant and functional.

]]>
https://undocumentedmatlab.com/blog/user-defined-tab-completions-take-2/feed 5
Matlab compilation quirks – take 2https://undocumentedmatlab.com/blog/matlab-compilation-quirks-take-2 https://undocumentedmatlab.com/blog/matlab-compilation-quirks-take-2#respond Wed, 31 May 2017 18:00:42 +0000 http://undocumentedmatlab.com/?p=6919

Related posts:
1. Matlab compiler bug and workaround Both the Matlab compiler and the publish function have errors when parsing block-comments in Matlab m-code. ...
2. UDD Properties UDD provides a very convenient way to add customizable properties to existing Matlab object handles...
3. Disabling menu entries in deployed docked figures Matlab's standard menu items can and should be removed from deployed docked figures. This article explains how. ...
4. Handle Graphics Behavior HG behaviors are an important aspect of Matlab graphics that enable custom control of handle functionality. ...

]]>
Once again I would like to welcome guest blogger Hanan Kavitz of Applied Materials. Hanan posted a couple of guest posts here over the past few years, including a post last year about quirks with Matlab-compiled DLLs. Today Hanan will follow up on that post by discussing several additional quirks that they have encountered with Matlab compilations/deployment.

### Don’t fix it, if it ain’t broke…

In Applied Materials Israel (PDC) we use Matlab code for both algorithm development and deployment (production). As part of the dev-ops build system, which builds our product software versions, we build Matlab artifacts (binaries) from the Matlab source code.

A typical software version has several hundreds Matlab artifacts that are automatically rebuilt on a daily basis, and we have many such versions – totaling many thousands of compilations each day.

This process takes a long time, so we were looking for a way to make it more efficient.

The idea that we chose to implement sounds simple – take a single binary module in any software version (Ex. foo.exe – Matlab-compiled exe) and check it: if the source code for this module has not changed since the last compilation then simply don’t compile it, just copy it from previous software version repository. Since most of our code doesn’t change daily (some of it hasn’t changed in years), we can skip the compilation time of most binaries and just copy them from some repository of previously compiled binaries.

In a broader look, avoiding lengthy compilations cycles by not compiling unchanged code is a common programming practice, implemented by all modern compilers. For example, the ‘make’ utility uses a ‘makefile’ to check the time stamps of all dependencies of every object file in order to decide which object requires recompilation. In reality, this is not always the best solution as time stamps may be incorrect, but it works well in the vast majority of cases.

Coming back to Matlab, now comes the hard part – how could our build system know that nothing has changed in module X and that something has changed in module Y? How does it even know which source files it needs to ensure didn’t change?

The credit for the idea goes to my manager, Lior Cohen, as follows: You can actually check the dependency of a given binary after compilation. The basis of the solution is that a Matlab executable is in fact a compressed (zip) file. The idea is then to:

1. Compile the binary once
2. Unzip the binary and “see” all your dependencies (source files are encrypted and resources are not, but we only need the list of file names – not their content).
3. Now build a list of all your dependency files and compute the CRC value of each from the source control. Save it for the next time you are required to compile this module.
4. In the next compilation cycle, find this dependency list, review it, dependency source file at a time and make sure CRC of the dependency hasn’t changed since last time.
5. If no dependency CRC has changed, then copy the binary from the repository of previous software version, without compiling.
6. Otherwise, recompile the binary and rebuild the CRC list of all dependencies again, in preparation for the next compilation cycle.

That’s it! That simple? Well… not really – the reality is a bit more complex since there are many other dependencies that need to be checked. Some of them are:

1. Did the requested Matlab version of the binary change since the last compilation?
2. Did the compilation instructions themselves (we have a sort of ‘makefile’) change?

Basically, I implemented a policy that if anything changed, or if the dependency check itself failed, then we don’t take any chances and just compile this binary. Keeping in mind that this dependencies check and file copying is much faster than a Matlab compilation, we save a lot of actual compilation time using this method.

Bottom line: Given a software version containing hundreds of compilation instructions to execute and assuming not much has changed in the version (which is often the case), we skip over 90% of compilations altogether and only rebuild what really changed. The result is a version build that takes about half an hour, instead of many hours. Moreover, since the compilation process is working significantly less, we get fewer failures, fewer stuck or crashed mcc processes, and [not less importantly] less maintenance required by me.

Note that in our implementation we rely on the undocumented fact that Matlab binaries are in fact compressed zip archives. If and when a future Matlab release will change the implementation such that the binaries will no longer be zip archives, another way will need to be devised in order to ensure the consistency of the target executable with its dependent source files.

### Don’t kill it, if it ain’t bad…

I want to share a very weird issue I investigated over a year ago when using Matlab compiled exe. It started with a user showed me a Matlab compiled exe that didn’t run – I’m not talking about a regular Matlab exception: the process was crashing with an MS Windows popup window popping, stating something very obscure.

It was a very weird behavior that I couldn’t explain – the compiler seemed to work well but the compiled executable process kept crashing. Compiling completely different code showed the same behavior.

This issue has to do with the system compiler configuration that is being used. As you might know, when installing the Matlab compiler, before the first compilation is ever made, the user has to state the C compiler that the Matlab compiler should use in its compilation process. This is done by command ‘mbuild –setup’. This command asks the users to choose the C compiler and saves the configuration (batch file back then, xml in the newer versions of Matlab) in the user’s prefdir folder. At the time we were using Microsoft Visual C++ compiler 9.0 SP1.

The breakthrough in the investigation came when I ran mcc command with –verbose flag, which outputs much more compilation info than I would typically ever want… I discovered that although the target executable file had been created, a post compilation step failed to execute, while issuing a very cryptic error message:

mt.exe : general error c101008d: Failed to write the updated manifest to the resource of file “…”. Access is denied.

cryptic compilation error (click to zoom)

The failure was in one of the ‘post link’ commands in the configuration batch file – something obscure such as this:

set POSTLINK_CMDS2=mt.exe -outputresource: %MBUILD_OUTPUT_FILE_NAME%;%MANIFEST_RESOURCE% -manifest "%MANIFEST_FILE_NAME%"

This line of code takes an XML manifest file and inserts it into the generated binary file (additional details).

If you open a valid R2010a (and probably other old versions as well) Matlab-generated exe in a text editor you can actually see a small XML code embedded in it, while in a non-functioning exe I could not see this XML code.

So why would this command fail?

It turned out, as funny as it sounds, to be an antivirus issue – our IT department updated its antivirus policies and this ‘post link’ command suddenly became an illegal operation. Once our IT eased the policy, this command worked well again and the compiled executables stopped crashing, to our great joy.

]]>
https://undocumentedmatlab.com/blog/matlab-compilation-quirks-take-2/feed 0
GUI formatting using HTMLhttps://undocumentedmatlab.com/blog/gui-formatting-using-html https://undocumentedmatlab.com/blog/gui-formatting-using-html#comments Wed, 05 Apr 2017 20:26:44 +0000 http://undocumentedmatlab.com/?p=6877

Related posts:
1. Icon images & text in Matlab uicontrols HTML can be used to add image icons to Matlab listbox and popup (drop-down) controls. ...
2. Spicing up Matlab uicontrol tooltips Matlab uicontrol tooltips can be spiced-up using HTML and CSS, including fonts, colors, tables and images...
3. Multi-line uitable column headers Matlab uitables can present long column headers in multiple lines, for improved readability. ...
4. Rich-contents log panel Matlab listboxes and editboxes can be used to display rich-contents HTML-formatted strings, which is ideal for log panels. ...

]]>
As I’ve mentioned several times in the past, HTML can be used for simple formatting of GUI controls, including font colors/sizes/faces/angles. With a bit of thought, HTML (and some CSS) can also be used for non-trivial formatting, that would otherwise require the use of Java, such as text alignment, background color, and using a combination of text and icons in the GUI control’s contents.

### Alignment

For example, a question that I am often asked (latest example) is whether it is possible to left/center/right align the label within a Matlab button, listbox or table. While Matlab does not (yet) have properties that control alignment in uicontrols, we can indeed use HTML for this. There’s a catch though: if we simply tried to use <div align="left">…, it will not work. No error will be generated but we will not see any visible left-alignment. The reason is that internally, the text is contained within a snugly-fitting box. Aligning anything within a tight-fitting box obviously has no effect.

To solve the problem, we need to tell Matlab (or rather, the HTML interpreter used by the underlying Java control) to widen this internal box. One way to do this is to specify the width of the div tag, which can be enormous in order to span the entire available apace (<div width="999px" align="left">…). Another method is to simulate a simple HTML table that contains a single cell that holds the text, and then tell HTML the table cell’s width:

hButton.String = '<html><tr><td width=9999 align=left>Left-aligned'; % left-align within a button hTable.Data{2,1} = '<html><tr><td width=9999 align=right>And right'; % right-align within a specific uitable cell

Centered (default) and right-aligned button labels

Non-default alignment of uitable cells

I discussed the specific aspect of uicontrol content alignment in another post last year.

### Background color

The same problem (and solution) applies to background colors: if we don’t enlarge the snugly-fitting internal bounding-box, any HTML bgcolor that we specify would only be shown under the text (i.e., within the internal box’s confines). In order to display bgcolor across the entire control/cell width, we need to enlarge the internal box’s width (the align and bgcolor tags can of course be used together):

hButton.String = '<html><tr><td width=9999 bgcolor=#ffff00>Yellow'; % bgcolor within a button hTable.Data{2,1} = '<html><tr><td width=9999 bgcolor=#ffff00>Yellow'; % bgcolor within a specific uitable cell

### CSS

We can also use simple CSS, which provides more formatting customizability than plain HTML:

hTable.Data{2,1} = '<html><tr><td width=9999 style="background-color:yellow">Yellow';

HTML/CSS formatting is a poor-man’s hack. It is very crude compared to the numerous customization options available via Java. However, it does provide a reasonable solution for many use-cases, without requiring any Java. I discussed the two approaches for uitable cell formatting in this post.

### [Non-]support in uifigures

Important note: HTML formatting is NOT [yet] supported by the new web-based uifigures. While uifigures can indeed be hacked with HTML/CSS content (details), this is not an easy task. Since it should be trivially easy for MathWorks to enable HTML content in the new web-based uifigures, I implore anyone who uses HTML in their Matlab GUI to let MathWorks know about it so that they could prioritize this R&D effort into an upcoming Matlab release. You can send an email to George.Caia at mathworks.com, who apparently handles such aspects in MathWorks’ R&D efforts to transition from Java-based GUIs to web-based ones. In my previous post I spotlit MathWorks user-feedback surveys about users’ use of Java GUI aspects, aimed in order to migrate as many of the use-cases as possible onto the new web-based framework. HTML/CSS support is a natural by-product of the fact that Matlab’s non-web-based GUI is based on Java Swing components (that inherently support HTML/CSS). But unfortunately the MathWorks surveys are specific to the javacomponent function and the figure’s JavaFrame property. In other words, many users might be using undocumented Java aspects by simply using HTML content in their GUI, without ever realizing it or using javacomponent. So I think that in this case a simple email to George.Caia at mathworks.com to let him know how you’re using HTML would be more useful. Maybe one day MathWorks will be kind enough to post a similar survey specific to HTML support, or maybe one day they’s just add the missing HTML support, if only to be done with my endless nagging.

p.s. – I am well aware that we can align and bgcolor buttons in AppDesigner. But we can’t do this with individual table/listbox cells, and in general we can’t use HTML within uifigures without extensive hacks. I merely used the simple examples of button and uitable cell formatting in today’s post to illustrate the issue. So please don’t get hung up on the specifics, but rather on the broader issue of HTML support in uifigures.

And in the meantime, for as long as non-web-based GUI is still supported in Matlab, keep on enjoying the benefits that HTML/CSS provides.

### Automated bug-fix emails

In an unrelated matter, I wish to express my Kudos to the nameless MathWorkers behind the scenes who, bit by bit, improve Matlab and the user experience: Over the years I’ve posted a few times my frustrations with the opaqueness of MathWorks’ bug-reporting mechanism. One of my complaints was that users who file bugs are not notified when a fix or workaround becomes available. That at least seems to have been fixed now. I just received a seemingly-automated email notifying me that one of the bugs that I reported a few years ago has been fixed. This is certainly a good step in the right direction, so thank you!

Happy Passover/Easter to all!

]]>
https://undocumentedmatlab.com/blog/gui-formatting-using-html/feed 1

Related posts:
1. uiundo – Matlab’s undocumented undo/redo manager The built-in uiundo function provides easy yet undocumented access to Matlab's powerful undo/redo functionality. This article explains its usage....
2. Undocumented Profiler options part 3 An undocumented feature of the Matlab Profiler can report call history timeline - part 3 of series. ...
3. Undocumented Profiler options part 4 Several undocumented features of the Matlab Profiler can make it much more useful - part 4 of series. ...
4. Pinning annotations to graphs Annotation object can be programmatically set at, and pinned-to, plot axes data points. ...

]]>
Matlab’s license function returns the primary license number/ID used by Matlab, but no information about the various toolboxes that may be installed. The ver function returns a bit more information, listing the version number and installation date of installed toolboxes (even user toolboxes, such as my IB-Matlab toolbox). However, no additional useful information is provided beyond that:

>> license ans = 123456 % actual number redacted   >> ver ---------------------------------------------------------------------------------------------------- MATLAB Version: 9.1.0.441655 (R2016b) MATLAB License Number: 123456 Operating System: Microsoft Windows 7 Professional Version 6.1 (Build 7601: Service Pack 1) Java Version: Java 1.7.0_60-b19 with Oracle Corporation Java HotSpot(TM) 64-Bit Server VM mixed mode ---------------------------------------------------------------------------------------------------- MATLAB Version 9.1 (R2016b) Curve Fitting Toolbox Version 3.5.4 (R2016b) Database Toolbox Version 7.0 (R2016b) Datafeed Toolbox Version 5.4 (R2016b) Financial Instruments Toolbox Version 2.4 (R2016b) Financial Toolbox Version 5.8 (R2016b) GUI Layout Toolbox Version 2.2.1 (R2015b) Global Optimization Toolbox Version 3.4.1 (R2016b) IB-Matlab - Matlab connector to InteractiveBrokers Version 1.89 Expires: 1-Apr-2018 Image Processing Toolbox Version 9.5 (R2016b) MATLAB Coder Version 3.2 (R2016b) MATLAB Report Generator Version 5.1 (R2016b) Optimization Toolbox Version 7.5 (R2016b) Parallel Computing Toolbox Version 6.9 (R2016b) Statistical Graphics Toolbox Version 1.2 Statistics and Machine Learning Toolbox Version 11.0 (R2016b)   >> v = ver v = 1×16 struct array with fields: Name Version Release Date   >> v(1) ans = struct with fields:   Name: 'Curve Fitting Toolbox' Version: '3.5.4' Release: '(R2016b)' Date: '25-Aug-2016'   >> v(8) ans = struct with fields:   Name: 'IB-Matlab - Matlab connector to InteractiveBrokers' Version: '1.89' Release: 'Expires: 1-Apr-2018' Date: '02-Feb-2017'

It is sometimes useful to know which license number “owns” which product/toolbox, and the expiration date is associated with each of them. Unfortunately, there is no documented way to retrieve this information in Matlab – the only documented way is to go to your account section on the MathWorks website and check there.

Luckily, there is a simpler way that can be used to retrieve additional information, from right inside Matlab, using matlab.internal.licensing.getFeatureInfo:

>> all_data = matlab.internal.licensing.getFeatureInfo all_data = 23×1 struct array with fields: feature expdate keys license_number entitlement_id   >> all_data(20) ans = struct with fields:   feature: 'optimization_toolbox' expdate: '31-mar-2018' keys: 0 license_number: '123456' entitlement_id: '1409891'   >> all_data(21) ans = struct with fields:   feature: 'optimization_toolbox' expdate: '07-mar-2017' keys: 0 license_number: 'DEMO' entitlement_id: '3749959'

As can be seen in this example, I have the Optimization toolbox licensed under my main Matlab license (123456 [actual number redacted]) until 31-mar-2018, and also licensed under a trial (DEMO) license that expires in 3 weeks. As long as a toolbox has any future expiration date, it will continue to function, so in this case I’m covered until March 2018.

We can also request information about a specific toolbox (“feature”):

>> data = matlab.internal.licensing.getFeatureInfo('matlab') data = 3×1 struct array with fields: feature expdate keys license_number entitlement_id   >> data(1) data = struct with fields:   feature: 'matlab' expdate: '31-mar-2018' keys: 0 license_number: '123456' entitlement_id: '1409891'

The drawback of this functionality is that it only provides information about MathWorks’ toolbox, not any user-provided toolboxes (such as my IB-Matlab connector, or MathWorks’ own GUI Layout toolbox). Also, some of the toolbox names may be difficult to understand (“gads_toolbox” apparently stands for the Global Optimization Toolbox, for example):

>> {all_data.feature} ans = 1×23 cell array Columns 1 through 4 'curve_fitting_toolbox' 'database_toolbox' 'datafeed_toolbox' 'distrib_computing_toolbox' Columns 5 through 8 'distrib_computing_toolbox' 'excel_link' 'fin_instruments_toolbox' 'financial_toolbox' Columns 9 through 15 'gads_toolbox' 'gads_toolbox' 'image_toolbox' 'image_toolbox' 'matlab' 'matlab' 'matlab' Columns 16 through 20 'matlab_coder' 'matlab_coder' 'matlab_report_gen' 'matlab_report_gen' 'optimization_toolbox' Columns 21 through 23 'optimization_toolbox' 'optimization_toolbox' 'statistics_toolbox'

A related undocumented builtin function is matlab.internal.licensing.getLicInfo:

% Information on a single toolbox/product: >> matlab.internal.licensing.getLicInfo('matlab') ans = struct with fields:   license_number: {'123456' 'Prerelease' 'T3749959'} expiration_date: {'31-mar-2018' '30-sep-2016' '07-mar-2017'}   % Information on multiple toolboxes/products: >> matlab.internal.licensing.getLicInfo({'matlab', 'image_toolbox'}) % cell array of toolbox/feature names ans = 1×2 struct array with fields: license_number expiration_date   % The full case-insensitive names of the toolboxes can also be used: >> matlab.internal.licensing.getLicInfo({'Matlab', 'Image Processing toolbox'}) ans = 1×2 struct array with fields: license_number expiration_date   % And here's how to get the full list (MathWorks products only): >> v=ver; data=matlab.internal.licensing.getLicInfo({v.Name}) data = 1×16 struct array with fields: license_number expiration_date

I have [still] not found any way to associate a user toolbox/product (such as my IB-Matlab) in a way that will report it in a unified manner with the MathWorks products. If anyone finds a way to do this, please do let me know.

p.s. – don’t even think of asking questions or posting comments on this website related to illegal uses or hacks of the Matlab license…

]]>
Quirks with parfor vs. forhttps://undocumentedmatlab.com/blog/quirks-with-parfor-vs-for https://undocumentedmatlab.com/blog/quirks-with-parfor-vs-for#comments Thu, 05 Jan 2017 17:15:48 +0000 http://undocumentedmatlab.com/?p=6821

Related posts:
1. Matlab mex in-place editing Editing Matlab arrays in-place can be an important technique for optimizing calculations. This article shows how to do it using Mex. ...
2. Preallocation performance Preallocation is a standard Matlab speedup technique. Still, it has several undocumented aspects. ...
3. Array resizing performance Several alternatives are explored for dynamic array growth performance in Matlab loops. ...
4. Matlab’s internal memory representation Matlab's internal memory structure is explored and discussed. ...

]]>
A few months ago, I discussed several tips regarding Matlab’s parfor command, which is used by the Parallel Computing Toolbox (PCT) for parallelizing loops. Today I wish to extend that post with some unexplained oddities when using parfor, compared to a standard for loop.

### Data serialization quirks

Dimitri Shvorob may not appear at first glance to be a prolific contributor on Matlab Central, but from the little he has posted over the years I regard him to be a Matlab power-user. So when Dimitri reports something, I take it seriously. Such was the case several months ago, when he contacted me regarding very odd behavior that he saw in his code: the for loop worked well, but the parfor version returned different (incorrect) results. Eventually, Dimitry traced the problem to something originally reported by Dan Austin on his Fluffy Nuke It blog.

The core issue is that if we have a class object that is used within a for loop, Matlab can access the object directly in memory. But with a parfor loop, the object needs to be serialized in order to be sent over to the parallel workers, and deserialized within each worker. If this serialization/deserialization process involves internal class methods, the workers might see a different version of the class object than the one seen in the serial for loop. This could happen, for example, if the serialization/deserialization method croaks on an error, or depends on some dynamic (or random) conditions to create data.

In other words, when we use data objects in a parfor loop, the data object is not necessarily sent “as-is”: additional processing may be involved under the hood that modify the data in a way that may be invisible to the user (or the loop code), resulting in different processing results of the parallel (parfor) vs. serial (for) loops.

For additional aspects of Matlab serialization/deserialization, see my article from 2 years ago (and its interesting feedback comments).

### Data precision quirks

The following section was contributed by guest blogger Lior Perlmuter-Shoshany, head algorithmician at a private equity fund.

In my work, I had to work with matrixes in the order of 109 cells. To reduce the memory footprint (and hopefully also improve performance), I decided to work with data of type single instead of Matlab’s default double. Furthermore, in order to speed up the calculation I use parfor rather than for in the main calculation. In the end of the run I am running a mini for-loop to see the best results.

What I discovered to my surprise is that the results from the parfor and for loop variants is not the same!

The following simplified code snippet illustrate the problem by calculating a simple standard-deviation (std) over the same data, in both single– and double-precision. Note that the loops are ran with only a single iteration, to illustrate the fact that the problem is with the parallelization mechanism (probably the serialization/deserialization parts once again), not with the distribution of iterations among the workers.

clear rng('shuffle','twister');   % Prepare the data in both double and single precision arr_double = rand(1,100000000); arr_single = single(arr_double);   % No loop - direct computation std_single0 = std(arr_single); std_double0 = std(arr_double);   % Loop #1 - serial for loop std_single = 0; std_double = 0; for i=1 std_single(i) = std(arr_single); std_double(i) = std(arr_double); end   % Loop #2 - parallel parfor loop par_std_single = 0; par_std_double = 0; parfor i=1 par_std_single(i) = std(arr_single); par_std_double(i) = std(arr_double); end   % Compare results of for loop vs. non-looped computation isForSingleOk = isequal(std_single, std_single0) isForDoubleOk = isequal(std_double, std_double0)   % Compare results of single-precision data (for vs. parfor) isParforSingleOk = isequal(std_single, par_std_single) parforSingleAccuracy = std_single / par_std_single   % Compare results of double-precision data (for vs. parfor) isParforDoubleOk = isequal(std_double, par_std_double) parforDoubleAccuracy = std_double / par_std_double

Output example :

isForSingleOk = 1 % <= true (of course!) isForDoubleOk = 1 % <= true (of course!)   isParforSingleOk = 0 % <= false (odd!) parforSingleAccuracy = 0.73895227413361 % <= single-precision results are radically different in parfor vs. for   isParforDoubleOk = 0 % <= false (odd!) parforDoubleAccuracy = 1.00000000000021 % <= double-precision results are almost [but not exactly] the same in parfor vs. for

From my testing, the larger the data array, the bigger the difference is between the results of single-precision data when running in for vs. parfor.

In other words, my experience has been that if you have a huge data matrix, it’s better to parallelize it in double-precision if you wish to get [nearly] accurate results. But even so, I find it deeply disconcerting that the results are not exactly identical (at least on R2015a-R2016b on which I tested) even for the native double-precision .

Hmmm… bug?

### Upcoming travels – Zürich & Geneva

I will shortly be traveling to clients in Zürich and Geneva, Switzerland. If you are in the area and wish to meet me to discuss how I could bring value to your work with some advanced Matlab consulting or training, then please email me (altmany at gmail):

• Zürich: January 15-17
• Geneva: January 18-21

Happy new year everybody!

]]>
https://undocumentedmatlab.com/blog/quirks-with-parfor-vs-for/feed 7
Afterthoughts on implicit expansionhttps://undocumentedmatlab.com/blog/afterthoughts-on-implicit-expansion https://undocumentedmatlab.com/blog/afterthoughts-on-implicit-expansion#comments Wed, 30 Nov 2016 20:28:44 +0000 http://undocumentedmatlab.com/?p=6750

Related posts:
1. tic / toc – undocumented option Matlab's built-in tic/toc functions have an undocumented option enabling multiple nested clockings...
2. Plot LimInclude properties The plot objects' XLimInclude, YLimInclude, ZLimInclude, ALimInclude and CLimInclude properties are an important feature, that has both functional and performance implications....
3. Matrix processing performance Matrix operations performance is affected by internal subscriptions in a counter-intuitive way....
4. Performance: accessing handle properties Handle object property access (get/set) performance can be significantly improved using dot-notation. ...

]]>
Matlab release R2016b introduced implicit arithmetic expansion, which is a great and long-awaited natural expansion of Matlab’s arithmetic syntax (if you are still unaware of this or what it means, now would be a good time to read about it). This is a well-documented new feature. The reason for today’s post is that this new feature contains an undocumented aspect that should very well have been documented and even highlighted.

The undocumented aspect that I’m referring to is the fact that code that until R2016a produced an error, in R2016b produces a valid result:

% R2016a >> [1:5] + [1:3]' Error using + Matrix dimensions must agree.   % R2016b >> [1:5] + [1:3]' ans = 2 3 4 5 6 3 4 5 6 7 4 5 6 7 8 

This incompatibility is indeed documented, but not where it matters most (read on).

I first discovered this feature by chance when trying to track down a very strange phenomenon with client code that produced different numeric results on R2015b and earlier, compared to R2016a Pre-release. After some debugging the problem was traced to a code snippet in the client’s code that looked something like this (simplified):

% Ensure compatible input data try dataA + dataB; % this will (?) error if dataA, dataB are incompatible catch dataB = dataB'; end

The code snippet relied on the fact that incompatible data (row vs. col) would error when combined, as it did up to R2015b. But in R2016a Pre-release it just gave a valid numeric matrix, which caused numerically incorrect results downstream in the code. The program never crashed, so everything appeared to be in order, it just gave different numeric results. I looked at the release notes and none of the mentioned release incompatibilities appeared relevant. It took me quite some time, using side-by-side step-by-step debugging on two separate instances of Matlab (R2015b and R2016aPR) to trace the problem to this new feature.

This implicit expansion feature was removed from the official R2016a release for performance reasons. This was apparently fixed in time for R2016b’s release.

I’m totally in favor of this great new feature, don’t get me wrong. I’ve been an ardent user of bsxfun for many years and (unlike many) have even grown fond of it, but I still find the new feature to be better. I use it wherever there is no significant performance penalty, a need to support older Matlab releases, or a possibility of incorrect results due to dimensional mismatch.

### So what’s my point?

What I am concerned about is that I have not seen the new feature highlighted as a potential backward compatibility issue in the documentation or the release notes. Issues of far lesser importance are clearly marked for their backward incompatibility in the release notes, but not this important major change. A simple marking of the new feature with the warning icon () and in the “Functionality being removed or changed” section would have saved my client and me a lot of time and frustration.

MathWorks are definitely aware of the potential problems that the new feature might cause in rare use cases such as this. As Steve Eddins recently noted, there were plenty of internal discussions about this very thing. MathWorks were careful to ensure that the feature’s benefits far outweigh its risks (and I concur). But this also highlights the fact that MathWorks were fully aware that in some rare cases it might indeed break existing code. For those cases, I believe that they should have clearly marked the incompatibility implications in the release notes and elsewhere.

I have several clients who scour Matlab’s release notes before each release, trying to determine the operational risk of a Matlab upgrade. Having a program that returns different results in R2016b compared to R2016a, without being aware of this risk, is simply unacceptable to them, and leaves users with a disinclination to upgrade Matlab, to MathWorks’ detriment.

MathWorks in general are taking a very serious methodical approach to compatibility issues, and are clearly investing a lot of energy in this (a recent example). It’s too bad that sometimes this chain is broken. I find it a pity, and think that this can still be corrected in the online doc pages. If and when this is fixed, I’ll be happy to post an addendum here.

In my humble opinion from the backbenches, increasing the transparency on compatibility issues and open bugs will increase user confidence and result in greater adoption and upgrades of Matlab. Just my 2 cents…

Today MathWorks added the following compatibility warning to the release notes (R2016b, Mathematics section, first item) – thanks for listening MathWorks

]]>
https://undocumentedmatlab.com/blog/afterthoughts-on-implicit-expansion/feed 12
Icon images & text in Matlab uicontrolshttps://undocumentedmatlab.com/blog/icon-images-in-matlab-uicontrols https://undocumentedmatlab.com/blog/icon-images-in-matlab-uicontrols#comments Wed, 28 Sep 2016 10:28:04 +0000 http://undocumentedmatlab.com/?p=6687

Related posts:
1. Spicing up Matlab uicontrol tooltips Matlab uicontrol tooltips can be spiced-up using HTML and CSS, including fonts, colors, tables and images...
2. Rich-contents log panel Matlab listboxes and editboxes can be used to display rich-contents HTML-formatted strings, which is ideal for log panels. ...
3. Aligning uicontrol contents Matlab uicontrols can often be customized using plain HTML/CSS, without need for advanced Java. ...
4. GUI integrated browser control A fully-capable browser component is included in Matlab and can easily be incorporated in regular Matlab GUI applications. This article shows how....

]]>
One of my consulting clients recently asked me if I knew any builtin Matlab GUI control that could display a list of colormap names alongside their respective image icons, in a listbox or popup menu (drop-down/combo-box):

Matlab listbox (left) & popup menu (right) with icon images

My initial thought was that this should surely be possible, since Colormap is a documented figure property, that should therefore be listed inside the inspector window, and should therefore have an associated builtin Java control for the dropdown (just like other inspector controls, which are part of the com.mathworks.mlwidgets package, or possibly as a standalone control in the com.mathworks.mwswing package). To my surprise it turns out that for some unknown reason MathWorks neglected to add the Colormap property (and associated Java controls) to the inspector. This property is fully documented and all, just like Color and other standard figure properties, but unlike them Colormap can only be modified programmatically, not via the inspector window. Matlab does provide the related colormapeditor function and associated dialog window, but I would have expected a simple drop-down of the standard builtin colormaps to be available in the inspector. Anyway, this turned out to be a dead-end.

It turns out that we can relatively easily implement the requested listbox/combo-box using a bit of HTML magic, as I explained last week. The basic idea is for each of the listbox/combobox items to be an HTML string that contains both an <img> tag for the icon and the item label text. For example, such a string might contain something like this (parula is Matlab’s default colormap in HG2, starting in R2014b):

<html><img src="http://www.mathworks.com/help/matlab/ref/colormap_parula.png">parula

parula colormap image

Of course, it would be a bit inefficient for each of the icons to be fetched from the internet. Luckily, the full set of Matlab documentation is typically installed on the local computer as part of the standard Matlab installation, beneath the docroot folder (e.g., C:\Program Files\Matlab\R2016b\help). In our specific case, the parula colormap image is located in:

imageFilename = [docroot, '/matlab/ref/colormap_parula.png']

Note that for a local image to be accepted by HTML, it needs to follow certain conventions. In our case, the HTML string for displaying the above image is:

<html><img src="file:///C:/Program%20Files/Matlab/R2016b/help/matlab/ref/colormap_parula.png">parula

Warning: it’s easy when dealing with HTML images in Matlab to get the format confused, resulting in a red-x icon. I discussed this issue some 4 years ago, which is still relevant.

How can we get the list of available builtin colormaps? The standard Matlab way of doing this would be something like this:

>> possibleColormaps = set(gcf,'Colormap') possibleColormaps = {}

but as we can see, for some unknown reason (probably another MathWorks omission), Matlab does not list the names of its available builtin colormaps.

Fortunately, all the builtin colormaps have image filenames that follow the same convention, which make it easy to get this list by simply listing the names of the relevant files, from which we can easily create the necessary HTML strings:

>> iconFiles = dir([docroot, '/matlab/ref/colormap_*.png']);   >> colormapNames = regexprep({iconFiles.name}, '.*_(.*).png', '1') colormapNames = Columns 1 through 9 'autumn' 'bone' 'colorcube' 'cool' 'copper' 'flag' 'gray' 'hot' 'hsv' Columns 10 through 18 'jet' 'lines' 'parula' 'pink' 'prism' 'spring' 'summer' 'white' 'winter' >> htmlStrings = strcat('<html><img width=200 height=10 src="file:///C:/Program%20Files/Matlab/R2016a/help/matlab/ref/colormap_', colormapNames', '.png">', colormapNames') str = '<html><img width=200 height=10 src="file:///C:/Program%20Files/Matlab/R2016a/help/matlab/ref/colormap_autumn.png">autumn' '<html><img width=200 height=10 src="file:///C:/Program%20Files/Matlab/R2016a/help/matlab/ref/colormap_bone.png">bone' '<html><img width=200 height=10 src="file:///C:/Program%20Files/Matlab/R2016a/help/matlab/ref/colormap_colorcube.png">colorcube' ... >> hListbox = uicontrol(gcf, 'Style','listbox', 'Units','pixel', 'Pos',[10,10,270,200], 'String',htmlStrings); >> hPopup = uicontrol(gcf, 'Style','popup', 'Units','pixel', 'Pos',[10,500,270,20], 'String',htmlStrings); …which results in the screenshots at the top of this post. Note how I scaled the images to 10px high (so that the labels would be shown and not cropped vertically) and 200px wide (so that it becomes narrower than the default 434px). There’s really no need in this case for the full 434×27 image size – such flat images scale very nicely, even when their aspect ratio is not preserved. You can adjust the height and width values for a best fit with you GUI. Unfortunately, it seems that HTML strings are not supported in the new web-based uifigure controls. This is not really Matlab’s fault because the way to customize labels in HTML controls is via CSS: directly embedding HTML code in labels does not work (it’s a Java-Swing feature, not a browser feature). I really hope that either HTML or CSS processing will be enabled for web-based uicontrol in a future Matlab release, because until that time uifigure uicontrols will remain seriously deficient compared to standard figure uicontrols. Until then, if we must use uifigures and wish to customize our labels or listbox items, we can directly access the underlying web controls, as Iliya explained here. A blog reader recently complained that I’m abusing Swing and basically making Matlab work in unnatural ways, “something it was never meant to be“. I feel that using HTML as I’ve shown last week and in this post would fall under the same category in his eyes. To him and to others who complain I say that I have absolutely no remorse about doing this. When I purchase anything I have the full rights (within the scope of the license) to adapt it in whatever way fits my needs. As a software developer and manager for over 25 years, I’ve developed in dozens of programming languages and environments, and I still enjoy [ab]using Matlab. Matlab is a great environment to get things done quickly and if this sometimes requires a bit of HTML or Java hacks that make some people cringe, then that’s their problem, not mine – I’m content with being able to do in Matlab [nearly] everything I want, quickly, and move on to the next project. As long as it gets the job done, that’s fine by me. If this makes me more of an engineer than a computer scientist, then so be it. On the flip side, I say to those who claim that Matlab is lacking in this or that aspect, that in most likelihood the limitation is only in their minds, not in Matlab – we can do amazing stuff with Matlab if we just open our minds, and possibly use some undocumented hacks. I’m not saying that Matlab has no limitations, I’m just saying that in most cases they can be overcome if we took the time and trouble to look for a solution. Matlab is a great tool and yet many people are not aware of its potential. Blaming Matlab for its failings is just an easy excuse in many cases. Of course, MathWorks could help my crusade on this subject by enabling useful features such as easy GUI component customizations… On this sad day, I wish you all Shanah Tova! ]]> https://undocumentedmatlab.com/blog/icon-images-in-matlab-uicontrols/feed 5 Aligning uicontrol contentshttps://undocumentedmatlab.com/blog/aligning-uicontrol-contents https://undocumentedmatlab.com/blog/aligning-uicontrol-contents#comments Thu, 22 Sep 2016 13:10:18 +0000 http://undocumentedmatlab.com/?p=6663   Related posts: 1. Spicing up Matlab uicontrol tooltips Matlab uicontrol tooltips can be spiced-up using HTML and CSS, including fonts, colors, tables and images... 2. Rich-contents log panel Matlab listboxes and editboxes can be used to display rich-contents HTML-formatted strings, which is ideal for log panels. ... 3. Multi-line uitable column headers Matlab uitables can present long column headers in multiple lines, for improved readability. ... 4. Undocumented button highlighting Matlab button uicontrols can easily be highlighted by simply setting their Value property. ...   ]]> Matlab automatically aligns the text contents of uicontrols: button labels are centered, listbox contents are left-aligned, and table cells align depending on their contents (left-aligned for strings, centered for logical values, and right-aligned for numbers). Unfortunately, the control’s HorizontalAlignment property is generally ignored by uicontrols. So how can we force Matlab buttons (for example) to have right-aligned labels, or for listbox/table cells to be centered? Undocumented Matlab has the answer, yet again… It turns out that there are at least two distinct ways to set uicontrol alignment, using HTML and using Java. Today I will only discuss the HTML variant. The HTML method relies on the fact that Matlab uicontrols accept and process HTML strings. This was true ever since Matlab GUI started relying on Java Swing components (which inherently accept HTML labels) over a decade ago. This is expected to remain true even in Matlab’s upcoming web-based GUI system, since Matlab would need to consciously disable HTML in its web components, and I see no reason for MathWorks to do so. In short, HTML parsing of GUI control strings is here to stay for the foreseeable future. % note: no need to close HTML tags, e.g. </font></html> uicontrol('Style','list', 'Position',[10,10,70,70], 'String', ... {'<HTML><FONT color="red">Hello</Font></html>', 'world', ... '<html><font style="font-family:impact;color:green"><i>What a', ... '<Html><FONT color="blue" face="Comic Sans MS">nice day!'}); Listbox with HTML items While HTML formatting is generally frowned-upon compared to the alternatives, it provides a very quick and easy way to format text labels in various different manners, including using a combination of font faces, sizes, colors and other aspects (bold, italic, super/sub-script, underline etc.) within a single text label. This is naturally impossible to do with Matlab’s standard properties, but is super-easy with HTML placed in the label’s String property. Unfortunately, while Java Swing (and therefore Matlab) honors only a [large] sub-set of HTML and CSS. The most important directives are parsed but some others are not, and this is often difficult to debug. Luckily, using HTML and CSS there are often multiple ways to achieve the same visual effect, so if one method fails we can usually find an alternative. Such was the case when a reader asked me why the following seemingly-simple HTML snippet failed to right-align his button label: hButton.String = '<html><div style="text-align:right">text'; As I explained in my answer, it’s not Matlab that ignores the CSS align directive but rather the underlying Swing behavior, which snugly fits the text in the center of the button, and of course aligning text within a tight-fitting box has no effect. The workaround that I suggested simply forces Swing to use a non-tightly-fitting boundary box, within which we can indeed align the text: pxPos = getpixelposition(hButton); hButton.String = ['<html><div width="' num2str(pxPos(3)-20) 'px" align="right">text']; % button margins use 20px Centered (default) and right-aligned button labels This solution is very easy to set up and maintain, and requires no special knowledge other than a bit of HTML/CSS, which most programmers know in this day and age. Of course, the solution relies on the actual button size. So, if the button is created with normalized units and changes its size when its parent container is resized, we’d need to set a callback function on the parent (e.g., SizeChangedFcn of a uipanel) to automatically adjust the button’s string based on its updated size. A better solution that would be independent of the button’s pixel-size and would work even when the button is resized needs to use Java. A related solution for table cells uses a different HTML-based trick: this time, we embed an HTML table cell within the Matlab control’s cell, employing the fact that HTML table cells can easily be aligned. We just need to ensure that the HTML cell is defined to be larger than the actual cell width, so that the alignment fits well. We do this by setting the HTML cell width to 9999 pixels (note that the tr and td HTML tags are necessary, but the table tag is optional): uitable('Units','norm','Pos',[0,0,0.3,0.3], 'Data', ... {'Left', ... '<html><tr><td align=center width=9999>Center', ... '<html><tr><td align=right width=9999>Right'}); Non-default alignment of uitable cells As noted above, a better solution might be to set the underlying Java component’s alignment properties (or in the case of the uitable, its underlying JTable component’s cellrenderer’s alignment). But in the general case, simple HTML such as above could well be sufficient. ]]> https://undocumentedmatlab.com/blog/aligning-uicontrol-contents/feed 6 Zero-testing performancehttps://undocumentedmatlab.com/blog/zero-testing-performance https://undocumentedmatlab.com/blog/zero-testing-performance#comments Wed, 31 Aug 2016 17:00:44 +0000 http://undocumentedmatlab.com/?p=6622   Related posts: 1. uicontextmenu performance Matlab uicontextmenus are not automatically deleted with their associated objects, leading to leaks and slow-downs. ... 2. rmfield performance The performance of the builtin rmfield function (as with many other builtin functions) can be improved by simple profiling. ... 3. tic / toc – undocumented option Matlab's built-in tic/toc functions have an undocumented option enabling multiple nested clockings... 4. Solving a MATLAB bug by subclassing Matlab's Image Processing Toolbox's impoint function contains an annoying bug that can be fixed using some undocumented properties....   ]]> I would like to introduce guest blogger Ken Johnson, a MATLAB Connections partner specializing in electromagnetic optics simulation. Today Ken will explore some performance subtleties of zero testing in Matlab. I often have a need to efficiently test a large Matlab array for any nonzero elements, e.g. >> a = zeros(1e4); >> tic, b = any(a(:)~=0); toc Elapsed time is 0.126118 seconds. Simple enough. In this case, when a is all-zero, the internal search algorithm has no choice but to inspect every element of the array to determine whether it contains any nonzeros. In the more typical case where a contains many nonzeros you would expect the search to terminate almost immediately, as soon as it finds the first nonzero. But that’s not how it works: >> a = round(rand(1e4)); >> tic, b = any(a(:)~=0); toc Elapsed time is 0.063404 seconds. There is significant runtime overhead in constructing the logical array “a(:)~=0”, although the “any(…)” operation apparently terminates at the first true value it finds. The overhead can be eliminated by taking advantage of the fact that numeric values may be used as logicals in Matlab, with zero implicitly representing false and nonzero representing true. Repeating the above test without “~=0”, we get a huge runtime improvement: >> a = round(rand(1e4)); >> tic, b = any(a(:)); toc Elapsed time is 0.000026 seconds. However, there is no runtime benefit when a is all-zero: >> a = zeros(1e4); >> tic, b = any(a(:)); toc Elapsed time is 0.125120 seconds. (I do not quite understand this. There should be some runtime benefit from bypassing the logical array construction.) ### NaN values There is also another catch: The above efficiency trick does not work when a contains NaN values (if you consider NaN to be nonzero), e.g. >> any([0,nan]) ans = 0 The any function ignores entries that are NaN, meaning it treats NaNs as zero-equivalent. This is inconsistent with the behavior of the inequality operator: >> any([0,nan]~=0) ans = 1 To avoid this problem, an explicit isnan test is needed. Efficiency is not impaired when a contains many nonzeros, but there is a 2x efficiency loss when a is all-zero: >> a = round(rand(1e4)); >> tic, b = any(a(:)) || any(isnan(a(:))); toc Elapsed time is 0.000027 seconds. >> a = zeros(1e4); >> tic, b = any(a(:)) || any(isnan(a(:))); toc Elapsed time is 0.256604 seconds. For testing all-nonzero the NaN problem does not occur: >> all([1 nan]) ans = 1 In this context NaN is treated as nonzero and the all-nonzero test is straightforward: >> a = round(rand(1e4)); >> tic, b = all(a(:)); toc Elapsed time is 0.000029 seconds. For testing any-zero and all-zero, use the complements of the above tests: >> b = ~any(a(:)) || any(isnan(a(:))); % all zero? >> b = ~all(a(:)); % any zero? ### Efficient find The find operation can also be optimized by bypassing construction of a logical temporary array, e.g. >> a = round(rand(1e4)); >> tic, b = find(a(:)~=0, 1); toc Elapsed time is 0.065697 seconds. >> tic, b = find(a(:), 1); toc Elapsed time is 0.000029 seconds. There is no problem with NaNs in this case; the find function treats NaN as nonzero, e.g. >> find([0,nan,1], 1) ans = 2 ]]> https://undocumentedmatlab.com/blog/zero-testing-performance/feed 5 AppDesigner’s mlapp file formathttps://undocumentedmatlab.com/blog/appdesigner-mlapp-file-format https://undocumentedmatlab.com/blog/appdesigner-mlapp-file-format#comments Wed, 17 Aug 2016 17:00:04 +0000 http://undocumentedmatlab.com/?p=6613   Related posts: 1. A couple of internal Matlab bugs and workarounds A couple of undocumented Matlab bugs have simple workarounds. ... 2. Undocumented button highlighting Matlab button uicontrols can easily be highlighted by simply setting their Value property. ... 3. uiundo – Matlab’s undocumented undo/redo manager The built-in uiundo function provides easy yet undocumented access to Matlab's powerful undo/redo functionality. This article explains its usage.... 4. Solving a Matlab hang problem A very common Matlab hang is apparently due to an internal timing problem that can easily be solved. ...   ]]> Six years ago, I exposed the fact that *.fig files are simply MAT files in disguise. This information, in addition to the data format that I explained in that article, can help us to introspect and modify FIG files without having to actually display the figure onscreen. Matlab has changed significantly since 2010, and one of the exciting new additions is the AppDesigner, Matlab’s new GUI layout designer/editor. Unfortunately, AppDesigner still has quite a few limitations in functionality and behavior. I expect that this will improve in upcoming releases since AppDesigner is undergoing active development. But in the meantime, it makes sense to see whether we could directly introspect and potentially manipulate AppDesigner’s output (*.mlapp files), as we could with GUIDE’s output (*.fig files). A situation for checking this was recently raised by a reader on the Answers forum: apparently AppDesigner becomes increasingly sluggish when the figure’s code has more than a few hundred lines of code (i.e., a very simplistic GUI). In today’s post I intend to show how we can explore the resulting *.mlapp file, and possibly manipulate it in a text editor outside AppDesigner. Matlab's new AppDesigner (a somewhat outdated screenshot) ### The MLAPP file format Apparently, *.mlapp files are simply ZIP files in disguise (note: not MAT files as for *.fig files). A typical MLAPP’s zipped contents contains the following files (note that this might be a bit different on different Matlab releases): • [Content_Types].xml – this seems to be application-independent: <?xml version="1.0" encoding="UTF-8" standalone="true"?> <Types xmlns="http://schemas.openxmlformats.org/package/2006/content-types"> <Default Extension="mat" ContentType="application/vnd.mathworks.matlab.appDesigner.appModel+mat"/> <Default Extension="rels" ContentType="application/vnd.openxmlformats-package.relationships+xml"/> <Default Extension="xml" ContentType="application/vnd.mathworks.matlab.code.document+xml;plaincode=true"/> <Override ContentType="application/vnd.openxmlformats-package.core-properties+xml" PartName="/metadata/coreProperties.xml"/> <Override ContentType="application/vnd.mathworks.package.coreProperties+xml" PartName="/metadata/mwcoreProperties.xml"/> <Override ContentType="application/vnd.mathworks.package.corePropertiesExtension+xml" PartName="/metadata/mwcorePropertiesExtension.xml"/> </Types> • _rels/.rels – also application-independent: <?xml version="1.0" encoding="UTF-8" standalone="true"?> <Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships"> <Relationship Type="http://schemas.mathworks.com/matlab/code/2013/relationships/document" Target="matlab/document.xml" Id="rId1"/> <Relationship Type="http://schemas.mathworks.com/package/2012/relationships/coreProperties" Target="metadata/mwcoreProperties.xml" Id="rId2"/> <Relationship Type="http://schemas.mathworks.com/package/2014/relationships/corePropertiesExtension" Target="metadata/mwcorePropertiesExtension.xml" Id="rId3"/> <Relationship Type="http://schemas.openxmlformats.org/package/2006/relationships/metadata/core-properties" Target="metadata/coreProperties.xml" Id="rId4"/> <Relationship Type="http://schemas.mathworks.com/appDesigner/app/2014/relationships/appModel" Target="appdesigner/appModel.mat" Id="rId5"/> </Relationships> • metadata/coreProperties.xml – contains the timestamp of figure creation and last update: <?xml version="1.0" encoding="UTF-8" standalone="true"?> <cp:coreProperties xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:dcterms="http://purl.org/dc/terms/" xmlns:dcmitype="http://purl.org/dc/dcmitype/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:cp="http://schemas.openxmlformats.org/package/2006/metadata/core-properties"> <dcterms:created xsi:type="dcterms:W3CDTF">2016-08-01T18:20:26Z</dcterms:created> <dcterms:modified xsi:type="dcterms:W3CDTF">2016-08-01T18:20:27Z</dcterms:modified> </cp:coreProperties> • metadata/mwcoreProperties.xml – contains information on the generating Matlab release: <?xml version="1.0" encoding="UTF-8" standalone="true"?> <mwcoreProperties xmlns="http://schemas.mathworks.com/package/2012/coreProperties"> <contentType>application/vnd.mathworks.matlab.app</contentType> <contentTypeFriendlyName>MATLAB App</contentTypeFriendlyName> <matlabRelease>R2016a</matlabRelease> </mwcoreProperties> • metadata/mwcorePropertiesExtension.xml – more information about the generating Matlab release. Note that the version number is not exactly the same as the main Matlab version number: here we have 9.0.0.328027 whereas the main Matlab version number is 9.0.0.341360. I do not know whether this is checked anywhere. <?xml version="1.0" encoding="UTF-8" standalone="true"?> <mwcoreProperties xmlns="http://schemas.mathworks.com/package/2014/corePropertiesExtension"> <matlabVersion>9.0.0.328027</matlabVersion> </mwcoreProperties> • appdesigner/appModel.mat – This is a simple MAT file that holds a single Matlab object called “appData” (of type appdesigner.internal.serialization.app.AppData) the information about the uifigure, similar in concept to the *.fig files generated by the old GUIDE: >> d = load('C:\Yair\App3\appdesigner\appModel.mat') Warning: Functionality not supported with figures created with the uifigure function. For more information, see Graphics Support in App Designer. (Type "warning off MATLAB:ui:uifigure:UnsupportedAppDesignerFunctionality" to suppress this warning.) d = appData: [1x1 appdesigner.internal.serialization.app.AppData] >> d.appData ans = AppData with properties: UIFigure: [1x1 Figure] CodeData: [1x1 appdesigner.internal.codegeneration.model.CodeData] Metadata: [1x1 appdesigner.internal.serialization.app.AppMetadata] ToolboxVer: '2016a' >> d.appData.CodeData ans = CodeData with properties: GeneratedClassName: 'App3' Callbacks: [0x0 appdesigner.internal.codegeneration.model.AppCallback] StartupFcn: [1x1 appdesigner.internal.codegeneration.model.AppCallback] EditableSection: [1x1 appdesigner.internal.codegeneration.model.CodeSection] ToolboxVer: '2016a' >> d.appData.Metadata ans = AppMetadata with properties: GroupHierarchy: {} ToolboxVer: '2016a' • matlab/document.xml – this file contains a copy of the figure’s classdef code in plain-text XML: <?xml version="1.0" encoding="UTF-8"?> <w:document xmlns:w="http://schemas.openxmlformats.org/wordprocessingml/2006/main"> <w:body> <w:p> <w:pPr> <w:pStyle w:val="code"/> </w:pPr> <w:r> <w:t> <![CDATA[classdef App2 < matlab.apps.AppBase % Properties that correspond to app components properties (Access = public) UIFigure matlab.ui.Figure UIAxes matlab.ui.control.UIAxes Button matlab.ui.control.Button CheckBox matlab.ui.control.CheckBox ListBoxLabel matlab.ui.control.Label ListBox matlab.ui.control.ListBox end methods (Access = public) function results = func(app) % Yair 1/8/2016 end end % App initialization and construction methods (Access = private) % Create UIFigure and components function createComponents(app) % Create UIFigure app.UIFigure = uifigure; app.UIFigure.Position = [100 100 640 480]; app.UIFigure.Name = 'UI Figure'; setAutoResize(app, app.UIFigure, true) % Create UIAxes app.UIAxes = uiaxes(app.UIFigure); title(app.UIAxes, 'Axes'); xlabel(app.UIAxes, 'X'); ylabel(app.UIAxes, 'Y'); app.UIAxes.Position = [23 273 300 185]; % Create Button app.Button = uibutton(app.UIFigure, 'push'); app.Button.Position = [491 378 100 22]; % Create CheckBox app.CheckBox = uicheckbox(app.UIFigure); app.CheckBox.Position = [491 304 76 15]; % Create ListBoxLabel app.ListBoxLabel = uilabel(app.UIFigure); app.ListBoxLabel.HorizontalAlignment = 'right'; app.ListBoxLabel.Position = [359 260 43 15]; app.ListBoxLabel.Text = 'List Box'; % Create ListBox app.ListBox = uilistbox(app.UIFigure); app.ListBox.Position = [417 203 100 74]; end end methods (Access = public) % Construct app function app = App2() % Create and configure components createComponents(app) % Register the app with App Designer registerApp(app, app.UIFigure) if nargout == 0 clear app end end % Code that executes before app deletion function delete(app) % Delete UIFigure when app is deleted delete(app.UIFigure) end end end]]> </w:t> </w:r> </w:p> </w:body> </w:document> I do not know why the code is duplicated, both in document.xml and (twice!) in appModel.mat. On the face of it, this does not seem to be a wise design decision. ### Editing MLAPP files outside AppDesigner We can presumably edit the app in an external editor as follow: 1. Open the *.mlapp file in your favorite zip viewer (e.g., winzip or winrar). You may need to rename/copy the file as *.zip. 2. Edit the contents of the contained matlab/document.xml file in your favorite text editor (Matlab’s editor for example) 3. Load appdesigner/appModel.mat into Matlab workspace. 4. Go to appData.CodeData.EditableSection.Code and update the cell array with the lines of your updated code (one cell element per user-code line). 5. Do the same with appData.CodeData.GeneratedCode (if existing), which holds the same data as appData.CodeData.EditableSection.Code but also including the AppDesigner-generated [non-editable] code. 6. Save the modified appData struct back into appdesigner/appModel.mat 7. Update the zip file (*.mlapp) with the updated appModel.mat and document.xml In theory, it is enough to extract the classdef code and same it in a simple *.m file, but then you would not be able to continue using AppDesigner to make layout modifications, and you would need to make all the changes manually in the m-file. If you wish to continue using AppDesigner after you modified the code, then you need to save it back into the *.mlapp file as explained above. If you think this is not worth all the effort, then you’re probably right. But you must admit that it’s a bit fun to poke around… One day maybe I’ll create wrapper utilities (mlapp2m and m2mlapp) that do all this automatically, in both directions. Or maybe one of my readers here will pick up the glove and do it sooner – are you up for the challenge? ### Caveat Emptor Note that the MLAPP file format is deeply undocumented and subject to change without prior notice in upcoming Matlab releases. In fact, MathWorker Chris Portal warns us that: A word of caution for anyone that tries this undocumented/unsupported poking into their MLAPP file. Taking this approach will almost certainly guarantee your app to not load in one of the subsequent releases. Just something to consider in your off-roading expedition! Then again, the same could have been said about the FIG and other binary file formats used by Matlab, which remained essentially the same for the past decade: Some internal field values may have changed but not the general format, and in any case the newer releases still accept files created with previous releases. For this reason, I speculate that future AppDesigners will accept MLAPP files created by older releases, possibly even hand-modified MLAPP files. Perhaps a CRC hash code of some sort will be expected, but I believe that any MLAPP that we modify today will still work in future releases. However, I could well be mistaken, so please be very careful with this knowledge. I trust that you can make up your own mind about whether it is worth the risk (and fun) or not. AppDesigner is destined to gradually replace the aging GUIDE over the upcoming years. They currently coexist since AppDesigner (and its web-based uifigures) still does not contain all the functionality that GUIDE (and JFrame-based figures) provides (a few examples). I already posted a few short posts about AppDesigner (use the AppDesigner tag to list them), and today’s article is another in that series. Over the next few years I intend to publish more on AppDesigner and its associated new GUI framework (uifigures). ### Zurich visit, 21-31 Aug 2016 I will be traveling to Zürich for a business trip between August 21-31. If you are in the Zürich area and wish to meet me to discuss how I could bring value to your work, then please email me (altmany at gmail). ]]> https://undocumentedmatlab.com/blog/appdesigner-mlapp-file-format/feed 6 Customizing axes part 5 – origin crossover and labelshttps://undocumentedmatlab.com/blog/customizing-axes-part-5-origin-crossover-and-labels https://undocumentedmatlab.com/blog/customizing-axes-part-5-origin-crossover-and-labels#comments Wed, 27 Jul 2016 17:00:02 +0000 http://undocumentedmatlab.com/?p=6564   Related posts: 1. Customizing axes rulers HG2 axes can be customized in numerous useful ways. This article explains how to customize the rulers. ... 2. Customizing axes part 2 Matlab HG2 axes can be customized in many different ways. This article explains some of the undocumented aspects. ... 3. Undocumented scatter plot jitter Matlab's scatter plot can automatically jitter data to enable better visualization of distribution density. ... 4. Performance: accessing handle properties Handle object property access (get/set) performance can be significantly improved using dot-notation. ...   ]]> When HG2 graphics was finally released in R2014b, I posted a series of articles about various undocumented ways by which we can customize Matlab’s new graphic axes: rulers (axles), baseline, box-frame, grid, back-drop, and other aspects. Today I extend this series by showing how we can customize the axes rulers’ crossover location. Non-default axes crossover location ### The documented/supported stuff Until R2015b, we could only specify the axes’ YAxisLocation as 'left' (default) or 'right', and XAxisLocation as 'bottom' (default) or 'top'. For example: x = -2*pi : .01 : 2*pi; plot(x, sin(x)); hAxis = gca; hAxis.YAxisLocation = 'left'; % 'left' (default) or 'right' hAxis.XAxisLocation = 'bottom'; % 'bottom' (default) or 'top' Default axis locations: axes crossover is non-fixed The crossover location is non-fixed in the sense that if we zoom or pan the plot, the axes crossover will remain at the bottom-left corner, which changes its coordinates depending on the X and Y axes limits. Since R2016a, we can also specify 'origin' for either of these properties, such that the X and/or Y axes pass through the chart origin (0,0) location. For example, move the YAxisLocation to the origin: hAxis.YAxisLocation = 'origin'; Y-axis location at origin: axes crossover at 0 (fixed), -1 (non-fixed) And similarly also for XAxisLocation: hAxis.XAxisLocation = 'origin'; X and Y-axis location at origin: axes crossover fixed at (0,0) The axes crossover location is now fixed at the origin (0,0), so as we move or pan the plot, the crossover location changes its position in the chart area, without changing its coordinates. This functionality has existed in other graphic packages (outside Matlab) for a long time and until now required quite a bit of coding to emulate in Matlab, so I’m glad that we now have it in Matlab by simply updating a single property value. MathWorks did a very nice job here of dynamically updating the axles, ticks and labels as we pan (drag) the plot towards the edges – try it out! ### The undocumented juicy stuff So far for the documented stuff. The undocumented aspect is that we are not limited to using the (0,0) origin point as the fixed axes crossover location. We can use any x,y crossover location, using the FirstCrossoverValue property of the axes’ hidden XRuler and YRuler properties. In fact, we could do this since R2014b, when the new HG2 graphics engine was released, not just starting in R2016a! % Set a fixed crossover location of (pi/2,-0.4) hAxis.YRuler.FirstCrossoverValue = pi/2; hAxis.XRuler.FirstCrossoverValue = -0.4; Custom fixed axes crossover location at (π/2,-0.4) For some reason (bug?), setting XAxisLocation/YAxisLocation to ‘origin’ has no visible effect in 3D plots, nor is there any corresponding ZAxisLocation property. Luckily, we can set the axes crossover location(s) in 3D plots using FirstCrossoverValue just as easily as for 2D plots. The rulers also have a SecondCrossoverValue property (default = -inf) that controls the Z-axis crossover, as Yaroslav pointed out in a comment below. For example: N = 49; x = linspace(-10,10,N); M = peaks(N); mesh(x,x,M); Default crossover locations at (-10,±10,-10) hAxis.XRuler.FirstCrossoverValue = 0; % X crossover with Y axis hAxis.YRuler.FirstCrossoverValue = 0; % Y crossover with X axis hAxis.ZRuler.FirstCrossoverValue = 0; % Z crossover with X axis hAxis.ZRuler.SecondCrossoverValue = 0; % Z crossover with Y axis Custom fixed axes crossover location at (0,0,-10) hAxis.XRuler.SecondCrossoverValue = 0; % X crossover with Z axis hAxis.YRuler.SecondCrossoverValue = 0; % Y crossover with Z axis Custom fixed axes crossover location at (0,0,0) ### Labels Users will encounter the following unexpected behavior (bug?) when using either the documented *AxisLocation or the undocumented FirstCrossoverValue properties: when setting an x-label (using the xlabel function, or the internal axes properties), the label moves from the center of the axes (as happens when XAxisLocation=’top’ or ‘bottom’) to the right side of the axes, where the secondary label (e.g., exponent) usually appears, whereas the secondary label is moved to the left side of the axis: Unexpected label positions In such cases, we would expect the labels locations to be reversed, with the main label on the left and the secondary label in its customary location on the right. The exact same situation occurs with the Y labels, where the main label unexpectedly appears at the top and the secondary at the bottom. Hopefully MathWorks will fix this in the next release (it is probably too late to make it into R2016b, but hopefully R2017a). Until then, we can simply switch the strings of the main and secondary label to make them appear at the expected locations: % Switch the Y-axes labels: ylabel(hAxis, '\times10^{3}'); % display secondary ylabel (x10^3) at top set(hAxis.YRuler.SecondaryLabel, 'Visible','on', 'String','main y-label'); % main label at bottom % Switch the X-axes labels: xlabel(hAxis, '2^{nd} label'); % display secondary xlabel at right set(hAxis.XRuler.SecondaryLabel, 'Visible','on', 'String','xlabel'); % main label at left As can be seen from the screenshot, there’s an additional nuisance: the main label appears a bit larger than the axes font size (the secondary label uses the correct font size). This is because by default Matlab uses a 110% font-size for the main axes label, ostensibly to make them stand out. We can modify this default factor using the rulers’ hidden LabelFontSizeMultiplier property (default=1.1). For example: hAxis.YRuler.LabelFontSizeMultiplier = 1; % use 100% font-size (same as tick labels) hAxis.XRuler.LabelFontSizeMultiplier = 0.8; % use 80% (smaller than standard) font-size Note: I described the ruler objects in my first article of the axes series. Feel free to read it for more ideas on customizing the axes rulers. ]]> https://undocumentedmatlab.com/blog/customizing-axes-part-5-origin-crossover-and-labels/feed 5 Customizing uifigures part 1https://undocumentedmatlab.com/blog/customizing-uifigures-part-1 https://undocumentedmatlab.com/blog/customizing-uifigures-part-1#comments Thu, 21 Jul 2016 10:32:51 +0000 http://undocumentedmatlab.com/?p=6554   Related posts: 1. HG2 update HG2 appears to be nearing release. It is now a stable mature system. ... 2. Customizing print setup Matlab figures print-setup can be customized to automatically prepare the figure for printing in a specific configuration... 3. Plot LineSmoothing property LineSmoothing is a hidden and undocumented plot line property that creates anti-aliased (smooth unpixelized) lines in Matlab plots... 4. getundoc – get undocumented object properties getundoc is a very simple utility that displays the hidden (undocumented) properties of a specified handle object....   ]]> Last month, I posted an article that summarized a variety of undocumented customizations to Matlab figure windows. As I noted in that post, Matlab figures have used Java JFrames as their underlying technology since R14 (over a decade ago), but this is expected to change a few years from now with the advent of web-based uifigures. uifigures first became available in late 2014 with the new App Designer preview (the much-awaited GUIDE replacement), and were officially released in R2016a. AppDesigner is actively being developed and we should expect to see exciting new features in upcoming Matlab releases. Matlab's new AppDesigner (a somewhat outdated screenshot) However, while AppDesigner has become officially supported, the underlying technology used for the new uifigures remained undocumented. This is not surprising: MathWorks did a good job of retaining backward compatibility with the existing figure handle, and so a new uifigure returns a handle that programmatically appears similar to figure handles, reducing the migration cost when MathWorks decides (presumably around 2018-2020) that web-based (rather than Java-based) figures should become the default figure type. By keeping the underlying figure technology undocumented and retaining the documented top-level behavior (properties and methods of the figure handle), Matlab users who only use the documented interface should expect a relatively smooth transition at that time. So does this mean that users who start using AppDesigner today (and especially in a few years when web figures become the default) can no longer enjoy the benefits of figure-based customization offered to the existing Java-based figure users (which I listed in last month’s post)? Absolutely not! All we need is to get a hook into the uifigure‘s underlying object and then we can start having fun. ### The uifigure Controller One way to do this is to use the uifigure handle’s hidden (private) Controller property (a matlab.ui.internal.controller.FigureController MCOS object whose source-code appears in %matlabroot%/toolbox/matlab/uitools/uicomponents/components/+matlab/+ui/+internal/+controller/). Controller is not only a hidden but also a private property of the figure handle, so we cannot simply use the get function to get its value. This doesn’t stop us of course: We can get the controller object using either my getundoc utility or the builtin struct function (which returns private/protected properties as an undocumented feature): >> hFig = uifigure('Name','Yair', ...); >> figProps = struct(hFig); % or getundoc(hFig) Warning: Calling STRUCT on an object prevents the object from hiding its implementation details and should thus be avoided. Use DISP or DISPLAY to see the visible public details of an object. See 'help struct' for more information. (Type "warning off MATLAB:structOnObject" to suppress this warning.) Warning: figure JavaFrame property will be obsoleted in a future release. For more information see the JavaFrame resource on the MathWorks web site. (Type "warning off MATLAB:HandleGraphics:ObsoletedProperty:JavaFrame" to suppress this warning.) figProps = JavaFrame: [] JavaFrame_I: [] Position: [87 40 584 465] PositionMode: 'auto' ... Controller: [1x1 matlab.ui.internal.controller.FigureController] ControllerMode: 'auto' ... >> figProps.Controller ans = FigureController with properties: Canvas: [] ProxyView: [1x1 struct] >> figProps.Controller.ProxyView ans = PeerNode: [1x1 com.mathworks.peermodel.impl.PeerNodeImpl] PeerModelManager: [1x1 com.mathworks.peermodel.impl.PeerModelManagerImpl] >> struct(figProps.Controller) Warning: Calling STRUCT on an object prevents the object from hiding its implementation details and should thus be avoided. Use DISP or DISPLAY to see the visible public details of an object. See 'help struct' for more information. (Type "warning off MATLAB:structOnObject" to suppress this warning.) ans = PositionListener: [1x1 event.listener] ContainerPositionCorrection: [1 1 0 0] Container: [1x1 matlab.ui.internal.controller.FigureContainer] Canvas: [] IsClientReady: 1 PeerEventListener: [1x1 handle.listener] ProxyView: [1x1 struct] Model: [1x1 Figure] ParentController: [0x0 handle] PropertyManagementService: [1x1 matlab.ui.internal.componentframework.services.core.propertymanagement.PropertyManagementService] IdentificationService: [1x1 matlab.ui.internal.componentframework.services.core.identification.WebIdentificationService] EventHandlingService: [1x1 matlab.ui.internal.componentframework.services.core.eventhandling.WebEventHandlingService] I will discuss all the goodies here in a future post (if you are curious then feel free to start drilling in there yourself, I promise it won’t bite you…). However, today I wish to concentrate on more immediate benefits from a different venue: ### The uifigure webwindow uifigures are basically webpages rather than desktop windows (JFrames). They use an entirely different UI mechanism, based on HTML webpages served from a localhost webserver that runs CEF (Chromium Embedded Framework version 3.2272 on Chromium 41 in R2016a). This runs the so-called CEF client (apparently an adaptation of the CefClient sample application that comes with CEF; the relevant Matlab source-code is in %matlabroot%/toolbox/matlab/cefclient/). It uses the DOJO Javascript toolkit for UI controls visualization and interaction, rather than Java Swing as in the existing JFrame figures. I still don’t know if there is a way to combine the seemingly disparate sets of GUIs (namely adding Java-based controls to web-based figures or vice-versa). Anyway, the important thing to note for my purposes today is that when a new uifigure is created, the above-mentioned Controller object is created, which in turn creates a new matlab.internal.webwindow. The webwindow class (%matlabroot%/toolbox/matlab/cefclient/+matlab/+internal/webwindow.m) is well-documented and easy to follow (although the non camel-cased class name escaped someone’s attention), and allows access to several important figure-level customizations. The figure’s webwindow reference can be accessed via the Controller‘s Container‘s CEF property: >> hFig = uifigure('Name','Yair', ...); >> warning off MATLAB:structOnObject % suppress warning (yes, we know it's naughty...) >> figProps = struct(hFig); >> controller = figProps.Controller; % Controller is a private hidden property of Figure >> controllerProps = struct(controller); >> container = controllerProps.Container % Container is a private hidden property of FigureController container = FigureContainer with properties: FigurePeerNode: [1x1 com.mathworks.peermodel.impl.PeerNodeImpl] Resizable: 1 Position: [86 39 584 465] Tag: '' Title: 'Yair' Icon: 'C:\Program Files\Matlab\R2016a\toolbox\matlab\uitools\uicomponents\resources\images…' Visible: 1 URL: 'http://localhost:31417/toolbox/matlab/uitools/uifigureappjs/componentContainer.html…' HTML: 'toolbox/matlab/uitools/uifigureappjs/componentContainer.html' ConnectorPort: 31417 DebugPort: 0 IsWindowValid: 1 >> win = container.CEF % CEF is a regular (public) hidden property of FigureContainer win = webwindow with properties: URL: 'http://localhost:31417/toolbox/matlab/uitools/uifigureappjs/component…' Title: 'Yair' Icon: 'C:\Program Files\Matlab\R2016a\toolbox\matlab\uitools\uicomponents\re…' Position: [86 39 584 465] CustomWindowClosingCallback: @(o,e)this.Model.hgclose() CustomWindowResizingCallback: @(event,data)resizeRequest(this,event,data) WindowResizing: [] WindowResized: [] FocusGained: [] FocusLost: [] DownloadCallback: [] PageLoadFinishedCallback: [] MATLABClosingCallback: [] MATLABWindowExitedCallback: [] PopUpWindowCallback: [] RemoteDebuggingPort: 0 CEFVersion: '3.2272.2072' ChromiumVersion: '41.0.2272.76' isWindowValid: 1 isDownloadingFile: 0 isModal: 0 isWindowActive: 1 isAlwaysOnTop: 0 isAllActive: 1 isResizable: 1 MaxSize: [] MinSize: [] >> win.URL ans = http://localhost:31417/toolbox/matlab/uitools/uifigureappjs/componentContainer.html?channel=/uicontainer/393ed66a-5e34-41f3-8ac0-0b0f3b0738cd&snc=5C2353 An alternative way to get the webwindow is via the list of all webwindows stored by a central webwindowmanager: webWindows = matlab.internal.webwindowmanager.instance.findAllWebwindows(); % manager method returning an array of all open webwindows webWindows = matlab.internal.webwindowmanager.instance.windowList; % equivalent alternative via manager's windowList property Note that the controller, container and webwindow class objects, like most Matlab MCOS objects, have internal (hidden) properties/methods that you can explore. For example: >> getundoc(win) ans = Channel: [1x1 asyncio.Channel] CustomEventListener: [1x1 event.listener] InitialPosition: [100 100 600 400] JavaScriptReturnStatus: [] JavaScriptReturnValue: [] NewWindowBeingCreated: 0 NewWindowCreated: 1 UpdatedPosition: [86 39 584 465] WindowHandle: 2559756 newURL: 'http://localhost:31417/toolbox/matlab/uitools/uifigureappjs/componentContai…' ### Using webwindow for figure-level customizations We can use the methods of this webwindow object as follows: win.setAlwaysOnTop(true); % always on top of other figure windows (a.k.a. AOT) win.hide(); win.show(); win.bringToFront(); win.minimize(); win.maximize(); win.restore(); win.setMaxSize([400,600]); % enables resizing up to this size but not larger (default=[]) win.setMinSize([200,300]); % enables resizing down to this size but not smaller (default=[]) win.setResizable(false); win.setWindowAsModal(true); win.setActivateCurrentWindow(false); % disable interaction with this entire window win.setActivateAllWindows(false); % disable interaction with *ALL* uifigure (but not Java-based) windows result = win.executeJS(jsStr, timeout); % run JavaScript In addition to these methods, we can set callback functions to various callbacks exposed by the webwindow as regular properties (too bad that some of their names [like the class name itself] don’t follow Matlab’s standard naming convention, in this case by appending “Fcn” or “Callback”): win.FocusGained = @someCallbackFunc; win.FocusLost = @anotherCallbackFunc; In summary, while the possible customizations to Java-based figure windows are more extensive, the webwindow methods appear to cover most of the important ones. Since these functionalities (maximize/minimize, AOT, disable etc.) are now common to both the Java and web-based figures, I really hope that MathWorks will create fully-documented figure properties/methods for them. Now that there is no longer any question whether these features will be supported by the future technology, and since there is no question as to their usefulness, there is really no reason not to officially support them in both figure types. If you feel the same as I do, please let MathWorks know about this – if enough people request this, MathWorks will be more likely to add these features to one of the upcoming Matlab releases. Warning: the internal implementation is subject to change across releases, so be careful to make your code cross-release compatible whenever you rely on one of Matlab’s internal objects. Note that I labeled this post as “part 1” – I expect to post additional articles on uifigure customizations in upcoming years. ]]> https://undocumentedmatlab.com/blog/customizing-uifigures-part-1/feed 5 A few parfor tipshttps://undocumentedmatlab.com/blog/a-few-parfor-tips https://undocumentedmatlab.com/blog/a-few-parfor-tips#comments Wed, 06 Jul 2016 16:29:21 +0000 http://undocumentedmatlab.com/?p=6516   Related posts: 1. ismembc – undocumented helper function Matlab has several undocumented internal helper functions that can be useful on their own in some cases. This post presents the ismembc function.... 2. Datenum performance The performance of the built-in Matlab function datenum can be significantly improved by using an undocumented internal help function... 3. sprintfc – undocumented helper function The built-in sprintfc function can be used to quickly generate a cell-array of formatted strings. ... 4. Profiling Matlab memory usage mtic and mtoc were a couple of undocumented features that enabled users of past Matlab releases to easily profile memory usage. ...   ]]> Matlab Expo 2016 keynote presentation A few days ago, MathWorks uploaded a video recording of my recent keynote presentation at the Matlab Expo 2016 in Munich, Germany. During the presentation, I skimmed over a few tips for improving performance of parallel-processing (parfor) loops. In today’s post I plan to expand on these tips, as well as provide a few others that for lack of space and time I did not mention in the presentation. The overall effect can be dramatic: The performance (speed) difference between a sub-optimal and optimized parfor‘ed code can be up to a full order of magnitude, depending on the specific situation. Naturally, to use any of today’s tips, you need to have MathWorks’ Parallel Computing Toolbox (PCT). Before diving into the technical details, let me say that MathWorks has extensive documentation on PCT. In today’s post I will try not to reiterate the official tips, but rather those that I have not found mentioned elsewhere, and/or are not well-known (my apologies in advance if I missed an official mention of one or more of the following). Furthermore, I limit myself only to parfor in this post: much can be said about spmd, GPU and other parallel constructs, but not today. ### parpool(numCores) The first tip is to not [always] use the default number of workers created by parpool (or matlabpool in R2013a or earlier). By default, Matlab creates as many workers as logical CPU cores. On Intel CPUs, the OS reports two logical cores per each physical core due to hyper-threading, for a total of 4 workers on a dual-core machine. However, in many situations, hyperthreading does not improve the performance of a program and may even degrade it (I deliberately wish to avoid the heated debate over this: you can find endless discussions about it online and decide for yourself). Coupled with the non-negligible overhead of starting, coordinating and communicating with twice as many Matlab instances (workers are headless [=GUI-less] Matlab processes after all), we reach a conclusion that it may actually be better in many cases to use only as many workers as physical (not logical) cores. I know the documentation and configuration panel seem to imply that parpool uses the number of physical cores by default, but in my tests I have seen otherwise (namely, logical cores). Maybe this is system-dependent, and maybe there is a switch somewhere that controls this, I don’t know. I just know that in many cases I found it beneficial to reduce the number of workers to the actual number of physical cores: p = parpool; % use as many workers as logical CPUs (4 on my poor laptop...) p = parpool(2); % use only 2 parallel workers Of course, this can vary greatly across programs and platforms, so you should test carefully on your specific setup. I suspect that for the majority of Matlab programs it would turn out that using the number of physical cores is better. It would of course be better to dynamically retrieve the number of physical cores, rather than hard-coding a constant value (number of workers) into our program. We can get this value in Matlab using the undocumented feature(‘numcores’) function: numCores = feature('numcores'); p = parpool(numCores); Running feature(‘numcores’) without assigning its output displays some general debugging information: >> feature('numcores') MATLAB detected: 2 physical cores. MATLAB detected: 4 logical cores. MATLAB was assigned: 4 logical cores by the OS. MATLAB is using: 2 logical cores. MATLAB is not using all logical cores because hyper-threading is enabled. ans = 2  Naturally, this specific tip is equally valid for both parfor loops and spmd blocks, since both of them use the pool of workers started by parpool. ### Running separate code in parfor loops The conventional wisdom is that parfor loops (and loops in general) can only run a single code segment over all its iterations. Of course, we can always use conditional constructs (such as if or switch) based on the data. But what if we wanted some workers to run a different code path than the other workers? In spmd blocks we could use a conditional based on the labindex value, but unfortunately labindex is always set to the same value 1 within parfor loops. So how can we let worker A run a different code path than worker B? An obvious answer is to create a parfor loop having as many elements as there are separate code paths, and use a switch-case mechanism to run the separate paths, as follows: % Naive implementation example - do NOT use! parfor idx = 1 : 3 switch idx case 1, result{1} = fun1(data1, data2); case 2, result{2} = fun2(data3, data4, data5); case 3, result{3} = fun3(data6); end end There are several problems with this naive implementation. First, it unnecessarily broadcasts all the input data to all workers (more about this issue below). Secondly, it appears clunky and too verbose. A very nice extension of this mechanism, posted by StackOverflow heavyweight Jonas, uses indexed arrays of function handles and input args, thereby solving both problems: funcList = {@fun1, @fun2, @fun3}; dataList = {data1, data2, data3}; %# or pass file names parfor idx = 1 : length(funcList) result{idx} = funcList{idx}(dataList{idx}); end ### Reduce the amount of broadcast data It is often easy, too-easy, to convert for loops into parfor loops. In many cases, all we need to do is to add the “par” prefix to the for keyword and we’re done (assuming we have no incompatibly-used variables that should be converted into sliced variables etc.). This transformation was intentionally made simple by MathWorks (which is great!). On the other hand, it also hides a lot under the hood. One of the things that is often overlooked in such simple loop transformations is that a large part of the data used within the loop needs to be copied (broadcast) to each of the workers separately. This means that each of the data items needs to be serialized (i.e., copied in memory), packaged, communicated to and accepted by each of the workers. This can mean a lot of memory, networking bandwidth and time-consuming. It can even mean thrashing to hard-disk in case the number of workers times the amount of transferred data exceeds the available RAM. For example, if we have 10GB available RAM and try to communicate 3GB to 4 workers, we will not have enough RAM and the OS will start swapping to hard-disk. This will kill performance and Matlab will appear “hung” and will need to be hard-killed. You might think that it would be very difficult to reach the RAM limit, but in fact it can be far too easy when you consider the multiplication by the number of workers, and the fact that each worker uses 1+GB of memory just for its MATLAB process, even before the data, and all this in addition to the parent (client) Matlab process. That’s a lot of GBs flying around… Moreover, it’s enough for one small part of a Matlab struct or array to be used within the parfor loop for the entire Matlab construct to be broadcast to all workers. For example, a very common use-case is to store program data, both raw and processed, within a simple Matlab struct. Let’s say that we have data.raw and data.processed and within the loop we only need data.processed – the entire data variable (which might include many GBs due to the raw data) is broadcast, although the loop’s code only needs data.processed. In such cases, it makes sense to separate the broadcast data into standalone variables, and only use them within the loop: data.raw = ... data.processed = ... % Inefficient variant: parfor idx = 1 : N % do something with data.processed end % This is better: processedData = data.processed; parfor idx = 1 : N % do something with processedData end Moreover, if you can convert a broadcast variable into a sliced one, this would be even better: in this case each worker will only be communicated its small share (“slice”) of the entire data, rather than a full copy of the entire data. All this would of course be much simpler if Matlab’s computational engine was multi-threaded, since then PCT could be implemented using lightweight threads rather than heavyweight processes. The memory and communication overheads would then be drastically reduced and performance would improve significantly. Unfortunately, Matlab’s computational engine is [still] single-threaded, preventing this. Hopefully Matlab’s new engine (which debuted in R2015b) will enable true multithreading at some future release. PCT will still need to retain an option of using headless worker processes to run on multiple machines (i.e., distributed/grid/cloud computing), but single-machine parallelization should employ multithreading instead. Additional speedup tips can be found in my book “Accelerating MATLAB Performance“. Do you have some other important parfor tips that you found useful? If so, please post them in a comment below. ]]> https://undocumentedmatlab.com/blog/a-few-parfor-tips/feed 6 rmfield performancehttps://undocumentedmatlab.com/blog/rmfield-performance https://undocumentedmatlab.com/blog/rmfield-performance#comments Wed, 25 May 2016 07:00:48 +0000 http://undocumentedmatlab.com/?p=6427   Related posts: 1. tic / toc – undocumented option Matlab's built-in tic/toc functions have an undocumented option enabling multiple nested clockings... 2. Solving a MATLAB bug by subclassing Matlab's Image Processing Toolbox's impoint function contains an annoying bug that can be fixed using some undocumented properties.... 3. Plot LimInclude properties The plot objects' XLimInclude, YLimInclude, ZLimInclude, ALimInclude and CLimInclude properties are an important feature, that has both functional and performance implications.... 4. Class object creation performance Performance aspects of Matlab class object creation are discussed, with specific suggestions. ...   ]]> Once again I would like to introduce guest blogger Hanan Kavitz of Applied Materials. Several months ago Hanan discussed some quirks with compiled Matlab DLLs. Today Hanan will discuss how they overcame a performance bottleneck with Matlab’s builtin rmfield function, exemplifying the general idea that we can sometimes improve performance by profiling the core functionality that causes a performance hotspot and optimizing it, even when it is part of a builtin Matlab function. For additional ideas of improving Matlab peformance, search this blog for “Performance” articles, and/or get the book “Accelerating MATLAB Performance“. I’ve been using Matlab for many years now and from time to time I need to profile low-throughput code. When I profile this code sometimes I realize that a computational ‘bottleneck’ is due to a builtin Matlab function (part of the core language). I can often find ways to accelerate such builtin functions and get significant speedup in my code. I recently found Matlab’s builtin rmfield function being too slow for my needs. It works great when one needs to remove a few fields from a small structure, but in our case we needed to remove thousands of fields from a structure containing about 5000 fields – and this is executed in a function that is called many times inside an external loop. The program was significantly sluggish. It started when a co-worker asked me to look at a code that looked just slightly more intelligent than this: for i = 1:5000 myStruct = rmfield(myStruct,fieldNames{i}); end Running this code within a tic/toc pair yielded the following results: >> tic; myFunc(); t1 = toc t1 = 25.7713 In my opinion 25.77 secs for such a simple functionality seems like an eternity… The obvious thing was to change the code to the documented faster (vectorized) version: >> tic; myStruct = rmfield(myStruct,fieldNames); t2 = toc t2 = 0.6097 This is obviously much better but since rmfield is called many times in my application, I needed something even better. So I profiled rmfield and was not happy with the result. The original code of rmfield (%matlabroot%/toolbox/matlab/datatypes/rmfield.m) looks something like this (I deleted some non-essential code for brevity): function t = rmfield(s,field) % get fieldnames of struct f = fieldnames(s); % Determine which fieldnames to delete. idxremove = []; for i=1:length(field) j = find(strcmp(field{i},f) == true); idxremove = [idxremove;j]; end % set indices of fields to keep idxkeep = 1:length(f); idxkeep(idxremove) = []; % remove the specified fieldnames from the list of fieldnames. f(idxremove,:) = []; % convert struct to cell array c = struct2cell(s); % find size of cell array sizeofarray = size(c); newsizeofarray = sizeofarray; % adjust size for fields to be removed newsizeofarray(1) = sizeofarray(1) - length(idxremove); % rebuild struct t = cell2struct(reshape(c(idxkeep,:),newsizeofarray),f); When I profiled the code, the highlighted row was the bottleneck I was looking for. First, I noticed the string comparison equals to true part – while '==true' is not the cause of the bottleneck, it does leave an impression of bad coding style Perhaps this code was created as some apprentice project, which might also explain its suboptimal performance. The real performance problem here is that for each field that we wish to remove, rmfield compares it to all existing fields to find its location in a cell array of field names. This is algorithmically inefficient and makes the code hard to understand (just try – it took me hard, long minutes). So, I created a variant of rmfield.m called fast_rmfield.m, as follows (again, omitting some non-essential code): function t = fast_rmfield(s,field) % get fieldnames of struct f = fieldnames(s); [f,ia] = setdiff(f,field,'R2012a'); % convert struct to cell array c = squeeze(struct2cell(s)); % rebuild struct t = cell2struct(c(ia,:),f)'; This code is much shorter, easier to explain and maintain, but also (and most importantly) much faster: >> tic; myStruct = fast_rmfield(myStruct,fieldNames); t3 = toc t3 = 0.0302 >> t2/t3 ans = 20.1893 This resulted in a speedup of ~850x compared to the original version (of 25.77 secs), and ~20x compared to the vectorized version. A nice improvement in my humble opinion… The point in all this is that we can and should rewrite Matlab builtin functions when they are too slow for our needs, whether it is found to be an algorithmic flaw (as in this case), extraneous sanity checks (as in the case of ismember or datenum), bad default parameters (as in the case of fopen/fwrite or scatter), or merely slow implementation (as in the case of save, cellfun, or the conv family of functions). A good pattern is to save such code pieces in file names that hint to the original code. In our case, I used fast_rmfield to suggest that it is a faster alternative to rmfield. Do you know of any other example of a slow implementation in a built-in Matlab function that can be optimized? If so, please leave a comment below. ]]> https://undocumentedmatlab.com/blog/rmfield-performance/feed 5 Viewing saved profiling resultshttps://undocumentedmatlab.com/blog/viewing-saved-profiling-results https://undocumentedmatlab.com/blog/viewing-saved-profiling-results#respond Wed, 18 May 2016 18:00:37 +0000 http://undocumentedmatlab.com/?p=6421   Related posts: 1. Plot performance Undocumented inner plot mechanisms can significantly improve plotting performance ... 2. Undocumented Profiler options part 2 Several undocumented features of the Matlab Profiler can make it much more useful - part 2 of series. ... 3. Undocumented Profiler options part 3 An undocumented feature of the Matlab Profiler can report call history timeline - part 3 of series. ... 4. Undocumented Profiler options part 4 Several undocumented features of the Matlab Profiler can make it much more useful - part 4 of series. ...   ]]> Many Matlab users know and utilize Matlab’s built-in Profiler tool to identify performance bottlenecks and code-coverage issues. Unfortunately, not many are aware of the Profiler’s programmatic interface. In past articles as well as my performance book I explained how we can use this programmatic interface to save profiling results and analyze it offline. In fact, I took this idea further and even created a utility (profile_history) that displays the function call timeline in a standalone Matlab GUI, something that is a sorely missed feature in the built-in profiler: Function call timeline profiling (click for full-size image) Today I will discuss a related undocumented feature of the Profiler: loading and viewing pre-saved profiling results. ### Programmatic access to profiling results Matlab’s syntax for returning the detailed profiling results in a data struct is clearly documented in the profile function’s doc page. Although the documentation does not explain the resulting struct and sub-struct fields, they have meaningful names and we can relatively easily infer what each of them means (I added a few annotation comments for clarity): >> profile('on','-history') >> surf(peaks); drawnow >> profile('off') >> profData = profile('info') profData = FunctionTable: [26x1 struct] FunctionHistory: [2x56 double] ClockPrecision: 4.10517962829241e-07 ClockSpeed: 2501000000 Name: 'MATLAB' Overhead: 0 >> profData.FunctionTable(1) ans = CompleteName: 'C:\Program Files\Matlab\R2016a\toolbox\matlab\specgraph\peaks.m>peaks' FunctionName: 'peaks' FileName: 'C:\Program Files\Matlab\R2016a\toolbox\matlab\specgraph\peaks.m' Type: 'M-function' Children: [1x1 struct] Parents: [0x1 struct] ExecutedLines: [9x3 double] IsRecursive: 0 TotalRecursiveTime: 0 PartialData: 0 NumCalls: 1 TotalTime: 0.0191679078068094 >> profData.FunctionTable(1).Children ans = Index: 2 % index in profData.FunctionTable array NumCalls: 1 TotalTime: 0.00136415141013509 >> profData.FunctionTable(1).ExecutedLines % line number, number of calls, duration in secs ans = 43 1 0.000160102031282782 44 1 2.29890096200918e-05 45 1 0.00647592190637408 56 1 0.0017093970724654 57 1 0.00145036019621044 58 1 0.000304193859437286 60 1 4.39254290955326e-05 62 1 3.44835144301377e-05 63 1 0.000138755093778411 >> profData.FunctionHistory(:,1:5) ans = 0 0 1 1 0 % 0=enter, 1=exit 1 2 2 1 6 % index in profData.FunctionHistory array As we can see, this is pretty intuitive so far. ### Loading and viewing saved profiling results If we wish to save these results results in a file and later load and display them in the Profiler’s visualization browser, then we need to venture deeper into undocumented territory. It seems that while retrieving the profiling results (via profile(‘info’)) is fully documented, doing the natural complementary action (namely, loading this data into the viewer) is not. For the life of me I cannot understand the logic behind this decision, but that’s the way it is. Luckily, the semi-documented built-in function profview does exactly what we need: profview accepts 2 input args (function name and the profData struct) and displays the resulting profiling info. The first input arg (function name) accepts either a string (e.g., 'peaks' or 'view>isAxesHandle'), or the numeric value 0 which signifies the home (top-level) page: profView(0, profData); % display profiling home (top-level) page profview('peaks', profData); % display a specific profiling page I use the 0 input value much more frequently than the string inputs, because I often don’t know which functions exactly were profiled, and starting at the home page enables me to easily drill-down the profiling results interactively. ### Loading saved profiling results from a different computer Things get slightly complicated if we try to load saved profiling results from a different computer. If the other computer has exactly the same folder structure as our computer, and all our Matlab functions reside in exactly the same disk folders/path, then everything will work out of the box. The problem is that in general the other computer will have the functions in different folders. When we then try to load the profData on our computer, it will not find the associated Matlab functinos in order to display the line-by-line profiling results. We will only see the profiling data at the function level, not line level. This significantly reduces the usefulness of the profiling data. The Profiler page will display the following error message: This file was modified during or after profiling. Function listing disabled. We can solve this problem in either of two ways: 1. Modify our profData to use the correct folder path on the local computer, rather than the other computer’s path (which is invalid on the local computer). For example: % Save the profData on computer #1: profData = profile('info'); save('profData.mat', 'profData'); % Load the profData on computer #2: fileData = load('profData.mat'); profData = fileData.profData; path1 = 'N:\Users\Juan\programs\myProgram'; path2 = 'C:\Yair\consulting\clients\Intel\code'; for idx = 1 : numel(profData.FunctionTable) funcData = profData.FunctionTable(idx); funcData.FileName = strrep(funcData.FileName, path1, path2); funcData.CompleteName = strrep(funcData.CompleteName, path1, path2); profData.FunctionTable(idx) = funcData; end % note: this loop can be vectorized if you wish 2. As an alternative, we can modify Matlab’s profview.m function (%matlabroot%/toolbox/matlab/codetools/profview.m) to search for the function’s source code in the current Matlab path, if the specified direct path is not found (note that changing profview.m may require administrator priviledges). For example, the following is the code from R2016a’s profview.m file, line #506:  % g894021 - Make sure the MATLAB code file still exists if ~exist(fullName, 'file') [~,fname,fext] = fileparts(fullName); % Yair fname = which([fname fext]); % Yair if isempty(fname) % Yair mFileFlag = 0; else % Yair fullName = fname; % Yair end % Yair end These two workarounds complement each other: the first workaround does not require changing any installed Matlab code, and so is platform- and release-independent, but would require rerunning the code snippet for each and every profiling data file that we receive from external computers. On the other hand, the second workaround is a one-time operation that should work for multiple saved profiling results, although we would need to redo it whenever we install Matlab. ### Additional profview customizations Modifying the profview.m function can be used for different improvements as well. For example, several years ago I explained how this function can be modified to display 1 ms timing resolutions, rather than the default 10 mS. Another customization that I often do after I install Matlab is to change the default setting of truncating function lines longer than 40 characters – I typically modify this to 60 or 80 (depending on the computer monitor’s size…). All we need to do is to update the truncateDisplayName sub-function within profview.m as follows (taken from R2016a again, line #1762): function shortFileName = truncateDisplayName(longFileName,maxNameLen) %TRUNCATEDISPLAYNAME Truncate the name if it gets too long maxNameLen = max(60,maxNameLen); % YairshortFileName = escapeHtml(longFileName); if length(longFileName) > maxNameLen, shortFileName = char(com.mathworks.util.FileUtils.truncatePathname( ... shortFileName, maxNameLen)); end You can see additional undocumented profiling features in the “Related posts” section below, as well as in Chapter 2 of my book “Accelerating MATLAB Performance“. Do you have any other customization to the profiling results? If so, please share it in a comment. ]]> https://undocumentedmatlab.com/blog/viewing-saved-profiling-results/feed 0 Setting class property types – take 2https://undocumentedmatlab.com/blog/setting-class-property-types-2 https://undocumentedmatlab.com/blog/setting-class-property-types-2#comments Wed, 27 Apr 2016 20:20:22 +0000 http://undocumentedmatlab.com/?p=6390   Related posts: 1. Undocumented classdef attributes Matlab's object-oriented class definition enables usage of some useful undocumented attributes. ... 2. Setting class property types Matlab's class properties have a simple and effective mechanism for setting their type.... 3. Class object tab completion & improper field names Tab completions and property access can be customized for user-created Matlab classes. ... 4. Simulink Data Dictionary Simulink contains undocumented public API for access to its data dictionary functionality. ...   ]]> Three years ago, almost to the day, I wrote about a very handy undocumented feature of Matlab classes that enables us to specify type restrictions for any Matlab class property. We can specify property type (for example, char, double or any Matlab class) as well as dimensionality (scalar, vector, or matrix) and complexity indication (complex). Doing so has multiple benefits for code performance, robustness and maintainability. For example: % Undocumented syntax - works well since at least R2010a (possibly earlier) classdef Packet properties PacketType@char HeaderLength@uint16 PayloadLength@uint16 scalar = uint16(0); % initial value PacketData@uint8 vector end end In the recent release of Matlab R2016a, a similar feature have finally become fully supported and documented. The corresponding snippet above would look something like this: % Documented syntax - only works in R2016a or newer classdef Packet properties PacketType char HeaderLength uint16 PayloadLength uint16 = uint16(0); % initial value PacketData uint8 end end Unfortunately, I dislike the new documented functionality, so I didn’t feel like promoting it in this blog when it came out. But since a blog reader mentioned it a few days ago, I wanted to come out publicly with my opinion and a detailed explanation. If you look closely at the code snippets above, you will notice two important differences: 1. The “@” symbol was replaced with a space 2. The dimensionality and complexity cannot be specified The new syntax has some drawbacks compared to the previous (undocumented) one: 1. Backward compatibility – We can run the older (undocumented) syntax on any Matlab release since who-knows-when (at least as far back as R2010a [tested], and possibly older releases [untested]), including the very latest R2016a. On the other hand, the new (documented) syntax will only work on R2016a and will crash the program if you try to run it in older releases. This is not even something that you can catch with a try-catch block – the class will simply not load on any older Matlab release. If you need your code to run on older releases in addition to 16a, you have no choice other than to use the older syntax. 2. Dimensionality – the new syntax, unlike the undocumented syntax, does not enable users to limit the data dimensionality (scalar/vector/array). This is a very important feature for program robustness and maintainability. Complexity is another type limitation that is missing, although it is less important than the dimensionality. And just in case you were wondering – the new syntax does not accept the additional scalar, vector, matrix and complex attributes like the older syntax; using them with the new syntax evokes an error. 3. Cross compatibility – it is very confusing to users coming to Matlab from other programming languages, all of which (without any important exception) place the type name to the LEFT of the identifier name, not to its RIGHT. People coding in both Matlab and Java/C/C++ would easily get confused and frustrated. 4. Consistency – despite what I hoped, the new syntax still does not apply to function input args: we cannot (AFAIK) limit the input/output args of methods/functions in the same way that we can limit properties. If there’s a way to do this, I’d be delighted to learn (this comment may indicate that it is work in progress). It is true that this feature is not a drawback of the new syntax compared to the older one, since the old syntax didn’t have it either (AFAIK). But I would have expected a documented feature to be consistent across the Matlab language (or at least across the MCOS subset), and unfortunately the new feature fails this test. In fact, aside from the fact that the new syntax is documented, I can see no advantages that it offers over the older syntax, only disadvantages. Or am I missing something? Please do tell if you see any important advantages that I’ve missed. Luckily for us, the old syntax remains operational, side-by-side with the new one. This enables us to keep running our existing code without worrying [too much] that it might break in R2016a. Maybe the new syntax will grow on me (or improve) in upcoming years, but for the time being I see no benefit in switching away from the @ syntax. For the past few years, I hoped that the property typing feature will become documented and that it will be a continuation of the undocumented syntax rather than what eventually aired. I’m afraid it’s too late to revert it now that it has… Realistically speaking, the best we can hope for now is for the older syntax to remain operational, and not be withdrawn in some future Matlab release. Making the undocumented syntax documented as-is would be great, but I’m afraid it is unrealistic given the new circumstances. I’m sorry if I take the wind off MathWorks’ sails a bit here, but MathWorks knows that it can count on me to speak my mind without bullshit. Sometimes for the good, sometimes not. All in good spirit and the common interest of improving Matlab over time. No offence intended – it’s just my personal opinion after all. In my opinion this is one of those rare cases where the developers obviously intended to make something better but eventually came out with something worse. They should have stuck to what was. After all, the first and foremost rule of engineering is, and always was: Don’t fix it if it ain’t broke! ]]> https://undocumentedmatlab.com/blog/setting-class-property-types-2/feed 11 Secure SSL connection between Matlab and PostgreSQLhttps://undocumentedmatlab.com/blog/secure-ssl-connection-between-matlab-and-postgresql https://undocumentedmatlab.com/blog/secure-ssl-connection-between-matlab-and-postgresql#respond Fri, 18 Mar 2016 10:39:49 +0000 http://undocumentedmatlab.com/?p=6318   Related posts: 1. Matlab’s internal memory representation Matlab's internal memory structure is explored and discussed. ... 2. uiundo – Matlab’s undocumented undo/redo manager The built-in uiundo function provides easy yet undocumented access to Matlab's powerful undo/redo functionality. This article explains its usage.... 3. JMI wrapper – local MatlabControl part 2 An example using matlabcontrol for calling Matlab from within a Java class is explained and discussed... 4. Creating a simple UDD class This article explains how to create and test custom UDD packages, classes and objects...   ]]> I’d like to introduce guest blogger Jeff Mandel of the Perelman School of Medicine at the University of Pennsylvania. Today Jeff will discuss a how-to guide for setting up an SSL connection between Matlab and a PostgreSQL database. While this specific topic may be of interest to only a few readers, it involves hard-to-trace problems that are not well documented anywhere. The techniques discussed below may also be applicable, with necessary modifications, to other SSL targets and may thus be of use to a wider group of Matlab users. I’m developing software for pharmacokinetic control, and needed secure access to a central database from users at remote sites. The client software is written in Matlab, and while I have targeted MacOS, this could be adapted to Windows fairly easily. Hopefully, this will save someone the week it took me to figure all this out. My environment: • PostgreSQL 9.4 installed on the server (Windows 7 PC, but Linux would be equally good) • DynDNS CNAME pointing at the server (diseserver.mydomain.org) • CACert.org registration for domain mydomain.org • Matlab 2015b running on El Capitan Here are the neccesary steps: 1. First, we need a certificate for the server. We can generate this with OpenSSL: openssl req -out diseserver.csr -new -newkey rsa:2048 -nodes -keyout diseserver.key

Specify any information you want on the key, but ensure CN=diseserver.mydomain.org.

2. Paste the resulting diseserver.csr file into a new key request at CACert.org. Save the resulting certificate as diseserver.crt on your machine.
3. While still at CACert.org, grab the Class 1 root certificate and save it as root.crt.
4. Put the files diseserver.key, diseserver.crt, and root.crt in the PostgreSQL data directory.
ssl = on ssl_cert_file = 'diseserver.crt' # (change requires restart) ssl_key_file = 'diseserver.key' # (change requires restart) ssl_ca_file = 'root.crt' # (change requires restart)
6. Restart the PostgreSQL server. The server will now permit SSL connections, a necessary pre-condition for certificate authentication.
7. We now add 2 lines to pg_hba.conf:
hostnossl all all 0.0.0.0/0 reject hostssl mytable all 0.0.0.0/0 cert map=ssl clientcert=1

The first line causes all non-SSL connections to be rejected. The second allows certificate logins for mytable using the map ssl that is defined in pg_ident.conf:

ssl /^(.*).mydomain\.org$\1 this line extracts the username prefix from CN=username.mydomain.org. 8. Now we need to generate client certificates. PostgreSQL expects these to be in ~/.postgresql (Windows %appdata%\postgresql\): $mkdir ~/.postgresql $cd ~/.postgresql$openssl req -out postgresql.csr -new -newkey rsa:2048 -nodes -keyout postgresql.key

for this key, make CN=username.mydomain.org.

9. Again, paste the resulting postgresql.csr file into CACert.org, saving the certificate as postgresql.crt.
10. Test this:
$psql "sslmode=verify-full host=diseserver.mydomain.org dbname=effect user=username" The server should respond: psql (9.4.6, server 9.4.1) SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off) 11. Next we need to convert our key into pkcs8 format so that Java can read it: $openssl pkcs8 -topk8 -inform PEM -outform DER -in postgresql.key -out postgresql.pk8 -nocrypt
12. Next, ensure that we have the correct version of the JDBC driver (Java-to-database connector). From the Mac command line:
$java -version java version "1.8.0_05" and in Matlab: >> version -java ans = Java 1.7.0_75-b13 with Oracle Corporation Java HotSpot(TM) 64-Bit Server VM mixed mode This shows that although we have Java 8 installed on El Capitan (at the OS level), Matlab uses a private Java 7 version. So we need the correct version of the jdbc on our static java classpath that is used by Matlab: ~/Matlab/postgresql-9.4.1208.jre7.jar 13. The next part is very poorly documented in both the MathWorks and the PostgreSQL documentation, but I found it in Russel Gray’s Basildon Coder blog: We need to use the jdbc postgresql driver to check the client certificate. To do this, we need a custom SSLSocketFactory – LibPQFactory. This will grab our certificate and key from ~/.postgresql and present them to the server. The url is (note the trailing &): jdbc:postgresql://diseserver.mydomain.org/mytable?ssl=true&sslfactory=org.postgresql.ssl.jdbc4.LibPQFactory&sslmode=verify-full& 14. Next we need the username. Rather than hard-coding this in the source code, we get the system username: >> username = java.lang.System.getProperty('user.name'); 15. Bundle this all up in a Matlab function, stripping the trailing CR from the username: function dbtest driver = 'org.postgresql.Driver'; [~,username] = system('whoami'); url = 'jdbc:postgresql://diseserver.mydomain.org/mytable?ssl=true&sslfactory=org.postgresql.ssl.jdbc4.LibPQFactory&sslmode=verify-full&'; myconn = database('mytable', username, '', driver, url); if ~isempty(myconn.Message) fprintf(2,'%s\n', myconn.Message); else fprintf(1, 'Connected!\n'); end end Now we can connect from the Matlab command line or a Matlab program. What if we’re deployed? We also need to add the contents of our .postgresql directory, plus the jdbc jar file to our deployed app: >> mcc -m dbtest.m -a ~/.postgresql -a ~/Matlab/postgresql-9.4.1208.jre7.jar Let’s test the compiled program from the OS command line: $./run_dbtest.sh /Applications/Matlab/Matlab_Runtime/v90 Connected!

Note that the key and certificates are part of the encrypted bundle produced by Matlab’s mcc compiler.

I hope this helps someone!

Yair’s note: the Matlab code above uses Matlab’s Database Toolbox (specifically, the database function) to connect to the database. In future posts I plan to show how we can connect Matlab directly to a database via JDBC. This topic is covered in detail in chapter 2 of my Matlab-Java programming secrets book.

p.s. – this blog celebrates a 7-year anniversary tomorrow: I published my very first post here on March 19, 2009, showing how to change Matlab’s command-window colors (a post that later led to the now-famous cprintf utility). It’s been a long and very interesting ride indeed, but I have no plans to retire anytime soon

]]>
https://undocumentedmatlab.com/blog/secure-ssl-connection-between-matlab-and-postgresql/feed 0