Stock Matlab function – Undocumented Matlab https://undocumentedmatlab.com Charting Matlab's unsupported hidden underbelly Fri, 20 Oct 2017 09:57:44 +0000 en-US hourly 1 https://wordpress.org/?v=4.4.1 Runtime code instrumentationhttps://undocumentedmatlab.com/blog/runtime-code-instrumentation https://undocumentedmatlab.com/blog/runtime-code-instrumentation#comments Thu, 28 Sep 2017 13:36:17 +0000 http://undocumentedmatlab.com/?p=7063
 
Related posts:
  1. Creating a simple UDD class This article explains how to create and test custom UDD packages, classes and objects...
  2. Customizing print setup Matlab figures print-setup can be customized to automatically prepare the figure for printing in a specific configuration...
  3. UDD Events and Listeners UDD event listeners can be used to listen to property value changes and other important events of Matlab objects...
  4. Trapping warnings efficiently warnings can be trapped in a try-catch block just like regular errors. This articles explains how. ...
 
]]>
I regularly follow the MathWorks Pick-of-the-Week (POTW) blog. In a recent post, Jiro Doke highlighted Per Isakson’s tracer4m utility. Per is an accomplished Matlab programmer, who has a solid reputation in the Matlab user community for many years. His utility uses temporary conditional breakpoints to enable users to trace calls to their Matlab functions and class methods. This uses a little-known trick that I wish to highlight in this post.

tracer4m utility uses conditional breakpoints that evaluate but never become live

tracer4m utility uses conditional breakpoints that evaluate but never become live

Matlab breakpoints are documented and supported functionality, and yet their documented use is typically focused at interactive programming in the Matlab editor, or as interactive commands that are entered in the Matlab console using the set of db* functions: dbstop, dbclear, dbstatus, dbstack etc. However, nothing prevents us from using these db* functions directly within our code.

For example, the dbstack function can help us diagnose the calling tree for the current function, in order to do action A if one of the calling ancestors was FunctionX, or to do action B otherwise (for example, to avoid nested recursions).

Similarly, we could add a programmatic call to dbstop in order to stop at a certain code location downstream (for debugging), if a certain condition happens upstream.

Per extended this idea very cleverly in tracer4m: conditional breakpoints evaluate a string in run-time: if the result is true (non-zero) then the code run is stopped at that location, but if it’s false (or zero) then the code run continues normally. To instrument calls to specific functions, Per created a function tracer() that logs the function call (using dbstack) and always returns the value false. He then dynamically created a string that contains a call to this new function and used the dbstop function to create a conditional breakpoint based on this function, something similar to this:

dbstop('in', filename, 'at', location, 'if', 'tracer()');

We can use this same technique for other purposes. For example, if we want to do some action (not necessarily log – perhaps do something else) when a certain code point is reached. The benefit here is that we don’t need to modify the code at all – we’re adding ad-hoc code pieces using the conditional breakpoint mechanism without affecting the source code. This is particularly useful when we do not have access to the source code (such as when it’s compiled or write-protected). All you need to do is to ensure that the instrumentation function always returns false so that the breakpoint does not become live and for code execution to continue normally.

The tracer4m utility is quite sophisticated in the sense that it uses mlint and smart regexp to parse the code and know which functions/methods occur on which line numbers and have which type (more details). In this sense, Per used undocumented functionality. I’m certain that Jiro was not aware of the dependency on undocumented features when he posted about the utility, so please don’t take this to mean that Jiro or MathWorks officially support this or any other undocumented functionality. Undocumented aspects are often needed to achieve top functionality, and I’m happy that the POTW blog highlights utilities based on their importance and merit, even if they do happen to use some undocumented aspect.

tracer4m‘s code also contains references to the undocumented profiler option -history, but this is not in fact used by the code itself, only in comments. I use this feature in my profile_history utility, which displays the function call/timing history in an interactive GUI window. This utility complements tracer4m by providing a lot more information, but this can result in a huge amount of information for large and/or long-running programs. In addition, tracer4m has the benefit of only logging those functions/methods that the user finds useful, rather than all the function call, which enables easier debugging when the relevant code area is known. In short, I wish I had known about tracer4m when I created profile_history. Now that I know about it, maybe I’ll incorporate some of its ideas into profile_history in order to make it more useful. Perhaps another moral of this is that we should actively monitor the POTW blog, because true gems are quite often highlighted there.

Function call timeline profiling
Function call timeline profiling

For anyone who missed the announcement in my previous post, I’m hosting a series of live webinars on advanced Matlab topics in the upcoming 2 weeks – I’ll be happy if you would join.

]]>
https://undocumentedmatlab.com/blog/runtime-code-instrumentation/feed 3
Sending HTML emails from Matlabhttps://undocumentedmatlab.com/blog/sending-html-emails-from-matlab https://undocumentedmatlab.com/blog/sending-html-emails-from-matlab#respond Wed, 02 Aug 2017 21:19:42 +0000 http://undocumentedmatlab.com/?p=6986
 
Related posts:
  1. Uitable sorting Matlab's uitables can be sortable using simple undocumented features...
  2. Uitable customization report Matlab's uitable can be customized in many different ways. A detailed report explains how. ...
  3. Types of undocumented Matlab aspects This article lists the different types of undocumented/unsupported/hidden aspects in Matlab...
  4. Multi-line uitable column headers Matlab uitables can present long column headers in multiple lines, for improved readability. ...
 
]]>
A few months ago I wrote about various tricks for sending email/text messages from Matlab. Unfortunately, Matlab only sends text emails by default and provides no documented way to send HTML-formatted emails. Text-only emails are naturally very bland and all mail clients in the past 2 decades support HTML-formatted emails. Today I will show how we can send such HTML emails from Matlab.

A quick recap: Matlab’s sendmail function uses Java (specifically, the standard javax.mail package) to prepare and send emails. The Java classes are extremely powerful and so there is no wonder that Mathworks chose to use them rather than reinventing the wheel. However, Matlab’s sendmail function only uses part of the functionality exposed by these classes (admittedly, the most important parts that deal with the basic mail-sending mechanism), and does not expose external hooks or input args that would enable the user to take full advantage of the more advanced features, HTML formatting included.

Only two small changes are needed in sendmail.m to support HTML formatting:

  1. HTML formatting required calling the message-object’s setContent() method, rather than setText().
  2. We need to specify 'text/html' as part of the message’s encoding

To implement these features, change the following (lines #119-130 in the original sendmail.m file of R2017a, changed lines highlighted):

% Construct the body of the message and attachments.
body = formatText(theMessage);
if numel(attachments) == 0    if ~isempty(charset)        msg.setText(body, charset);
    else
        msg.setText(body);
    end
else
    % Add body text.
    messageBodyPart = MimeBodyPart;
    if ~isempty(charset)        messageBodyPart.setText(body, charset);
    ...

to this (changed lines highlighted):

% Construct the body of the message and attachments.
body = formatText(theMessage);
isHtml = ~isempty(body) && body(1) == '<';  % msg starting with '<' indicates HTMLif isHtml    if isempty(charset)        charset = 'text/html; charset=utf-8';    else        charset = ['text/html; charset=' charset];    endendif numel(attachments) == 0  && ~isHtml    if isHtml        msg.setContent(body, charset);    elseif ~isempty(charset)        msg.setText(body, charset);
    else
        msg.setText(body);
    end
    else
        % Add body text.
        messageBodyPart = MimeBodyPart;
        if isHtml            messageBodyPart.setContent(body, charset);        elseif ~isempty(charset)            messageBodyPart.setText(body, charset);
        ...

In addition, I also found it useful to remove the hard-coded 75-character line-wrapping in text messages. This can be done by changing the following (line #291 in the original sendmail.m file of R2017a):

maxLineLength = 75;

to this:

maxLineLength = inf;  % or some other large numeric value

Deployment

It’s useful to note two alternatives for making these fixes:

  • Making the changes directly in %matlabroot%/toolbox/matlab/iofun/sendmail.m. You will need administrator rights to edit this file. You will also need to redo the fix whenever you install Matlab, either installation on a different machine, or installing a new Matlab release. In general, I discourage changing Matlab’s internal files because it is simply not very maintainable.
  • Copying %matlabroot%/toolbox/matlab/iofun/sendmail.m into a dedicated wrapper function (e.g., sendEmail.m) that has a similar function signature and exists on the Matlab path. This has the benefit of working on multiple Matlab releases, and being copied along with the rest of our m-files when we install our Matlab program on a different computer. The downside is that our wrapper function will be stuck with the version of sendmail.m that we copied into it, and we’d lose any possible improvements that Mathworks may implement in future Matlab releases.

The basic idea for the second alternative, the sendEmail.m wrapper, is something like this (the top highlighted lines are the additions made to the original sendmail.m, with everything placed in sendEmail.m on the Matlab path):

function sendEmail(to,subject,theMessage,attachments)%SENDEMAIL Send e-mail wrapper (with HTML formatting)   sendmail(to,subject,theMessage,attachments); 
% The rest of this file is copied from %matlabroot%/toolbox/matlab/iofun/sendmail.m (with the modifications mentioned above):
function sendmail(to,subject,theMessage,attachments)
%SENDMAIL Send e-mail.
%   SENDMAIL(TO,SUBJECT,MESSAGE,ATTACHMENTS) sends an e-mail.  TO is either a
%   character vector specifying a single address, or a cell array of character vector
...

We would then call the wrapper function as follows:

sendEmail('abc@gmail.com', 'email subject', 'regular text message');     % will send a regular text message
sendEmail('abc@gmail.com', 'email subject', '<b><font color="blue">HTML-formatted</font> <i>message');  % HTML-formatted message

In this case, the code automatically infers HTML formatting based on whether the first character in the message body is a ‘<‘ character. Instead, we could just as easily have passed an additional input argument (isHtml) to our sendEmail wrapper function.

Hopefully, in some future Matlab release Mathworks will be kind enough to enable sending 21st-century HTML-formatted emails without needing such hacks. Until then, note that sendmail.m relies on standard non-GUI Java networking classes, which are expected to be supported far into the future, well after Java-based GUI may cease to be supported in Matlab. For this reason I believe that while it seems a bit tricky, the changes that I outlined in today’s post actually have a low risk of breaking in a future Matlab release.

Do you have some other advanced email feature that you use in your Matlab program by some crafty customization to sendmail? If so, please share it in a comment below.

]]>
https://undocumentedmatlab.com/blog/sending-html-emails-from-matlab/feed 0
Matlab compilation quirks – take 2https://undocumentedmatlab.com/blog/matlab-compilation-quirks-take-2 https://undocumentedmatlab.com/blog/matlab-compilation-quirks-take-2#respond Wed, 31 May 2017 18:00:42 +0000 http://undocumentedmatlab.com/?p=6919
 
Related posts:
  1. Matlab compiler bug and workaround Both the Matlab compiler and the publish function have errors when parsing block-comments in Matlab m-code. ...
  2. UDD Properties UDD provides a very convenient way to add customizable properties to existing Matlab object handles...
  3. Disabling menu entries in deployed docked figures Matlab's standard menu items can and should be removed from deployed docked figures. This article explains how. ...
  4. Handle Graphics Behavior HG behaviors are an important aspect of Matlab graphics that enable custom control of handle functionality. ...
 
]]>
Once again I would like to welcome guest blogger Hanan Kavitz of Applied Materials. Hanan posted a couple of guest posts here over the past few years, including a post last year about quirks with Matlab-compiled DLLs. Today Hanan will follow up on that post by discussing several additional quirks that they have encountered with Matlab compilations/deployment.

Don’t fix it, if it ain’t broke…

In Applied Materials Israel (PDC) we use Matlab code for both algorithm development and deployment (production). As part of the dev-ops build system, which builds our product software versions, we build Matlab artifacts (binaries) from the Matlab source code.

A typical software version has several hundreds Matlab artifacts that are automatically rebuilt on a daily basis, and we have many such versions – totaling many thousands of compilations each day.

This process takes a long time, so we were looking for a way to make it more efficient.

The idea that we chose to implement sounds simple – take a single binary module in any software version (Ex. foo.exe – Matlab-compiled exe) and check it: if the source code for this module has not changed since the last compilation then simply don’t compile it, just copy it from previous software version repository. Since most of our code doesn’t change daily (some of it hasn’t changed in years), we can skip the compilation time of most binaries and just copy them from some repository of previously compiled binaries.

In a broader look, avoiding lengthy compilations cycles by not compiling unchanged code is a common programming practice, implemented by all modern compilers. For example, the ‘make’ utility uses a ‘makefile’ to check the time stamps of all dependencies of every object file in order to decide which object requires recompilation. In reality, this is not always the best solution as time stamps may be incorrect, but it works well in the vast majority of cases.

Coming back to Matlab, now comes the hard part – how could our build system know that nothing has changed in module X and that something has changed in module Y? How does it even know which source files it needs to ensure didn’t change?

The credit for the idea goes to my manager, Lior Cohen, as follows: You can actually check the dependency of a given binary after compilation. The basis of the solution is that a Matlab executable is in fact a compressed (zip) file. The idea is then to:

  1. Compile the binary once
  2. Unzip the binary and “see” all your dependencies (source files are encrypted and resources are not, but we only need the list of file names – not their content).
  3. Now build a list of all your dependency files and compute the CRC value of each from the source control. Save it for the next time you are required to compile this module.
  4. In the next compilation cycle, find this dependency list, review it, dependency source file at a time and make sure CRC of the dependency hasn’t changed since last time.
  5. If no dependency CRC has changed, then copy the binary from the repository of previous software version, without compiling.
  6. Otherwise, recompile the binary and rebuild the CRC list of all dependencies again, in preparation for the next compilation cycle.

That’s it! That simple? Well… not really – the reality is a bit more complex since there are many other dependencies that need to be checked. Some of them are:

  1. Did the requested Matlab version of the binary change since the last compilation?
  2. Did the compilation instructions themselves (we have a sort of ‘makefile’) change?

Basically, I implemented a policy that if anything changed, or if the dependency check itself failed, then we don’t take any chances and just compile this binary. Keeping in mind that this dependencies check and file copying is much faster than a Matlab compilation, we save a lot of actual compilation time using this method.

Bottom line: Given a software version containing hundreds of compilation instructions to execute and assuming not much has changed in the version (which is often the case), we skip over 90% of compilations altogether and only rebuild what really changed. The result is a version build that takes about half an hour, instead of many hours. Moreover, since the compilation process is working significantly less, we get fewer failures, fewer stuck or crashed mcc processes, and [not less importantly] less maintenance required by me.

Note that in our implementation we rely on the undocumented fact that Matlab binaries are in fact compressed zip archives. If and when a future Matlab release will change the implementation such that the binaries will no longer be zip archives, another way will need to be devised in order to ensure the consistency of the target executable with its dependent source files.

Don’t kill it, if it ain’t bad…

I want to share a very weird issue I investigated over a year ago when using Matlab compiled exe. It started with a user showed me a Matlab compiled exe that didn’t run – I’m not talking about a regular Matlab exception: the process was crashing with an MS Windows popup window popping, stating something very obscure.

It was a very weird behavior that I couldn’t explain – the compiler seemed to work well but the compiled executable process kept crashing. Compiling completely different code showed the same behavior.

This issue has to do with the system compiler configuration that is being used. As you might know, when installing the Matlab compiler, before the first compilation is ever made, the user has to state the C compiler that the Matlab compiler should use in its compilation process. This is done by command ‘mbuild –setup’. This command asks the users to choose the C compiler and saves the configuration (batch file back then, xml in the newer versions of Matlab) in the user’s prefdir folder. At the time we were using Microsoft Visual C++ compiler 9.0 SP1.

The breakthrough in the investigation came when I ran mcc command with –verbose flag, which outputs much more compilation info than I would typically ever want… I discovered that although the target executable file had been created, a post compilation step failed to execute, while issuing a very cryptic error message:

mt.exe : general error c101008d: Failed to write the updated manifest to the resource of file “…”. Access is denied.

cryptic compilation error (click to zoom)

cryptic compilation error (click to zoom)

The failure was in one of the ‘post link’ commands in the configuration batch file – something obscure such as this:

set POSTLINK_CMDS2=mt.exe -outputresource: %MBUILD_OUTPUT_FILE_NAME%;%MANIFEST_RESOURCE% -manifest "%MANIFEST_FILE_NAME%"

This line of code takes an XML manifest file and inserts it into the generated binary file (additional details).

If you open a valid R2010a (and probably other old versions as well) Matlab-generated exe in a text editor you can actually see a small XML code embedded in it, while in a non-functioning exe I could not see this XML code.

So why would this command fail?

It turned out, as funny as it sounds, to be an antivirus issue – our IT department updated its antivirus policies and this ‘post link’ command suddenly became an illegal operation. Once our IT eased the policy, this command worked well again and the compiled executables stopped crashing, to our great joy.

]]>
https://undocumentedmatlab.com/blog/matlab-compilation-quirks-take-2/feed 0
Additional license datahttps://undocumentedmatlab.com/blog/additional-license-data https://undocumentedmatlab.com/blog/additional-license-data#comments Wed, 15 Feb 2017 18:01:55 +0000 http://undocumentedmatlab.com/?p=6852
 
Related posts:
  1. uiundo – Matlab’s undocumented undo/redo manager The built-in uiundo function provides easy yet undocumented access to Matlab's powerful undo/redo functionality. This article explains its usage....
  2. Undocumented Profiler options part 3 An undocumented feature of the Matlab Profiler can report call history timeline - part 3 of series. ...
  3. Undocumented Profiler options part 4 Several undocumented features of the Matlab Profiler can make it much more useful - part 4 of series. ...
  4. Pinning annotations to graphs Annotation object can be programmatically set at, and pinned-to, plot axes data points. ...
 
]]>
Matlab’s license function returns the primary license number/ID used by Matlab, but no information about the various toolboxes that may be installed. The ver function returns a bit more information, listing the version number and installation date of installed toolboxes (even user toolboxes, such as my IB-Matlab toolbox). However, no additional useful information is provided beyond that:

>> license
ans =
123456   % actual number redacted
 
>> ver
----------------------------------------------------------------------------------------------------
MATLAB Version: 9.1.0.441655 (R2016b)
MATLAB License Number: 123456
Operating System: Microsoft Windows 7 Professional  Version 6.1 (Build 7601: Service Pack 1)
Java Version: Java 1.7.0_60-b19 with Oracle Corporation Java HotSpot(TM) 64-Bit Server VM mixed mode
----------------------------------------------------------------------------------------------------
MATLAB                                                Version 9.1         (R2016b)           
Curve Fitting Toolbox                                 Version 3.5.4       (R2016b)           
Database Toolbox                                      Version 7.0         (R2016b)           
Datafeed Toolbox                                      Version 5.4         (R2016b)           
Financial Instruments Toolbox                         Version 2.4         (R2016b)           
Financial Toolbox                                     Version 5.8         (R2016b)           
GUI Layout Toolbox                                    Version 2.2.1       (R2015b)           
Global Optimization Toolbox                           Version 3.4.1       (R2016b)           
IB-Matlab - Matlab connector to InteractiveBrokers    Version 1.89        Expires: 1-Apr-2018
Image Processing Toolbox                              Version 9.5         (R2016b)           
MATLAB Coder                                          Version 3.2         (R2016b)           
MATLAB Report Generator                               Version 5.1         (R2016b)           
Optimization Toolbox                                  Version 7.5         (R2016b)           
Parallel Computing Toolbox                            Version 6.9         (R2016b)           
Statistical Graphics Toolbox                          Version 1.2                            
Statistics and Machine Learning Toolbox               Version 11.0        (R2016b)           
 
>> v = ver
v = 
  1×16 struct array with fields:
    Name
    Version
    Release
    Date
 
>> v(1)
ans = 
  struct with fields:
 
       Name: 'Curve Fitting Toolbox'
    Version: '3.5.4'
    Release: '(R2016b)'
       Date: '25-Aug-2016'
 
>> v(8)
ans = 
  struct with fields:
 
       Name: 'IB-Matlab - Matlab connector to InteractiveBrokers'
    Version: '1.89'
    Release: 'Expires: 1-Apr-2018'
       Date: '02-Feb-2017'

It is sometimes useful to know which license number “owns” which product/toolbox, and the expiration date is associated with each of them. Unfortunately, there is no documented way to retrieve this information in Matlab – the only documented way is to go to your account section on the MathWorks website and check there.

Luckily, there is a simpler way that can be used to retrieve additional information, from right inside Matlab, using matlab.internal.licensing.getFeatureInfo:

>> all_data = matlab.internal.licensing.getFeatureInfo
all_data = 
  23×1 struct array with fields:
    feature
    expdate
    keys
    license_number
    entitlement_id
 
>> all_data(20)
ans = 
  struct with fields:
 
           feature: 'optimization_toolbox'
           expdate: '31-mar-2018'
              keys: 0
    license_number: '123456'
    entitlement_id: '1409891'
 
>> all_data(21)
ans = 
  struct with fields:
 
           feature: 'optimization_toolbox'
           expdate: '07-mar-2017'
              keys: 0
    license_number: 'DEMO'
    entitlement_id: '3749959'

As can be seen in this example, I have the Optimization toolbox licensed under my main Matlab license (123456 [actual number redacted]) until 31-mar-2018, and also licensed under a trial (DEMO) license that expires in 3 weeks. As long as a toolbox has any future expiration date, it will continue to function, so in this case I’m covered until March 2018.

We can also request information about a specific toolbox (“feature”):

>> data = matlab.internal.licensing.getFeatureInfo('matlab')
data = 
  3×1 struct array with fields:
    feature
    expdate
    keys
    license_number
    entitlement_id
 
>> data(1)
data = 
  struct with fields:
 
           feature: 'matlab'
           expdate: '31-mar-2018'
              keys: 0
    license_number: '123456'
    entitlement_id: '1409891'

The drawback of this functionality is that it only provides information about MathWorks’ toolbox, not any user-provided toolboxes (such as my IB-Matlab connector, or MathWorks’ own GUI Layout toolbox). Also, some of the toolbox names may be difficult to understand (“gads_toolbox” apparently stands for the Global Optimization Toolbox, for example):

>> {all_data.feature}
ans =
  1×23 cell array
  Columns 1 through 4
    'curve_fitting_toolbox'    'database_toolbox'    'datafeed_toolbox'    'distrib_computing_toolbox'
  Columns 5 through 8
    'distrib_computing_toolbox'    'excel_link'    'fin_instruments_toolbox'    'financial_toolbox'
  Columns 9 through 15
    'gads_toolbox'    'gads_toolbox'    'image_toolbox'    'image_toolbox'    'matlab'    'matlab'    'matlab'
  Columns 16 through 20
    'matlab_coder'    'matlab_coder'    'matlab_report_gen'    'matlab_report_gen'    'optimization_toolbox'
  Columns 21 through 23
    'optimization_toolbox'    'optimization_toolbox'    'statistics_toolbox'

A related undocumented builtin function is matlab.internal.licensing.getLicInfo:

% Information on a single toolbox/product:
>> matlab.internal.licensing.getLicInfo('matlab')
ans = 
  struct with fields:
 
     license_number: {'123456'  'Prerelease'  'T3749959'}
    expiration_date: {'31-mar-2018'  '30-sep-2016'  '07-mar-2017'}
 
% Information on multiple toolboxes/products:
>> matlab.internal.licensing.getLicInfo({'matlab', 'image_toolbox'})  % cell array of toolbox/feature names
ans = 
  1×2 struct array with fields:
    license_number
    expiration_date
 
% The full case-insensitive names of the toolboxes can also be used:
>> matlab.internal.licensing.getLicInfo({'Matlab', 'Image Processing toolbox'})
ans = 
  1×2 struct array with fields:
    license_number
    expiration_date
 
% And here's how to get the full list (MathWorks products only):
>> v=ver; data=matlab.internal.licensing.getLicInfo({v.Name})
data = 
  1×16 struct array with fields:
    license_number
    expiration_date

I have [still] not found any way to associate a user toolbox/product (such as my IB-Matlab) in a way that will report it in a unified manner with the MathWorks products. If anyone finds a way to do this, please do let me know.

p.s. – don’t even think of asking questions or posting comments on this website related to illegal uses or hacks of the Matlab license…

]]>
https://undocumentedmatlab.com/blog/additional-license-data/feed 5
Parsing XML stringshttps://undocumentedmatlab.com/blog/parsing-xml-strings https://undocumentedmatlab.com/blog/parsing-xml-strings#comments Wed, 01 Feb 2017 09:52:45 +0000 http://undocumentedmatlab.com/?p=6838
 
Related posts:
  1. Undocumented XML functionality Matlab's built-in XML-processing functions have several undocumented features that can be used by Java-savvy users...
  2. Matlab-Java memory leaks, performance Internal fields of Java objects may leak memory - this article explains how to avoid this without sacrificing performance. ...
  3. Types of undocumented Matlab aspects This article lists the different types of undocumented/unsupported/hidden aspects in Matlab...
  4. Pause for the better Java's thread sleep() function is much more accurate than Matlab's pause() function. ...
 
]]>
I have recently consulted in a project where data was provided in XML strings and needed to be parsed in Matlab memory in an efficient manner (in other words, as quickly as possible). Now granted, XML is rather inefficient in storing data (JSON would be much better for this, for example). But I had to work with the given situation, and that required processing the XML.

I basically had two main alternatives:

  • I could either create a dedicated string-parsing function that searches for a particular pattern within the XML string, or
  • I could use a standard XML-parsing library to create the XML model and then parse its nodes

The first alternative is quite error-prone, since it relies on the exact format of the data in the XML. Since the same data can be represented in multiple equivalent XML ways, making the string-parsing function robust as well as efficient would be challenging. I was lazy expedient, so I chose the second alternative.

Unfortunately, Matlab’s xmlread function only accepts input filenames (of *.xml files), it cannot directly parse XML strings. Yummy!

The obvious and simple solution is to simply write the XML string into a temporary *.xml file, read it with xmlread, and then delete the temp file:

% Store the XML data in a temp *.xml file
filename = [tempname '.xml'];
fid = fopen(filename,'Wt');
fwrite(fid,xmlString);
fclose(fid);
 
% Read the file into an XML model object
xmlTreeObject = xmlread(filename);
 
% Delete the temp file
delete(filename);
 
% Parse the XML model object
...

This works well and we could move on with our short lives. But cases such as this, where a built-in function seems to have a silly limitation, really fire up the investigative reporter in me. I decided to drill into xmlread to discover why it couldn’t parse XML strings directly in memory, without requiring costly file I/O. It turns out that xmlread accepts not just file names as input, but also Java object references (specifically, java.io.File, java.io.InputStream or org.xml.sax.InputSource). In fact, there are quite a few other inputs that we could use, to specify a validation parser etc. – I wrote about this briefly back in 2009 (along with other similar semi-documented input altermatives in xmlwrite and xslt).

In our case, we could simply send xmlread as input a java.io.StringBufferInputStream(xmlString) object (which is an instance of java.io.InputStream) or org.xml.sax.InputSource(java.io.StringReader(xmlString)):

% Read the xml string directly into an XML model object
inputObject = java.io.StringBufferInputStream(xmlString);                % alternative #1
inputObject = org.xml.sax.InputSource(java.io.StringReader(xmlString));  % alternative #2
 
xmlTreeObject = xmlread(inputObject);
 
% Parse the XML model object
...

If we don’t want to depend on undocumented functionality (which might break in some future release, although it has remained unchanged for at least the past decade), and in order to improve performance even further by passing xmlread‘s internal validity checks and processing, we can use xmlread‘s core functionality to parse our XML string directly. We can add a fallback to the standard (fully-documented) functionality, just in case something goes wrong (which is good practice whenever using any undocumented functionality):

try
    % The following avoids the need for file I/O:
    inputObject = java.io.StringBufferInputStream(xmlString);  % or: org.xml.sax.InputSource(java.io.StringReader(xmlString))
    try
        % Parse the input data directly using xmlread's core functionality
        parserFactory = javaMethod('newInstance','javax.xml.parsers.DocumentBuilderFactory');
        p = javaMethod('newDocumentBuilder',parserFactory);
        xmlTreeObject = p.parse(inputObject);
    catch
        % Use xmlread's semi-documented inputObject input feature
        xmlTreeObject = xmlread(inputObject);
    end
catch
    % Fallback to standard xmlread usage, using a temporary XML file:
 
    % Store the XML data in a temp *.xml file
    filename = [tempname '.xml'];
    fid = fopen(filename,'Wt');
    fwrite(fid,xmlString);
    fclose(fid);
 
    % Read the file into an XML model object
    xmlTreeObject = xmlread(filename);
 
    % Delete the temp file
    delete(filename);
end
 
% Parse the XML model object
...
]]>
https://undocumentedmatlab.com/blog/parsing-xml-strings/feed 4
Quirks with parfor vs. forhttps://undocumentedmatlab.com/blog/quirks-with-parfor-vs-for https://undocumentedmatlab.com/blog/quirks-with-parfor-vs-for#comments Thu, 05 Jan 2017 17:15:48 +0000 http://undocumentedmatlab.com/?p=6821
 
Related posts:
  1. Matlab mex in-place editing Editing Matlab arrays in-place can be an important technique for optimizing calculations. This article shows how to do it using Mex. ...
  2. Preallocation performance Preallocation is a standard Matlab speedup technique. Still, it has several undocumented aspects. ...
  3. Array resizing performance Several alternatives are explored for dynamic array growth performance in Matlab loops. ...
  4. Matlab’s internal memory representation Matlab's internal memory structure is explored and discussed. ...
 
]]>
A few months ago, I discussed several tips regarding Matlab’s parfor command, which is used by the Parallel Computing Toolbox (PCT) for parallelizing loops. Today I wish to extend that post with some unexplained oddities when using parfor, compared to a standard for loop.

Data serialization quirks

Dimitri Shvorob may not appear at first glance to be a prolific contributor on Matlab Central, but from the little he has posted over the years I regard him to be a Matlab power-user. So when Dimitri reports something, I take it seriously. Such was the case several months ago, when he contacted me regarding very odd behavior that he saw in his code: the for loop worked well, but the parfor version returned different (incorrect) results. Eventually, Dimitry traced the problem to something originally reported by Dan Austin on his Fluffy Nuke It blog.

The core issue is that if we have a class object that is used within a for loop, Matlab can access the object directly in memory. But with a parfor loop, the object needs to be serialized in order to be sent over to the parallel workers, and deserialized within each worker. If this serialization/deserialization process involves internal class methods, the workers might see a different version of the class object than the one seen in the serial for loop. This could happen, for example, if the serialization/deserialization method croaks on an error, or depends on some dynamic (or random) conditions to create data.

In other words, when we use data objects in a parfor loop, the data object is not necessarily sent “as-is”: additional processing may be involved under the hood that modify the data in a way that may be invisible to the user (or the loop code), resulting in different processing results of the parallel (parfor) vs. serial (for) loops.

For additional aspects of Matlab serialization/deserialization, see my article from 2 years ago (and its interesting feedback comments).

Data precision quirks

The following section was contributed by guest blogger Lior Perlmuter-Shoshany, head algorithmician at a private equity fund.

In my work, I had to work with matrixes in the order of 109 cells. To reduce the memory footprint (and hopefully also improve performance), I decided to work with data of type single instead of Matlab’s default double. Furthermore, in order to speed up the calculation I use parfor rather than for in the main calculation. In the end of the run I am running a mini for-loop to see the best results.

What I discovered to my surprise is that the results from the parfor and for loop variants is not the same!

The following simplified code snippet illustrate the problem by calculating a simple standard-deviation (std) over the same data, in both single– and double-precision. Note that the loops are ran with only a single iteration, to illustrate the fact that the problem is with the parallelization mechanism (probably the serialization/deserialization parts once again), not with the distribution of iterations among the workers.

clear
rng('shuffle','twister');
 
% Prepare the data in both double and single precision
arr_double = rand(1,100000000);
arr_single = single(arr_double);
 
% No loop - direct computation
std_single0 = std(arr_single);
std_double0 = std(arr_double);
 
% Loop #1 - serial for loop
std_single = 0;
std_double = 0;
for i=1
    std_single(i) = std(arr_single);
    std_double(i) = std(arr_double);
end
 
% Loop #2 - parallel parfor loop
par_std_single = 0;
par_std_double = 0;
parfor i=1
    par_std_single(i) = std(arr_single);
    par_std_double(i) = std(arr_double);
end
 
% Compare results of for loop vs. non-looped computation
isForSingleOk = isequal(std_single, std_single0)
isForDoubleOk = isequal(std_double, std_double0)
 
% Compare results of single-precision data (for vs. parfor)
isParforSingleOk = isequal(std_single, par_std_single)
parforSingleAccuracy = std_single / par_std_single
 
% Compare results of double-precision data (for vs. parfor)
isParforDoubleOk = isequal(std_double, par_std_double)
parforDoubleAccuracy = std_double / par_std_double

Output example :

isForSingleOk = 
    1                   % <= true (of course!)
isForDoubleOk =
    1                   % <= true (of course!)
 
isParforSingleOk =
    0                   % <= false (odd!)
parforSingleAccuracy =
    0.73895227413361    % <= single-precision results are radically different in parfor vs. for
 
isParforDoubleOk =
    0                   % <= false (odd!)
parforDoubleAccuracy =
    1.00000000000021    % <= double-precision results are almost [but not exactly] the same in parfor vs. for

From my testing, the larger the data array, the bigger the difference is between the results of single-precision data when running in for vs. parfor.

In other words, my experience has been that if you have a huge data matrix, it’s better to parallelize it in double-precision if you wish to get [nearly] accurate results. But even so, I find it deeply disconcerting that the results are not exactly identical (at least on R2015a-R2016b on which I tested) even for the native double-precision .

Hmmm… bug?

Upcoming travels – Zürich & Geneva

I will shortly be traveling to clients in Zürich and Geneva, Switzerland. If you are in the area and wish to meet me to discuss how I could bring value to your work with some advanced Matlab consulting or training, then please email me (altmany at gmail):

  • Zürich: January 15-17
  • Geneva: January 18-21

Happy new year everybody!

]]>
https://undocumentedmatlab.com/blog/quirks-with-parfor-vs-for/feed 7
Sending email/text messages from Matlabhttps://undocumentedmatlab.com/blog/sending-email-text-messages-from-matlab https://undocumentedmatlab.com/blog/sending-email-text-messages-from-matlab#comments Wed, 07 Dec 2016 21:24:03 +0000 http://undocumentedmatlab.com/?p=6765
 
Related posts:
  1. Types of undocumented Matlab aspects This article lists the different types of undocumented/unsupported/hidden aspects in Matlab...
  2. Legend ‘-DynamicLegend’ semi-documented feature The built-in Matlab legend function has a very useful semi-documented feature for automatic dynamic update, which is explained here....
  3. Undocumented XML functionality Matlab's built-in XML-processing functions have several undocumented features that can be used by Java-savvy users...
  4. Inactive Control Tooltips & Event Chaining Inactive Matlab uicontrols cannot normally display their tooltips. This article shows how to do this with a combination of undocumented Matlab and Java hacks....
 
]]>
In this day and age, applications are expected to communicate with users by sending email/text messages to alert them about applicative events (“IBM stock purchased @$99.99” or “House is on fire!”). Matlab has included the sendmail function to handle this for many years. Unfortunately, sendmail requires some tweaking to be useful on all but the most basic/insecure mail servers. Today’s post will hopefully fill the missing gaps.

None of the information I’ll present today is really new – it was all there already if you just knew what to search for online. But hopefully today’s post will concentrate all these loose ends in a single place, so it may have some value:

Using a secure mail server

All modern mail servers use end-to-end TLS/SSL encryption. The sendmail function needs extra configuration to handle such connections, since it is configured for a non-encrypted connection by default. Here’s the code that does this for gmail, using SMTP server smtp.gmail.com and default port #465 (for other SMTP servers, see here):

setpref('Internet', 'SMTP_Server',   'smtp.gmail.com');
setpref('Internet', 'SMTP_Username', username);
setpref('Internet', 'SMTP_Password', password);
 
props = java.lang.System.getProperties;
props.setProperty('mail.smtp.auth',                'true');  % Note: 'true' as a string, not a logical value!
props.setProperty('mail.smtp.starttls.enable',     'true');  % Note: 'true' as a string, not a logical value!
props.setProperty('mail.smtp.socketFactory.port',  '465');   % Note: '465'  as a string, not a numeric value!
props.setProperty('mail.smtp.socketFactory.class', 'javax.net.ssl.SSLSocketFactory');
 
sendmail(recipient, title, body, attachments);  % e.g., sendmail('recipient@gmail.com', 'Hello world', 'What a nice day!', 'C:\images\sun.jpg')

All this is not enough to enable Matlab to connect to gmail’s SMTP servers. In addition, we need to set the Google account to allow access from “less secure apps” (details, direct link). Without this, Google will not allow Matlab to relay emails. Other mail servers may require similar server-side account configurations to enable Matlab’s access.

Note: This code snippet uses a bit of Java as you can see. Under the hood, all networking code in Matlab relies on Java, and sendmail is no exception. For some reason that I don’t fully understand, MathWorks chose to label the feature of using sendmail with secure mail servers as a feature that relies on “undocumented commands” and is therefore not listed in sendmail‘s documentation. Considering the fact that all modern mail servers are secure, this seems to make sendmail rather useless without the undocumented extension. I assume that TMW are well aware of this, which is the reason they posted a partial documentation in the form of an official tech-support answer. I hope that one day MathWorks will incorporate it into sendmail as optional input args, so that using sendmail with secure servers would become fully documented and officially supported.

Emailing multiple recipients

To specify multiple email recipients, it is not enough to set sendmail‘s recipient input arg to a string with , or ; delimiters. Instead, we need to provide a cell array of individual recipient strings. For example:

sendmail({'recipient1@gmail.com','recipient2@gmail.com'}, 'Hello world', 'What a nice day!')

Note: this feature is actually fully documented in sendmail‘s doc-page, but for some reason I see that some users are not aware of it (to which it might be said: RTFM!).

Sending text messages

With modern smartphones, text (SMS) messages have become rather outdated, as most users get push notifications of incoming emails. Still, for some users text messages may still be a useful. To send such messages, all we need is to determine our mobile carrier’s email gateway for SMS messages, and send a simple text message to that email address. For example, to send a text message to T-Mobile number 123-456-7890 in the US, simply email the message to 1234567890@tmomail.net (details).

Ke Feng posted a nice Matlab File Exchange utility that wraps this messaging for a wide variety of US carriers.

User configuration panel

Many GUI programs contain configuration panels/tabs/windows. Enabling the user to set up their own email provider is a typical use-case for such a configuration. Naturally, you’d want your config panel not to display plain-text password, nor non-integer port numbers. You’d also want the user to be able to test the email connection.

Here’s a sample implementation for such a panel that I implemented for a recent project – I plan to discuss the implementation details of the password and port (spinner) controls in my next post, so stay tuned:

User configuration of emails in Matlab GUI (click to zoom-in)
User configuration of emails in Matlab GUI (click to zoom-in)

]]>
https://undocumentedmatlab.com/blog/sending-email-text-messages-from-matlab/feed 5
Afterthoughts on implicit expansionhttps://undocumentedmatlab.com/blog/afterthoughts-on-implicit-expansion https://undocumentedmatlab.com/blog/afterthoughts-on-implicit-expansion#comments Wed, 30 Nov 2016 20:28:44 +0000 http://undocumentedmatlab.com/?p=6750
 
Related posts:
  1. tic / toc – undocumented option Matlab's built-in tic/toc functions have an undocumented option enabling multiple nested clockings...
  2. Plot LimInclude properties The plot objects' XLimInclude, YLimInclude, ZLimInclude, ALimInclude and CLimInclude properties are an important feature, that has both functional and performance implications....
  3. Matrix processing performance Matrix operations performance is affected by internal subscriptions in a counter-intuitive way....
  4. Performance: accessing handle properties Handle object property access (get/set) performance can be significantly improved using dot-notation. ...
 
]]>
Matlab release R2016b introduced implicit arithmetic expansion, which is a great and long-awaited natural expansion of Matlab’s arithmetic syntax (if you are still unaware of this or what it means, now would be a good time to read about it). This is a well-documented new feature. The reason for today’s post is that this new feature contains an undocumented aspect that should very well have been documented and even highlighted.

The undocumented aspect that I’m referring to is the fact that code that until R2016a produced an error, in R2016b produces a valid result:

% R2016a
>> [1:5] + [1:3]'
Error using  + 
Matrix dimensions must agree.
 
% R2016b
>> [1:5] + [1:3]'
ans =
     2     3     4     5     6
     3     4     5     6     7
     4     5     6     7     8

This incompatibility is indeed documented, but not where it matters most (read on).

I first discovered this feature by chance when trying to track down a very strange phenomenon with client code that produced different numeric results on R2015b and earlier, compared to R2016a Pre-release. After some debugging the problem was traced to a code snippet in the client’s code that looked something like this (simplified):

% Ensure compatible input data
try
    dataA + dataB;  % this will (?) error if dataA, dataB are incompatible
catch
    dataB = dataB';
end

The code snippet relied on the fact that incompatible data (row vs. col) would error when combined, as it did up to R2015b. But in R2016a Pre-release it just gave a valid numeric matrix, which caused numerically incorrect results downstream in the code. The program never crashed, so everything appeared to be in order, it just gave different numeric results. I looked at the release notes and none of the mentioned release incompatibilities appeared relevant. It took me quite some time, using side-by-side step-by-step debugging on two separate instances of Matlab (R2015b and R2016aPR) to trace the problem to this new feature.

This implicit expansion feature was removed from the official R2016a release for performance reasons. This was apparently fixed in time for R2016b’s release.

I’m totally in favor of this great new feature, don’t get me wrong. I’ve been an ardent user of bsxfun for many years and (unlike many) have even grown fond of it, but I still find the new feature to be better. I use it wherever there is no significant performance penalty, a need to support older Matlab releases, or a possibility of incorrect results due to dimensional mismatch.

So what’s my point?

What I am concerned about is that I have not seen the new feature highlighted as a potential backward compatibility issue in the documentation or the release notes. Issues of far lesser importance are clearly marked for their backward incompatibility in the release notes, but not this important major change. A simple marking of the new feature with the warning icon () and in the “Functionality being removed or changed” section would have saved my client and me a lot of time and frustration.

MathWorks are definitely aware of the potential problems that the new feature might cause in rare use cases such as this. As Steve Eddins recently noted, there were plenty of internal discussions about this very thing. MathWorks were careful to ensure that the feature’s benefits far outweigh its risks (and I concur). But this also highlights the fact that MathWorks were fully aware that in some rare cases it might indeed break existing code. For those cases, I believe that they should have clearly marked the incompatibility implications in the release notes and elsewhere.

I have several clients who scour Matlab’s release notes before each release, trying to determine the operational risk of a Matlab upgrade. Having a program that returns different results in R2016b compared to R2016a, without being aware of this risk, is simply unacceptable to them, and leaves users with a disinclination to upgrade Matlab, to MathWorks’ detriment.

MathWorks in general are taking a very serious methodical approach to compatibility issues, and are clearly investing a lot of energy in this (a recent example). It’s too bad that sometimes this chain is broken. I find it a pity, and think that this can still be corrected in the online doc pages. If and when this is fixed, I’ll be happy to post an addendum here.

In my humble opinion from the backbenches, increasing the transparency on compatibility issues and open bugs will increase user confidence and result in greater adoption and upgrades of Matlab. Just my 2 cents…

Addendum December 27, 2016:

Today MathWorks added the following compatibility warning to the release notes (R2016b, Mathematics section, first item) – thanks for listening MathWorks :-)

MathWorks compatibility warning

]]>
https://undocumentedmatlab.com/blog/afterthoughts-on-implicit-expansion/feed 12
Customizing axes part 5 – origin crossover and labelshttps://undocumentedmatlab.com/blog/customizing-axes-part-5-origin-crossover-and-labels https://undocumentedmatlab.com/blog/customizing-axes-part-5-origin-crossover-and-labels#comments Wed, 27 Jul 2016 17:00:02 +0000 http://undocumentedmatlab.com/?p=6564
 
Related posts:
  1. Customizing axes rulers HG2 axes can be customized in numerous useful ways. This article explains how to customize the rulers. ...
  2. Customizing axes part 2 Matlab HG2 axes can be customized in many different ways. This article explains some of the undocumented aspects. ...
  3. Undocumented scatter plot jitter Matlab's scatter plot can automatically jitter data to enable better visualization of distribution density. ...
  4. Performance: accessing handle properties Handle object property access (get/set) performance can be significantly improved using dot-notation. ...
 
]]>
When HG2 graphics was finally released in R2014b, I posted a series of articles about various undocumented ways by which we can customize Matlab’s new graphic axes: rulers (axles), baseline, box-frame, grid, back-drop, and other aspects. Today I extend this series by showing how we can customize the axes rulers’ crossover location.

Non-default axes crossover location

Non-default axes crossover location


The documented/supported stuff

Until R2015b, we could only specify the axes’ YAxisLocation as 'left' (default) or 'right', and XAxisLocation as 'bottom' (default) or 'top'. For example:

x = -2*pi : .01 : 2*pi;
plot(x, sin(x));
hAxis = gca;
hAxis.YAxisLocation = 'left';    % 'left' (default) or 'right'
hAxis.XAxisLocation = 'bottom';  % 'bottom' (default) or 'top'

Default axis locations: axes crossover is non-fixed

Default axis locations: axes crossover is non-fixed

The crossover location is non-fixed in the sense that if we zoom or pan the plot, the axes crossover will remain at the bottom-left corner, which changes its coordinates depending on the X and Y axes limits.

Since R2016a, we can also specify 'origin' for either of these properties, such that the X and/or Y axes pass through the chart origin (0,0) location. For example, move the YAxisLocation to the origin:

hAxis.YAxisLocation = 'origin';

Y-axis location at origin: axes crossover at 0 (fixed), -1 (non-fixed)

Y-axis location at origin: axes crossover at 0 (fixed), -1 (non-fixed)

And similarly also for XAxisLocation:

hAxis.XAxisLocation = 'origin';

X and Y-axis location at origin: axes crossover fixed at (0,0)

X and Y-axis location at origin: axes crossover fixed at (0,0)

The axes crossover location is now fixed at the origin (0,0), so as we move or pan the plot, the crossover location changes its position in the chart area, without changing its coordinates. This functionality has existed in other graphic packages (outside Matlab) for a long time and until now required quite a bit of coding to emulate in Matlab, so I’m glad that we now have it in Matlab by simply updating a single property value. MathWorks did a very nice job here of dynamically updating the axles, ticks and labels as we pan (drag) the plot towards the edges – try it out!

The undocumented juicy stuff

So far for the documented stuff. The undocumented aspect is that we are not limited to using the (0,0) origin point as the fixed axes crossover location. We can use any x,y crossover location, using the FirstCrossoverValue property of the axes’ hidden XRuler and YRuler properties. In fact, we could do this since R2014b, when the new HG2 graphics engine was released, not just starting in R2016a!

% Set a fixed crossover location of (pi/2,-0.4)
hAxis.YRuler.FirstCrossoverValue = pi/2;
hAxis.XRuler.FirstCrossoverValue = -0.4;

Custom fixed axes crossover location at (π/2,-0.4)

Custom fixed axes crossover location at (π/2,-0.4)

For some reason (bug?), setting XAxisLocation/YAxisLocation to ‘origin’ has no visible effect in 3D plots, nor is there any corresponding ZAxisLocation property. Luckily, we can set the axes crossover location(s) in 3D plots using FirstCrossoverValue just as easily as for 2D plots. The rulers also have a SecondCrossoverValue property (default = -inf) that controls the Z-axis crossover, as Yaroslav pointed out in a comment below. For example:

N = 49;
x = linspace(-10,10,N);
M = peaks(N);
mesh(x,x,M);

Default crossover locations at (-10,±10,-10)

Default crossover locations at (-10,±10,-10)

hAxis.XRuler.FirstCrossoverValue  = 0; % X crossover with Y axis
hAxis.YRuler.FirstCrossoverValue  = 0; % Y crossover with X axis
hAxis.ZRuler.FirstCrossoverValue  = 0; % Z crossover with X axis
hAxis.ZRuler.SecondCrossoverValue = 0; % Z crossover with Y axis

Custom fixed axes crossover location at (0,0,-10)

Custom fixed axes crossover location at (0,0,-10)

hAxis.XRuler.SecondCrossoverValue = 0; % X crossover with Z axis
hAxis.YRuler.SecondCrossoverValue = 0; % Y crossover with Z axis

Custom fixed axes crossover location at (0,0,0)

Custom fixed axes crossover location at (0,0,0)

Labels

Users will encounter the following unexpected behavior (bug?) when using either the documented *AxisLocation or the undocumented FirstCrossoverValue properties: when setting an x-label (using the xlabel function, or the internal axes properties), the label moves from the center of the axes (as happens when XAxisLocation=’top’ or ‘bottom’) to the right side of the axes, where the secondary label (e.g., exponent) usually appears, whereas the secondary label is moved to the left side of the axis:

Unexpected label positions

Unexpected label positions

In such cases, we would expect the labels locations to be reversed, with the main label on the left and the secondary label in its customary location on the right. The exact same situation occurs with the Y labels, where the main label unexpectedly appears at the top and the secondary at the bottom. Hopefully MathWorks will fix this in the next release (it is probably too late to make it into R2016b, but hopefully R2017a). Until then, we can simply switch the strings of the main and secondary label to make them appear at the expected locations:

% Switch the Y-axes labels:
ylabel(hAxis, '\times10^{3}');  % display secondary ylabel (x10^3) at top
set(hAxis.YRuler.SecondaryLabel, 'Visible','on', 'String','main y-label');  % main label at bottom
 
% Switch the X-axes labels:
xlabel(hAxis, '2^{nd} label');  % display secondary xlabel at right
set(hAxis.XRuler.SecondaryLabel, 'Visible','on', 'String','xlabel');  % main label at left

As can be seen from the screenshot, there’s an additional nuisance: the main label appears a bit larger than the axes font size (the secondary label uses the correct font size). This is because by default Matlab uses a 110% font-size for the main axes label, ostensibly to make them stand out. We can modify this default factor using the rulers’ hidden LabelFontSizeMultiplier property (default=1.1). For example:

hAxis.YRuler.LabelFontSizeMultiplier = 1;   % use 100% font-size (same as tick labels)
hAxis.XRuler.LabelFontSizeMultiplier = 0.8; % use 80% (smaller than standard) font-size

Note: I described the ruler objects in my first article of the axes series. Feel free to read it for more ideas on customizing the axes rulers.

]]>
https://undocumentedmatlab.com/blog/customizing-axes-part-5-origin-crossover-and-labels/feed 5
rmfield performancehttps://undocumentedmatlab.com/blog/rmfield-performance https://undocumentedmatlab.com/blog/rmfield-performance#comments Wed, 25 May 2016 07:00:48 +0000 http://undocumentedmatlab.com/?p=6427
 
Related posts:
  1. tic / toc – undocumented option Matlab's built-in tic/toc functions have an undocumented option enabling multiple nested clockings...
  2. Solving a MATLAB bug by subclassing Matlab's Image Processing Toolbox's impoint function contains an annoying bug that can be fixed using some undocumented properties....
  3. Plot LimInclude properties The plot objects' XLimInclude, YLimInclude, ZLimInclude, ALimInclude and CLimInclude properties are an important feature, that has both functional and performance implications....
  4. Class object creation performance Performance aspects of Matlab class object creation are discussed, with specific suggestions. ...
 
]]>
Once again I would like to introduce guest blogger Hanan Kavitz of Applied Materials. Several months ago Hanan discussed some quirks with compiled Matlab DLLs. Today Hanan will discuss how they overcame a performance bottleneck with Matlab’s builtin rmfield function, exemplifying the general idea that we can sometimes improve performance by profiling the core functionality that causes a performance hotspot and optimizing it, even when it is part of a builtin Matlab function. For additional ideas of improving Matlab peformance, search this blog for “Performance” articles, and/or get the book “Accelerating MATLAB Performance“.

Accelerating MATLAB Performance
I’ve been using Matlab for many years now and from time to time I need to profile low-throughput code. When I profile this code sometimes I realize that a computational ‘bottleneck’ is due to a builtin Matlab function (part of the core language). I can often find ways to accelerate such builtin functions and get significant speedup in my code.

I recently found Matlab’s builtin rmfield function being too slow for my needs. It works great when one needs to remove a few fields from a small structure, but in our case we needed to remove thousands of fields from a structure containing about 5000 fields – and this is executed in a function that is called many times inside an external loop. The program was significantly sluggish.

It started when a co-worker asked me to look at a code that looked just slightly more intelligent than this:

for i = 1:5000
    myStruct = rmfield(myStruct,fieldNames{i});
end

Running this code within a tic/toc pair yielded the following results:

>> tic; myFunc(); t1 = toc
t1 =
      25.7713

In my opinion 25.77 secs for such a simple functionality seems like an eternity…

The obvious thing was to change the code to the documented faster (vectorized) version:

>> tic; myStruct = rmfield(myStruct,fieldNames); t2 = toc
t2 =
      0.6097

This is obviously much better but since rmfield is called many times in my application, I needed something even better. So I profiled rmfield and was not happy with the result.

The original code of rmfield (%matlabroot%/toolbox/matlab/datatypes/rmfield.m) looks something like this (I deleted some non-essential code for brevity):

function t = rmfield(s,field)
 
% get fieldnames of struct
f = fieldnames(s);
 
% Determine which fieldnames to delete.
idxremove = [];
for i=1:length(field)
   j = find(strcmp(field{i},f) == true);   idxremove = [idxremove;j];
end
 
% set indices of fields to keep
idxkeep = 1:length(f);
idxkeep(idxremove) = [];
 
% remove the specified fieldnames from the list of fieldnames.
f(idxremove,:) = [];
 
% convert struct to cell array
c = struct2cell(s);
 
% find size of cell array
sizeofarray = size(c);
newsizeofarray = sizeofarray;
 
% adjust size for fields to be removed
newsizeofarray(1) = sizeofarray(1) - length(idxremove);
 
% rebuild struct
t = cell2struct(reshape(c(idxkeep,:),newsizeofarray),f);

When I profiled the code, the highlighted row was the bottleneck I was looking for.

First, I noticed the string comparison equals to true part – while '==true' is not the cause of the bottleneck, it does leave an impression of bad coding style :-( Perhaps this code was created as some apprentice project, which might also explain its suboptimal performance.

The real performance problem here is that for each field that we wish to remove, rmfield compares it to all existing fields to find its location in a cell array of field names. This is algorithmically inefficient and makes the code hard to understand (just try – it took me hard, long minutes).

So, I created a variant of rmfield.m called fast_rmfield.m, as follows (again, omitting some non-essential code):

function t = fast_rmfield(s,field)
 
% get fieldnames of struct
f = fieldnames(s);
[f,ia] = setdiff(f,field,'R2012a');
 
% convert struct to cell array
c = squeeze(struct2cell(s));
 
% rebuild struct
t = cell2struct(c(ia,:),f)';

This code is much shorter, easier to explain and maintain, but also (and most importantly) much faster:

>> tic; myStruct = fast_rmfield(myStruct,fieldNames); t3 = toc
t3 =
      0.0302
 
>> t2/t3
ans =
      20.1893

This resulted in a speedup of ~850x compared to the original version (of 25.77 secs), and ~20x compared to the vectorized version. A nice improvement in my humble opinion…

The point in all this is that we can and should rewrite Matlab builtin functions when they are too slow for our needs, whether it is found to be an algorithmic flaw (as in this case), extraneous sanity checks (as in the case of ismember or datenum), bad default parameters (as in the case of fopen/fwrite or scatter), or merely slow implementation (as in the case of save, cellfun, or the conv family of functions).

A good pattern is to save such code pieces in file names that hint to the original code. In our case, I used fast_rmfield to suggest that it is a faster alternative to rmfield.

Do you know of any other example of a slow implementation in a built-in Matlab function that can be optimized? If so, please leave a comment below.

]]>
https://undocumentedmatlab.com/blog/rmfield-performance/feed 5
Viewing saved profiling resultshttps://undocumentedmatlab.com/blog/viewing-saved-profiling-results https://undocumentedmatlab.com/blog/viewing-saved-profiling-results#respond Wed, 18 May 2016 18:00:37 +0000 http://undocumentedmatlab.com/?p=6421
 
Related posts:
  1. Plot performance Undocumented inner plot mechanisms can significantly improve plotting performance ...
  2. Undocumented Profiler options part 2 Several undocumented features of the Matlab Profiler can make it much more useful - part 2 of series. ...
  3. Undocumented Profiler options part 3 An undocumented feature of the Matlab Profiler can report call history timeline - part 3 of series. ...
  4. Undocumented Profiler options part 4 Several undocumented features of the Matlab Profiler can make it much more useful - part 4 of series. ...
 
]]>
Many Matlab users know and utilize Matlab’s built-in Profiler tool to identify performance bottlenecks and code-coverage issues. Unfortunately, not many are aware of the Profiler’s programmatic interface. In past articles as well as my performance book I explained how we can use this programmatic interface to save profiling results and analyze it offline. In fact, I took this idea further and even created a utility (profile_history) that displays the function call timeline in a standalone Matlab GUI, something that is a sorely missed feature in the built-in profiler:

Function call timeline profiling (click for full-size image)
Function call timeline profiling (click for full-size image)

Today I will discuss a related undocumented feature of the Profiler: loading and viewing pre-saved profiling results.

Programmatic access to profiling results

Matlab’s syntax for returning the detailed profiling results in a data struct is clearly documented in the profile function’s doc page. Although the documentation does not explain the resulting struct and sub-struct fields, they have meaningful names and we can relatively easily infer what each of them means (I added a few annotation comments for clarity):

>> profile('on','-history')
>> surf(peaks); drawnow
>> profile('off')
>> profData = profile('info')
profData = 
      FunctionTable: [26x1 struct]
    FunctionHistory: [2x56 double]
     ClockPrecision: 4.10517962829241e-07
         ClockSpeed: 2501000000
               Name: 'MATLAB'
           Overhead: 0
 
>> profData.FunctionTable(1)
ans = 
          CompleteName: 'C:\Program Files\Matlab\R2016a\toolbox\matlab\specgraph\peaks.m>peaks'
          FunctionName: 'peaks'
              FileName: 'C:\Program Files\Matlab\R2016a\toolbox\matlab\specgraph\peaks.m'
                  Type: 'M-function'
              Children: [1x1 struct]
               Parents: [0x1 struct]
         ExecutedLines: [9x3 double]
           IsRecursive: 0
    TotalRecursiveTime: 0
           PartialData: 0
              NumCalls: 1
             TotalTime: 0.0191679078068094
 
>> profData.FunctionTable(1).Children
ans = 
        Index: 2   % index in profData.FunctionTable array
     NumCalls: 1
    TotalTime: 0.00136415141013509
 
>> profData.FunctionTable(1).ExecutedLines   % line number, number of calls, duration in secs
ans =
         43      1      0.000160102031282782
         44      1      2.29890096200918e-05
         45      1      0.00647592190637408
         56      1      0.0017093970724654
         57      1      0.00145036019621044
         58      1      0.000304193859437286
         60      1      4.39254290955326e-05
         62      1      3.44835144301377e-05
         63      1      0.000138755093778411
 
>> profData.FunctionHistory(:,1:5)
ans =
     0     0     1     1     0   % 0=enter, 1=exit
     1     2     2     1     6   % index in profData.FunctionHistory array

As we can see, this is pretty intuitive so far.

Loading and viewing saved profiling results

If we wish to save these results results in a file and later load and display them in the Profiler’s visualization browser, then we need to venture deeper into undocumented territory. It seems that while retrieving the profiling results (via profile(‘info’)) is fully documented, doing the natural complementary action (namely, loading this data into the viewer) is not. For the life of me I cannot understand the logic behind this decision, but that’s the way it is.

Luckily, the semi-documented built-in function profview does exactly what we need: profview accepts 2 input args (function name and the profData struct) and displays the resulting profiling info. The first input arg (function name) accepts either a string (e.g., 'peaks' or 'view>isAxesHandle'), or the numeric value 0 which signifies the home (top-level) page:

profView(0, profData);  % display profiling home (top-level) page
profview('peaks', profData);  % display a specific profiling page

I use the 0 input value much more frequently than the string inputs, because I often don’t know which functions exactly were profiled, and starting at the home page enables me to easily drill-down the profiling results interactively.

Loading saved profiling results from a different computer

Things get slightly complicated if we try to load saved profiling results from a different computer. If the other computer has exactly the same folder structure as our computer, and all our Matlab functions reside in exactly the same disk folders/path, then everything will work out of the box. The problem is that in general the other computer will have the functions in different folders. When we then try to load the profData on our computer, it will not find the associated Matlab functinos in order to display the line-by-line profiling results. We will only see the profiling data at the function level, not line level. This significantly reduces the usefulness of the profiling data. The Profiler page will display the following error message:

This file was modified during or after profiling. Function listing disabled.

We can solve this problem in either of two ways:

  1. Modify our profData to use the correct folder path on the local computer, rather than the other computer’s path (which is invalid on the local computer). For example:
    % Save the profData on computer #1:
    profData = profile('info');
    save('profData.mat', 'profData');
     
    % Load the profData on computer #2:
    fileData = load('profData.mat');
    profData = fileData.profData;
    path1 = 'N:\Users\Juan\programs\myProgram';
    path2 = 'C:\Yair\consulting\clients\Intel\code';
    for idx = 1 : numel(profData.FunctionTable)
       funcData = profData.FunctionTable(idx);
       funcData.FileName     = strrep(funcData.FileName,     path1, path2); 
       funcData.CompleteName = strrep(funcData.CompleteName, path1, path2);
       profData.FunctionTable(idx) = funcData;
    end
    % note: this loop can be vectorized if you wish
  2. As an alternative, we can modify Matlab’s profview.m function (%matlabroot%/toolbox/matlab/codetools/profview.m) to search for the function’s source code in the current Matlab path, if the specified direct path is not found (note that changing profview.m may require administrator priviledges). For example, the following is the code from R2016a’s profview.m file, line #506:
        % g894021 - Make sure the MATLAB code file still exists
        if ~exist(fullName, 'file')
            [~,fname,fext] = fileparts(fullName);  % Yair        fname = which([fname fext]);           % Yair        if isempty(fname)                      % Yair            mFileFlag = 0;
            else                                   % Yair            fullName = fname;                  % Yair        end                                    % Yair    end

These two workarounds complement each other: the first workaround does not require changing any installed Matlab code, and so is platform- and release-independent, but would require rerunning the code snippet for each and every profiling data file that we receive from external computers. On the other hand, the second workaround is a one-time operation that should work for multiple saved profiling results, although we would need to redo it whenever we install Matlab.

Additional profview customizations

Modifying the profview.m function can be used for different improvements as well.

For example, several years ago I explained how this function can be modified to display 1 ms timing resolutions, rather than the default 10 mS.

Another customization that I often do after I install Matlab is to change the default setting of truncating function lines longer than 40 characters – I typically modify this to 60 or 80 (depending on the computer monitor’s size…). All we need to do is to update the truncateDisplayName sub-function within profview.m as follows (taken from R2016a again, line #1762):

function shortFileName = truncateDisplayName(longFileName,maxNameLen)
%TRUNCATEDISPLAYNAME  Truncate the name if it gets too long
maxNameLen = max(60,maxNameLen);  % YairshortFileName = escapeHtml(longFileName);
if length(longFileName) > maxNameLen,
    shortFileName = char(com.mathworks.util.FileUtils.truncatePathname( ...
        shortFileName, maxNameLen));
end

You can see additional undocumented profiling features in the “Related posts” section below, as well as in Chapter 2 of my book “Accelerating MATLAB Performance“.

Do you have any other customization to the profiling results? If so, please share it in a comment.

]]>
https://undocumentedmatlab.com/blog/viewing-saved-profiling-results/feed 0
Convolution performancehttps://undocumentedmatlab.com/blog/convolution-performance https://undocumentedmatlab.com/blog/convolution-performance#comments Wed, 03 Feb 2016 19:00:35 +0000 http://undocumentedmatlab.com/?p=6256
 
Related posts:
  1. tic / toc – undocumented option Matlab's built-in tic/toc functions have an undocumented option enabling multiple nested clockings...
  2. Plot LimInclude properties The plot objects' XLimInclude, YLimInclude, ZLimInclude, ALimInclude and CLimInclude properties are an important feature, that has both functional and performance implications....
  3. Matrix processing performance Matrix operations performance is affected by internal subscriptions in a counter-intuitive way....
  4. Preallocation performance Preallocation is a standard Matlab speedup technique. Still, it has several undocumented aspects. ...
 
]]>
MathWorks’ latest MATLAB Digest (January 2016) featured my book “Accelerating MATLAB Performance“. I am deeply honored and appreciative of MathWorks for this.

Accelerating MATLAB Performance bookI would like to dedicate today’s post to a not-well-known performance trick from my book, that could significantly improve the speed when computing the convolution of two data arrays. Matlab’s internal implementation of convolution (conv, conv2 and convn) appears to rely on a sliding window approach, using implicit (internal) multithreading for speed.

However, this can often be sped up significantly if we use the Convolution Theorem, which states in essence that conv(a,b) = ifft(fft(a,N) .* fft(b,N)), an idea proposed by Bruno Luong. In the following usage example we need to remember to zero-pad the data to get comparable results:

% Prepare the input vectors (1M elements each)
x = rand(1e6,1);
y = rand(1e6,1);
 
% Compute the convolution using the builtin conv()
tic, z1 = conv(x,y); toc
 => Elapsed time is 360.521187 seconds.
 
% Now compute the convolution using fft/ifft: 780x faster!
n = length(x) + length(y) - 1;  % we need to zero-pad
tic, z2 = ifft(fft(x,n) .* fft(y,n)); toc
 => Elapsed time is 0.463169 seconds.
 
% Compare the relative accuracy (the results are nearly identical)
disp(max(abs(z1-z2)./abs(z1)))
 => 2.75200348450538e-10

This latest result shows that the results are nearly identical, up to a tiny difference, which is certainly acceptable in most cases when considering the enormous performance speedup (780x in this specific case). Bruno’s implementation (convnfft) is made even more efficient by using MEX in-place data multiplications, power-of-2 FFTs, and use of GPU/Jacket where available.

It should be noted that the builtin Matlab functions can still be faster for relatively small data arrays, or if your machine has a large number of CPU cores and free memory that Matlab’s builtin conv* functions can utilize, and of course also depending on the Matlab release. So, your mileage might well vary. But given the significant speedup potential, I contend that you should give it a try and see how well it performs on your specific system and data.

If you have read my book, please be kind enough to post your feedback about it on Amazon (link), for the benefit of others. Thanks in advance!

]]>
https://undocumentedmatlab.com/blog/convolution-performance/feed 6
Graphic sizing in Matlab R2015bhttps://undocumentedmatlab.com/blog/graphic-sizing-in-matlab-r2015b https://undocumentedmatlab.com/blog/graphic-sizing-in-matlab-r2015b#comments Wed, 20 Jan 2016 18:00:31 +0000 http://undocumentedmatlab.com/?p=6244
 
Related posts:
  1. HG2 update HG2 appears to be nearing release. It is now a stable mature system. ...
  2. Modifying default toolbar/menubar actions The default Matlab figure toolbar and menu actions can easily be modified using simple pure-Matlab code. This article explains how....
  3. FIG files format FIG files are actually MAT files in disguise. This article explains how this can be useful in Matlab applications....
  4. A couple of internal Matlab bugs and workarounds A couple of undocumented Matlab bugs have simple workarounds. ...
 
]]>
I would like to introduce Daniel Dolan of Sandia National Laboratories. Dan works on a variety of data analysis projects in Matlab, and is an active lurker on MATLAB Central. Dan has a habit of finding interesting bugs for the Mac version of Matlab. Today he will discuss graphic sizing in Matlab and important changes that occurred in release R2015b.

Matlab-generated graphics are often not displayed at their requested size. This problem has been known for some time and has a well-known solution: setting the root object’s ScreenPixelsPerInch property to the display’s actual DPI (dots per inch) value. Release R2015b no longer supports this solution, creating problems for publication graphics and general readability.

Physical sizing in R2015a vs. R2015b (click for full-size)

Physical sizing in R2015a vs. R2015b (click for full-size)


Physical sizing

Matlab supports graphic sizing in various physical units: inches, centimeters, and points. For example:

figure; axes('Box','on', 'Units','inches','Position',[0.3 0.3 4 4]);

requests to display an axes having square sizes measuring exactly 4″ (101.6 mm) each. It is evident, however, that the displayed axes is smaller than 4″. The mismatch between requested and physical size depends on the display and operating system — go ahead, try it on your system. The problem is particularly severe on Mac laptops, presumably even worse for those with Retina displays.

The problem is that Matlab cannot determine pixel size, which varies from one display to the other. Generating a figure spanning a particular number of pixels (e.g., 1024 x 768) is easy, but absolute physical units requires a conversion factor called ScreenPixelsPerInch, which is a root property (see related post on setting/getting default graphics property values):

DPI = 110;                             % dots per inch for my 27" Apple Cinema Display
set(0,    'ScreenPixelsPerInch',DPI);  % all releases prior to R2015b
set(groot,'ScreenPixelsPerInch',DPI);  % R2014b through R2015a

DPI values tend to be higher for laptops, usually in the 120-130 range. Retina displays are supposed to be >300 DPI, but I have not been able to test that myself.

There are several ways to determine the correct DPI setting for a particular display. It may be available in the hardware specifications, and it can be calculated from the diagonal size and the number of pixels. Unfortunately these methods are not always reliable. If you really care about physical sizing, the best approach is to actually calibrate your display. There are tools for doing this at Matlab Central, but it’s not hard to do manually:

  • Create a figure.
  • Manually resize the figure to match a convenient width. I often use a piece of US letter paper as 8.5″ guide on the display.
  • Determine the width of the figure in pixels:
    set(gcf,'Units','pixels');
    pos = get(gcf,'Position');
    width = 8.5; % inches
    DPI = pos(3) / width;

I usually apply the DPI settings in my startup file so that Matlab begins with a calibrated display.

What changed in 2015b?

ScreenPixelsPerInch is a read-only property in R2015b, so display calibration no longer works. The following sequence of commands:

figure('Units','inches', 'PaperPositionMode','auto', 'Position',[0 0 4 4]);
set(gcf, 'MenuBar','none', 'ToolBar','none', 'DockControls','off', 'NumberTitle','off');
axes('FontUnits','points', 'FontSize',10);
image

now renders differently in R2015b than does for a calibrated display in R2015a. Differences between the two outputs are shown in the screenshot at the top of this post. The grid behind the figures was rendered at 8.5″ x 8.5″ inches on my display; if your browser’s zoom level isn’t 100%, it may appear larger or smaller.

A side effect of improper graphic sizing is that text is difficult to read — the uncalibrated axes labels are clearly smaller than 10 points. These examples were rendered on ~110 DPI display. Matlab assumes that Macs use 72 DPI (96 DPI on Windows), so graphics appear at 65% of the request size.

The loss of ScreenPixelsPerInch as an adjustable setting strongly affects anyone using Matlab for publication graphics. Scientific and engineering journals are extremly strict about figure widths. With a calibrated screen, figure appear exactly as they will when printed to a file (usually EPS or PDF). Figures are often made as small as possible to and densely packed to save journal space, and accurate sized display helps the author determine legibility. Displaying accurately sized graphics is very difficult in R2015b, which is unfortunate given the many enhancements in this release.

Developers who create graphical interfaces for other users should also care about this change. A common complaint I get is that text and control labels is too small to easily read. Screen calibration deals with this problem, but this option is no longer available.

Where do we go from here?

I reported the above issues to the Mathworks several months ago. It does not appear as a formal bug, but technical support is aware of the problem. The change is part of the “DPI aware” nature of release R2015b. So far I have found no evidence this release is any more aware of pixel size than previous releases, but my experience is limited to non-Retina Macs. I welcome input from users on other operating systems, particularly those with high-resolution displays.

To be fair, correct physical sizing is not an easy across the many platforms that Matlab runs on. Display resolution is particularly tricky when it changes during a Matlab session, such as when computer is connector to projector/television or a laptop is connected to a docking station.

Thankfully, printed graphic sizes are rendered correctly when a figure’s PaperPositionMode property is 'auto'. Many users can (and will) ignore the display problem if they aren’t dealing with strict size requirements and text legibility isn’t too bad. Some users may be willing to periodically print publication figures to externally verify sizing, but this breaks the interactive nature of Matlab figures.

A potential work around is the creating of a new figure class that oversizes figures (as needed) to account for a particular display. I started working on such a class, but the problem is more complicated than one might think:

  • Child objects (axes, uicontrols, etc.) also must be resized if they are based on physical units.
  • Resized objects must be temporarily restored to their original size for printing, and new objects must be tracked whenever they are added.
  • Figure resolution may need to be changed when moving to different computer systems.

These capabilities are quite possible to implement, but this is a complicated solution to problem that was once easy to fix.

Retina displays don’t suffer as badly as one might think from the DPI mismatch. Even though the display specification may be greater than 200 DPI, OS X and/or Matlab must perform some intermediate size transformations. The effective DPI in R2015a is 110-120 for 13-15″ MacBook Pro laptops (at the default resolution). Objected sized with physical units still appear smaller than they should (~72/110), but not as small as I expected (<72/200).

Effect pixel size can also be changed by switching between different monitor scalings. This isn’t entirely surprising, but it can lead to some interesting results because Matlab only reads these settings at startup. Changing the display scaling during a session can cause square figures to appear rectangular. Also, the effective DPI changes for setting: I could reach values of ~60-110 DPI on an Apple Cinema Display.

So where does this leave us? Display calibration was always a finicky matter, but at least in principle one could make graphics appear exactly the same size on two different displays. Now it seems that sizing is completely variable between operation systems, displays, and display settings. For publication graphics, there will almost always be a disconnect between figure size on the screen and the printed output; some iteration may be needed to ensure everything looks right in the finished output. For graphical interfaces, font sizes may need to generated in normalized units and then converted to pixels (to avoid resizing).

Physical accuracy may not be important for non-publication figures, but the issue of text legibility remains. Some text objects–such as axes and tick labels–can easily be resized because the parent axes automatically adjusts itself as needed. Free floating text objects and uincontrols are much more difficult to deal with. Controls are often sized around the extent of their text label, so changing font sizes may require changes to the control position; adjacent controls may overlap after resizing for text clarity. Normalized units partially solve this problem, but their effect on uicontrols is not always desirable: do you really want push buttons to get larger/smaller when the figure is resized?

Can you think of a better workaround to this problem? If so, then please post a comment below. I will be very happy to hear your ideas, as I’m sure others who have high resolution displays would as well.

(cross-reference: CSSM newsgroup post)

Addendum Dec 31, 2016: Dan Dolan just posted a partial workaround on the MathWorks File Exchange. Also see the related recent article on working with non-standard DPI values.

]]>
https://undocumentedmatlab.com/blog/graphic-sizing-in-matlab-r2015b/feed 8
Customizing contour plotshttps://undocumentedmatlab.com/blog/customizing-contour-plots https://undocumentedmatlab.com/blog/customizing-contour-plots#comments Wed, 18 Nov 2015 18:00:55 +0000 http://undocumentedmatlab.com/?p=6075
 
Related posts:
  1. Draggable plot data-tips Matlab's standard plot data-tips can be customized to enable dragging, without being limitted to be adjacent to their data-point. ...
  2. Matlab’s HG2 mechanism HG2 is presumably the next generation of Matlab graphics. This article tries to explore its features....
  3. getundoc – get undocumented object properties getundoc is a very simple utility that displays the hidden (undocumented) properties of a specified handle object....
  4. Handle Graphics Behavior HG behaviors are an important aspect of Matlab graphics that enable custom control of handle functionality. ...
 
]]>
One of my clients asked me last week whether it is possible to access and customize individual contour lines and labels in HG2 (Matlab’s new graphics system, R2014+). Today’s post will discuss how this could indeed be done.

Matlab contour plot

Matlab contour plot

In HG1 (R2014a and earlier), contour handles were simple hggroup objects that incorporated text and patch child handles. The contour labels, lines and fill patches could easily be accessed via these child handles (contour lines and fills use the same patch object: the lines are simply the patch edges; fills are their faces). The lines could then be customized, the label strings changed, and the patch faces (fills) recolored:

[X,Y,Z] = peaks;
[C,hContour] = contour(X,Y,Z,20, 'ShowText','on');
hChildren = get(hContour, 'Children');
set(hChildren(1), 'String','Yair', 'Color','b');  % 1st text (contour label)
set(hChildren(end), 'EdgeColor',[0,1,1]);         % last patch (contour line)

The problem is that in HG2 (R2014b onward), contour (and its sibling functions, contourf etc.) return a graphic object that has no accessible children. In other words, hContour.Children returns an empty array:

>> hContour.Children
ans = 
  0x0 empty GraphicsPlaceholder array.
>> allchild(hContour)
ans = 
  0x0 empty GraphicsPlaceholder array.
>> isempty(hContour.Children)
ans =
     1

So how then can we access the internal contour patches and labels?

HG2’s contour object’s hidden properties

Skipping several fruitless dead-ends, it turns out that in HG2 the text labels, lines and fills are stored in undocumented hidden properties called TextPrims, EdgePrims and (surprise, surprise) FacePrims, which hold corresponding arrays of matlab.graphics.primitive.world.Text, matlab.graphics.primitive.world.LineStrip and matlab.graphics.primitive.world.TriangleStrip object handles (the drawnow part is also apparently very important, otherwise you might get errors due to the Prim objects not being ready by the time the code is reached):

>> drawnow;  % very important!
>> hContour.TextPrims  % row array of Text objects
ans = 
  1x41 Text array:
 
  Columns 1 through 14
    Text    Text    Text    Text    Text    Text    Text    Text    Text    Text    Text    Text    Text    Text
  Columns 15 through 28
    Text    Text    Text    Text    Text    Text    Text    Text    Text    Text    Text    Text    Text    Text
  Columns 29 through 41
    Text    Text    Text    Text    Text    Text    Text    Text    Text    Text    Text    Text
 
>> hContour.EdgePrims  % column array of LineStrip objects
ans = 
  20x1 LineStrip array:
 
  ineStrip
  LineStrip
  LineStrip
  ...
 
>> hContour.FacePrims  % column array of TriangleStrip objects (empty if no fill)
ans = 
  0x0 empty TriangleStrip array.

We can now access and customize the individual contour lines, labels and fills:

hContour.TextPrims(4).String = 'Dani';
hContour.TextPrims(7).Visible = 'off';
hContour.TextPrims(9).VertexData = single([-1.3; 0.5; 0]);  % Label location in data units
 
hContour.EdgePrims(2).ColorData = uint8([0;255;255;255]);  % opaque cyan
hContour.EdgePrims(5).Visible = 'off';

Note that the LineStrip objects here are the same as those used for the axes Axles, which I described a few months ago. Any customization that we could do to the axle LineStrips can also be applied to contour LineStrips, and vice versa.

For example, to achieve the appearance of a topographic map, we might want to modify some contour lines to use dotted LineStyle and other lines to appear bold by having larger LineWidth. Similarly, we may wish to hide some labels (by setting their Visible property to ‘off’) and make other labels bold (by setting their Font.Weight property to ‘bold’). There are really numerous customization possibilities here.

Here is a listing of the standard (non-hidden) properties exposed by these objects:

>> get(hContour.TextPrims(1))
        BackgroundColor: []
              ColorData: []
              EdgeColor: []
                   Font: [1x1 matlab.graphics.general.Font]
          FontSmoothing: 'on'
       HandleVisibility: 'on'
                HitTest: 'off'
    HorizontalAlignment: 'center'
            Interpreter: 'none'
                  Layer: 'middle'
              LineStyle: 'solid'
              LineWidth: 1
                 Margin: 1
                 Parent: [1x1 Contour]
          PickableParts: 'visible'
               Rotation: 7.24591082075548
                 String: '-5.1541'
          StringBinding: 'copy'
             VertexData: [3x1 single]
      VerticalAlignment: 'middle'
                Visible: 'on'
 
>> get(hContour.EdgePrims(1))
          AlignVertexCenters: 'off'
             AmbientStrength: 0.3
                ColorBinding: 'object'
                   ColorData: [4x1 uint8]
                   ColorType: 'truecolor'
             DiffuseStrength: 0.6
            HandleVisibility: 'on'
                     HitTest: 'off'
                       Layer: 'middle'
                     LineCap: 'none'
                    LineJoin: 'round'
                   LineStyle: 'solid'
                   LineWidth: 0.5
               NormalBinding: 'none'
                  NormalData: []
                      Parent: [1x1 Contour]
               PickableParts: 'visible'
    SpecularColorReflectance: 1
            SpecularExponent: 10
            SpecularStrength: 0.9
                   StripData: [1 18]
                     Texture: [0x0 GraphicsPlaceholder]
                  VertexData: [3x17 single]
               VertexIndices: []
                     Visible: 'on'
       WideLineRenderingHint: 'software'
 
>> get(hContour.FacePrims(1))
             AmbientStrength: 0.3
             BackFaceCulling: 'none'
                ColorBinding: 'object'
                   ColorData: [4x1 uint8]
                   ColorType: 'truecolor'
             DiffuseStrength: 0.6
            HandleVisibility: 'on'
                     HitTest: 'off'
                       Layer: 'middle'
               NormalBinding: 'none'
                  NormalData: []
                      Parent: [1x1 Contour]
               PickableParts: 'visible'
    SpecularColorReflectance: 1
            SpecularExponent: 10
            SpecularStrength: 0.9
                   StripData: [1 4 13 16 33 37 41 44 51 54 61 64 71 74 87 91 94 103]
                     Texture: [0x0 GraphicsPlaceholder]
            TwoSidedLighting: 'off'
                  VertexData: [3x102 single]
               VertexIndices: []
                     Visible: 'on'

But how did I know these properties existed? The easiest way in this case would be to use my getundoc utility, but we could also use my uiinspect utility or even the plain-ol’ struct function.

p.s. – there’s an alternative way, using the Java bean adapter that is associated with each Matlab graphics object: java(hContour). Specifically, this object apparent has the public method browseableChildren(java(hContour)) which returns the list of all children (in our case, 41 text labels [bean adapters], 20 lines, and a single object holding a ListOfPointsHighlight that corresponds to the regular hidden SelectionHandle property). However, I generally dislike working with the bean adapters, especially when there’s a much “cleaner” way to get these objects, in this case using the regular EdgePrims, FacePrims, TextPrims and SelectionHandle properties. Readers who are interested in Matlab internals can explore the bean adapters using a combination of my getundoc and uiinspect utilities.

So far for the easy part. Now for some more challenging questions:

Customizing the color

First, can we modify the contour fill to have a semi- (or fully-) transparent fill color? – indeed we can:

[~, hContour] = contourf(peaks(20), 10);
drawnow;  % this is important, to ensure that FacePrims is ready in the next line!
hFills = hContour.FacePrims;  % array of TriangleStrip objects
[hFills.ColorType] = deal('truecoloralpha');  % default = 'truecolor'
for idx = 1 : numel(hFills)
   hFills(idx).ColorData(4) = 150;   % default=255
end

Contour plot in HG2, with and without transparency

Contour plot in HG2, with and without transparency

Similar transparency effects can also be applied to the LineStrip and Text objects. A discussion of the various combinations of acceptable color properties can be found here.

Mouse clicks

Next, how can we set a custom context-menu for individual labels and contour lines?

Unfortunately, Text, LineStrip and TriangleStrip objects do not posses a ButtonDownFcn or UIContextMenu property, not even hidden. I tried searching in the internal/undocumented properties, but nothing came up.

Mouse click solution #1

So the next logical step would be to trap the mouse-click event at the contour object level. We cannot simply click the contour and check the clicked object because that would just give us the hContour object handle rather than the individual Text or LineStrip. So the idea would be to set hContour.HitTest='off', in the hope that the mouse click would be registered on the graphic object directly beneath the mouse cursor, namely the label or contour line. It turns out that the labels’ and lines’ HitTest property is ‘off’ by default, so, we also need to set them all to ‘on’:

hContour.HitTest = 'off';
[hContour.TextPrims.HitTest] = deal('on');
[hContour.EdgePrims.HitTest] = deal('on');
[hContour.FacePrims.HitTest] = deal('on');
hContour.ButtonDownFcn = @(h,e)disp(struct(e));

This seemed simple enough, but failed spectacularly: it turns out that because hContour.HitTest='off', mouse clicks are not registered on this objects, and on the other hand we cannot set the ButtonDownFcn on the primitive objects because they don’t have a ButtonDownFcn property!

Who said life is easy?

One workaround is to set the figure’s WindowButtonDownFcn property:

set(gcf, 'WindowButtonDownFcn', @myMouseClickCallback);

Now, inside your myMouseClickCallback function you can check the clicked object. We could use the undocumented builtin hittest(hFig) function to see which object was clicked. Alternatively, we could use the callback eventData‘s undocumented HitObject/HitPrimitive properties (this variant does not require the HitTest property modifications above):

function myMouseClickCallback(hFig, eventData)
   hitPrimitive = hittest(hFig);  % undocumented function
 
   hitObject    = eventData.HitObject;     % undocumented property => returns a Contour object (=hContour)
   hitPrimitive = eventData.HitPrimitive;  % undocumented property => returns a Text or LineStrip object
   hitPoint     = eventData.Point;         % undocumented property => returns [x,y] pixels from figure's bottom-left corner
 
   if strcmpi(hFig.SelectionType,'alt')  % right-click
      if isa(hitPrimitive, 'matlab.graphics.primitive.world.Text')  % label
         displayTextContextMenu(hitPrimitive, hitPoint)
      elseif isa(hitPrimitive, 'matlab.graphics.primitive.world.LineStrip')  % contour line
         displayLineContextMenu(hitPrimitive, hitPoint)
      elseif isa(hitPrimitive, 'matlab.graphics.primitive.world.TriangleStrip')  % contour fill
         displayFillContextMenu(hitPrimitive, hitPoint)
      else
         ...
      end
   end
end
Mouse click solution #2

A totally different solution is to keep the default hContour.HitTest='on' (and the primitives’ as ‘off’) and simply query the contour object’s ButtonDownFcn callback’s eventData‘s undocumented Primitive property:

hContour.ButtonDownFcn = @myMouseClickCallback;

And in the callback function:

function myMouseClickCallback(hContour, eventData)
   hitPrimitive = eventData.Primitive;  % undocumented property => returns a Text or LineStrip object
   hitPoint     = eventData.IntersectionPoint;  % [x,y,z] in data units
 
   hFig = ancestor(hContour, 'figure');
   if strcmpi(hFig.SelectionType,'alt')  % right-click
      if isa(hitPrimitive, 'matlab.graphics.primitive.world.Text')  % label
         displayTextContextMenu(hitPrimitive, hitPoint)
      elseif isa(hitPrimitive, 'matlab.graphics.primitive.world.LineStrip')  % contour line
         displayLineContextMenu(hitPrimitive, hitPoint)
      elseif isa(hitPrimitive, 'matlab.graphics.primitive.world.TriangleStrip')  % contour fill
         displayFillContextMenu(hitPrimitive, hitPoint)
      else
         ...
      end
   end
end

This article should be a good start in how to code the displayTextContextMenu etc. functions to display a context menu.

Customizations reset

Finally, there are apparently numerous things that cause our customized labels and lines to reset to their default appearance: resizing, updating contour properties etc. To update the labels in all these cases in one place, simply listen to the undocumented MarkedClean event:

addlistener(hContour, 'MarkedClean', @updateLabels);

Where updateLabels is a function were you set all the new labels.

Prediction about forward compatibility

I am marking this article as “High risk of breaking in future Matlab versions“, not because of the basic functionality (being important enough I don’t presume it will go away anytime soon) but because of the property names: TextPrims, EdgePrims and FacePrims don’t seem to be very user-friendly property names. So far MathWorks has been very diligent in making its object properties have meaningful names, and so I assume that when the time comes to expose these properties, they will be renamed (perhaps to TextHandles, EdgeHandles and FaceHandles, or perhaps LabelHandles, LineHandles and FillHandles). For this reason, even if you find out in some future Matlab release that TextPrims, EdgePrims and FacePrims don’t exist, perhaps they still exist and simply have different names.

]]>
https://undocumentedmatlab.com/blog/customizing-contour-plots/feed 8
Adding dynamic properties to graphic handleshttps://undocumentedmatlab.com/blog/adding-dynamic-properties-to-graphic-handles https://undocumentedmatlab.com/blog/adding-dynamic-properties-to-graphic-handles#comments Wed, 16 Sep 2015 17:26:44 +0000 http://undocumentedmatlab.com/?p=6006
 
Related posts:
  1. New information on HG2 More information on Matlab's new HG2 object-oriented handle-graphics system...
  2. Performance: accessing handle properties Handle object property access (get/set) performance can be significantly improved using dot-notation. ...
  3. uiundo – Matlab’s undocumented undo/redo manager The built-in uiundo function provides easy yet undocumented access to Matlab's powerful undo/redo functionality. This article explains its usage....
  4. Matlab’s HG2 mechanism HG2 is presumably the next generation of Matlab graphics. This article tries to explore its features....
 
]]>
A client recently asked me to extend one of Matlab’s built-in graphic containers (uiflowcontainer in this specific case) with automatic scrollbars that would enable the container to act as a scroll-panel. The basic idea would be to dynamically monitor the container’s contents and when it is determined that they overflow the container’s boundaries, then attach horizontal/vertical scrollbars to enable scrolling the contents into view:

Scrollable Matlab container

Scrollable Matlab container

This may sound simple, but there are actually quite a few undocumented hacks that make this possible, including listening to ObjectChildAdded/ObjectChildRemoved events, location/size/visibility events, layout changes etc. Maybe I’ll blog about it in some future article.

Today’s post is focused on a specific aspect of this project, attaching dynamic properties to the builtin uiflowcontainer, that would enable users to modify the container’s properties directly, as well as control aspects of the scrolling using the new properties: handles to the parent container, as well as the horizontal and vertical scrollbars, and even a new refresh() method.

The “textbook” approach to this would naturally be to create a new class that extends (inherits) uiflowcontainer and includes these new properties and methods. Unfortunately, for some reason that escapes my understanding, MathWorks saw fit to make all of its end-use graphic object classes Sealed, such that they cannot be extended by users. I did ask for this to be changed long ago, but the powers that be apparently decided that it’s better this way.

So the fallback would be to create our own dedicated class having all the new properties as well as those of the original container, and ensure that all the property values are synchronized in both directions. This is probably achievable, if you have a spare few days and a masochistic state of mind. Being the lazy bum and authority-rebel that I am, I decided to take an alternate approach that would simply add my new properties to the built-in container handle. The secret lies in the undocumented function schema.prop (for HG1, R2014a and older) and the fully-documented addprop function (for HG2, R2014b and newer).

In the examples below I use a panel, but this mechanism works equally well on any Matlab HG object: axes, lines, uicontrols, figures, etc.

HG2 – addprop function

The addprop function is actually a public method of the dynamicprops class. Both the dynamicprops class as well as its addprop function are fully documented. What is NOT documented, as far as I could tell, is that all of Matlab’s builtin handle graphics objects indirectly inherit dynamicprops, via matlab.graphics.Graphics, which is a high-level superclass for all HG objects. The bottom line is that we can dynamically add run-time properties to any HG object, without affecting any other object. In other words, the new properties will only be added to the handles that we specifically request, and not to any others. This suits me just fine:

hProp = addprop(hPanel, 'hHorizontalScrollBar');
hPanel.hHorizontalScrollBar = hMyScrollbar;
hProp.SetAccess = 'private';  % make this property read-only

The new property hHorizontalScrollBar is now added to the hPanel handle, and can be accessed just like any other read-only property. For example:

>> get(hPanel, 'hHorizontalScrollBar')
ans = 
    JavaWrapper

>> hPanel.hHorizontalScrollBar
ans = 
    JavaWrapper

>> hPanel.hHorizontalScrollBar = 123
You cannot set the read-only property 'hHorizontalScrollBar' of UIFlowContainer.

Adding new methods is more tricky, since we do not have a corresponding addmethod function. The trick I used was to create a new property having the requested new method’s name, and set its read-only value to a handle of the requested function. For example:

hProp = addprop(hPanel, 'refresh');
hPanel.refresh = @myRefreshFunc;
hProp.SetAccess = 'private';  % make this property read-only

We can then invoke the new refresh “method” using the familiar dot-notation:

hPanel.refresh();

Note: if you ever need to modify the initial value in your code, you should revert the property’s SetAccess meta-property to 'public' before Matlab will enable you to modify the value:

try
    % This will raise an exception if the property already exists
    hProp = addprop(hPanel, propName);
catch
    % Property already exists - find it and set its access to public
    hProp = findprop(hPanel, propName);
    hProp.SetAccess = 'public';
end
hPanel.(propName) = newValue;

HG1 – schema.prop function

In HG1 (R2014a and earlier), we can use the undocumented schema.prop function to add a new property to any HG handle (which is a numeric value in HG1). Donn Shull wrote about schema.prop back in 2011, as part of his series of articles on UDD (Unified Data Dictionary, MCOS’s precursor). In fact, schema.prop is so useful that it has its own blog tag here and appears in no less than 15 separate articles (excluding today). With HG2’s debut 2 years ago, MathWorks tried very hard to rid the Matlab code corpus of all the legacy schema-based, replacing most major functionalities with MCOS-based HG2 code. But so far it has proven impossible to get rid of schema completely, and so schema code is still used extensively in Matlab to this day (R2015b). Search your Matlab path for “schema.prop” and see for yourself.

Anyway, the basic syntax is this:

hProp = schema.prop(hPanel, propName, 'mxArray');

The 'mxArray' specifies that the new property can accept any data type. We can limit the property to only accept certain types of data by specifying a less-generic data type, among those recognized by UDD (details).

Note that the meta-properties of the returned hProp are somewhat different from those of HG2’s hProp. Taking this into account, here is a unified function that adds/updates a new property (with optional initial value) to any HG1/HG2 object:

function addProp(hObject, propName, initialValue, isReadOnly)
    try
        hProp = addprop(hObject, propName);  % HG2
    catch
        try
            hProp = schema.prop(hObject, propName, 'mxArray');  % HG1
        catch
            hProp = findprop(hObject, propName);
        end
    end
    if nargin > 2
        try
            hProp.SetAccess = 'public';  % HG2
        catch
            hProp.AccessFlags.PublicSet = 'on';  % HG1
        end
        hObject.(propName) = initialValue;
    end
    if nargin > 3 && isReadOnly
        try
            % Set the property as read-only
            hProp.SetAccess = 'private';  % HG2
        catch
            hProp.AccessFlags.PublicSet = 'off';  % HG1
        end
    end
end
]]>
https://undocumentedmatlab.com/blog/adding-dynamic-properties-to-graphic-handles/feed 4
Solving an mput (FTP) hang problemhttps://undocumentedmatlab.com/blog/solving-an-mput-ftp-hang-problem https://undocumentedmatlab.com/blog/solving-an-mput-ftp-hang-problem#comments Wed, 19 Aug 2015 20:10:17 +0000 http://undocumentedmatlab.com/?p=5974
 
Related posts:
  1. More undocumented timing features There are several undocumented ways in Matlab to get CPU and clock data...
  2. Uitable customization report Matlab's uitable can be customized in many different ways. A detailed report explains how. ...
  3. Waiting for asynchronous events The Matlab waitfor function can be used to wait for asynchronous Java/ActiveX events, as well as with timeouts. ...
  4. uisplittool & uitogglesplittool callbacks Matlab's undocumented uisplittool and uitogglesplittool are powerful toolbar controls - this article explains how to customize their behavior...
 
]]>
Matlab includes a variety of builtin utility functions that enable easy access to FTP. These functions are basically methods of an FTP object using Matlab’s old class system (for example, the mput function is implemented as mput.m in the %matlabroot%\toolbox\matlab\iofun\@ftp\ folder). These are pretty old files that haven’t changed much in years.

FTPI recently needed to upload files from Matlab onto an FTP server, and discovered that calling mput simply hang Matlab to the point that I needed to kill and restart the Matlab process. The problem was not in the FTP server, since it could be accessed normally using FTP clients such as FileZilla and WinSCP. So it had to be something internal to Matlab. My initial workaround was to create an automation script (using WinSCP in my case) for the file upload. But this kludge is both non-robust as well as slow. A fix to the Matlab problem would be much better.

Some online research yielded others who have complained about similar issues over the years, but I saw no concrete answer. I saw many references online to problems that relate to the combination of passive FTP with Windows 7 / firewall / Java 7, that suggested several fixes (example 1, example 2, example 3, example 4). However, none of them solved the problem: calling mput (or dir etc.) continued to freeze Matlab, forcing either a hard process kill or waiting several minutes for the timeout.

Today, Malcolm Lidierth, author of the Waterloo graphics package, told me that he also saw a similar situation on Mac. Fortunately, Malcolm discovered a much better solution to the problem than my external scripts workaround. It seems that simply setting the Java object underlying Matlab’s FTP object to use passive mode fixes the problem (possibly in addition to the other fixes mentioned above).

This can be done in two manners:

The easiest is to add the following highlighted line in mput.m, on or about line 69 (the exact line may change based on your Matlab release):

...
% Upload this file.
fis = java.io.FileInputStream(fileObject);
h.jobject.enterLocalPassiveMode();  % Use passive modestatus = h.jobject.storeFile(name,fis);
fis.close;
...

This works but has several limitations:

  1. it does not solve the problems of other passive FTP commands such as dir, although they can be solved in a similar manner
  2. it creates a counter-problem when connecting to non-passive FTP servers – users might wish to make this a parameter of the ftp constructor function
  3. and of course it requires changing Matlab’s installation files which is problematic in many aspects, as well as non-portable if you ever use your program on another machine or Matlab release.

Here’s a much simpler, portable and flexible solution: simply set the underlying Java object’s passive mode after connecting to the server (using the ftp constructor function) but before using mput or dir or any other command that uses passive FTP mode:

f = ftp('myftpserver.com',username,password);
cd(f);  sf=struct(f);  sf.jobject.enterLocalPassiveMode();mput(f,filename);
dir(f);
close(f);

Note that we need to connect first, then get the underlying Java reference using the struct hack (since direct access to f.jobject is prevented), then we enter local passive mode, and only then we call mput to upload the file. If we omit the highlighted line in the script above, then mput (and dir etc.) will hang.

This way, we can programmatically control when to use passive mode, and no changes to the Matlab install files is required: our Matlab script should now work on all platforms and Matlab releases.

Note the seemingly unnecessary call to cd(f) – this is done to ensure a valid connection before setting the passive mode. We should ensure to do this before each call to mput/dir etc., since otherwise, if the connection drops for any reason (e.g., timeout or some other disconnection), then the mput/dir command would reconnect without passive mode (causing a hang). By calling cd(f) we ensure that the connection is done and passive mode is re-entered before mput/dir are called.

As an interesting related note, ftp accepts an undocumented set of optional input parameters following its first 3 documented inputs (hostname, username and password). We can pass one or more of the following parameters in P-V (param name, param value) pairs format: System, LenientFutureDates, DefaultDateFormatStr, RecentDateFormatStr, ServerLanguageCode, ServerTimeZoneId, ShortMonthNames. All of these parameters expect string (char) values, except LenientFutureDates that expects a logical value (true/false). All the parameters have a default value of empty. An explanation of these parameters can be found in Apache’s documentation of the FTPClientConfig class (Matlab’s ftp uses a plain Apache FTPClient object, where you can also find an explanation of the enterLocalPassiveMode method that was used above).

Italy visit, Aug 26 – Sep 1, 2015

I will be traveling to north Italy between Aug 26 – Sep 1, 2015. If you happen to be in the area at that time, I will be happy to meet you to discuss how I could bring value to your work. Please email me (altmany at gmail) if you are interested.

Due to my travel, this blog will take a short summer vacation, and will return in early September. Stay tuned!

]]>
https://undocumentedmatlab.com/blog/solving-an-mput-ftp-hang-problem/feed 1
Using linkaxes vs. linkprophttps://undocumentedmatlab.com/blog/using-linkaxes-vs-linkprop https://undocumentedmatlab.com/blog/using-linkaxes-vs-linkprop#comments Wed, 22 Jul 2015 20:30:04 +0000 http://undocumentedmatlab.com/?p=5928
 
Related posts:
  1. Multi-column (grid) legend This article explains how to use undocumented axes listeners for implementing multi-column plot legends...
  2. Plot LimInclude properties The plot objects' XLimInclude, YLimInclude, ZLimInclude, ALimInclude and CLimInclude properties are an important feature, that has both functional and performance implications....
  3. FIG files format FIG files are actually MAT files in disguise. This article explains how this can be useful in Matlab applications....
  4. Handle Graphics Behavior HG behaviors are an important aspect of Matlab graphics that enable custom control of handle functionality. ...
 
]]>
One of my clients recently asked me to solve a very peculiar problem: He had several axes and was using Matlab’s builtin linkaxes function to link their axis limits. However, it didn’t behave quite the way that he expected. His axes were laid out as 2×2 subplots, and he wanted the two columns to be independently linked in the X axis, and the two rows to be independently linked in the Y axis:

% Prepare the axes
ax(1,1) = subplot(2,2,1); 
ax(1,2) = subplot(2,2,2); 
ax(2,1) = subplot(2,2,3); 
ax(2,2) = subplot(2,2,4);
 
% Plot something
x = 0 : 0.01 : 10;
line(x, sin(x),   'Parent',ax(1,1));
line(x, sin(2*x), 'Parent',ax(1,2));
line(x, cos(x),   'Parent',ax(2,1));
line(x, cos(5*x), 'Parent',ax(2,2));
 
% Link the relevant axes
linkaxes(ax(:,1),'x');  % left column
linkaxes(ax(:,2),'x');  % right column
linkaxes(ax(1,:),'y');  % top row
linkaxes(ax(2,:),'y');  % bottom row

The problem was that the plots didn’t behave as expected: when zooming in on the bottom-left axes, for example, only the bottom-right axes was updated (Y-limits synced), whereas the top-left axes’ X-limits remained unchanged:

Badly-synced axes limits

Badly-synced axes limits


Apparently, the second set of two linkaxes commands (to sync the rows’ Y-limits) overrode the first set of two linkaxes commands (to sync the columns’ X-limits).

Analysis

The reason for this unexpected behavior is that under the hood, linkaxes attaches property-change listeners to the corresponding axes, and stores these listeners in the axes’ hidden ApplicationData property (which is typically accessible via the getappdata / setappdata / isappdata / rmappdata set of functions). Specifically, up to a certain Matlab release (R2013b?), the listeners were placed in a field called ‘listener__’, and since then in a field called ‘graphics_linkaxes’. In either case, the field name was constant.

Therefore, when we placed the first set of linkaxes commands, the axes were correctly synced vertically (ax(1,1) with ax(2,1) in their X-limits, and similarly ax(1,2) with ax(2,2)). But when we placed the second set of linkaxes commands, the internal field in the axes’ ApplicationData property was overriden with the new listeners (that synced the rows’ Y-limits).

It so happens that Matlab listeners have a very nasty feature of being deleted when they are no longer referenced anywhere (within a workspace variable or object property). So when we overrode the first set of listener handles, we effectively deleted them, as if they were never set in the first place.

Some people may possibly complain about both issues at this point:

  • That Matlab listeners get deleted so easily without so much as a console warning, and certainly against naive intuition.
  • That repeated calls to linkaxes should override (rather than complement) each other.

As a side note, the addlistener function creates a listener and then persists it in the object’s hidden AutoListeners__ property. However, unlike the linkaxes behavior, addlistener‘s listener handles are always appended to AutoListeners__‘s contents, rather than replacing it. This ensures that all listeners are accessible and active until their container is deleted or they are specifically modified/removed. I wish that linkaxes used this mechanism, rather than its current ApplicationData one.

Workaround: linkprop

Luckily, there is a very easy and simple workaround, namely to use linkprop rather than linkaxes. The linkprop function is a lower-level function that creates a property-change listener that syncs corresponding properties in any specified array of object handles. In fact, linkaxes uses linkprop in order to create the necessary listeners. In our case, we can use linkprop directly, to independently attach such listeners to the axes’ XLim and YLim properties. We just need to ensure that all these listeners remain accessible to Matlab throughout the corresponding objects’ life-cycle. This is easily done using ApplicationData, as is done by linkaxes.m but in a smarter manner that does not override the previous values. The benefit of this is that when the axes are deleted, then so are the listeners; as long as the axes are accessible then so are the listeners. We just need to ensure that we don’t override these listener values:

setappdata(ax(1,1), 'YLim_listeners', linkprop(ax(1,:),'YLim')); 
setappdata(ax(2,1), 'YLim_listeners', linkprop(ax(2,:),'YLim'));
setappdata(ax(1,1), 'XLim_listeners', linkprop(ax(:,1),'XLim'));
setappdata(ax(1,2), 'XLim_listeners', linkprop(ax(:,2),'XLim'));

This results in the expected behavior:

properly-linked axes

properly-linked axes

Conclusions

The design decision by MathWorks to automatically delete Matlab listeners as soon as their reference count is zeroed and they get garbage-collected, causes a myriad of unexpected run-time behaviors, one of which is exemplified in today’s post on linkaxes. This would still have not caused any problem had the developers of linkaxes been aware of this listener feature and taken care to store the linked listener handles in an accumulating repository (e.g., adding the listener handle to an array of existing handles, rather than replacing a scalar handle).

Luckily, now that we know how Matlab listeners behave, we can easily identify abnormal behavior that results from listener handles going out of scope, and can easily take steps to persist the handles somewhere, so that they will remain active.

I wish to stress here that the listeners’ limited scope is fully documented in several places in the documentation (e.g., here as well as the linkprop doc page). The non-additive behavior of linkaxes is also documented, albeit in an almost-invisible footnote on its doc-page.

However, I humbly contend that the fact that these behaviors are documented doesn’t meant that they are correct. After all, figure windows or timers aren’t deleted when their handle goes out of scope, are they? At the very least, I hope that MathWorks will improve the relevant doc pages, to highlight these non-intuitive behaviors, and in the case of linkaxes to present a linkprop usage example as a workaround.

If you are interested in the topic of Matlab listeners, note that I’ve written quite a few listener-related posts over the years (about property-change listeners as well as event listeners). I urge you to take a look at the list of related articles presented below, or to use the search box at the top of the page.

]]>
https://undocumentedmatlab.com/blog/using-linkaxes-vs-linkprop/feed 5
Some performance-tuning tipshttps://undocumentedmatlab.com/blog/some-performance-tuning-tips https://undocumentedmatlab.com/blog/some-performance-tuning-tips#comments Wed, 17 Jun 2015 20:00:33 +0000 http://undocumentedmatlab.com/?p=5873
 
Related posts:
  1. Multi-line tooltips Multi-line tooltips are very easy to set up, once you know your way around a few undocumented hiccups....
  2. Solving a Matlab hang problem A very common Matlab hang is apparently due to an internal timing problem that can easily be solved. ...
  3. A couple of internal Matlab bugs and workarounds A couple of undocumented Matlab bugs have simple workarounds. ...
  4. Performance: accessing handle properties Handle object property access (get/set) performance can be significantly improved using dot-notation. ...
 
]]>
Today’s post is about performance. My goal is to show that contrary to widespread perception, Matlab is not inherently too slow to be used for real-life programs. In fact, by taking a small amount of time (compared to the overall dev time), Matlab programs can be accelerated by a large factor. I wish to demonstrate this claim with work that I recently completed for the Crustal Dynamics research group at Harvard University. They have created interactive Matlab GUIs for earthquake hazard, visualizing deformation and motion at plate boundary zones, recorded as GPS velocities:

Crustal dynamics visualization GUI

Crustal dynamics visualization GUI

These GUIs served them well for several years. But when they recently tried to analyze larger problems that involved far more data, it was getting so slow that it limited their ability to interrogate the GUI results effectively and do productive science (take a look at all the data points around New Zealand in the screenshot above). This is when I stepped in to help.

Two main performance issues stood out above the rest: Loading the data, and displaying the textual labels of numeric slip rates.

I/O: Loading data

I began with optimizing the data load time. This involves loading several different data files, which have custom textual formats. Of course, had the data files been initially created in some binary format, we could load it much faster. But we were faced with an existing situation where the textual data format was a given fact. Using Matlab’s profiler, it quickly emerged, as expected, that most of the time was spent parsing the text files. Two specific types of parsing were used, and they were both quite slow: reading the input files using textread, and parsing the input data using str2num.

To read the data from the files, I replaced the textread calls with corresponding textscan ones:

% Old (slow) code:
[Station.lon, Station.lat, Station.eastVel, Station.northVel, ...
 Station.eastSig, Station.northSig, Station.corr, ...
 Station.other1, Station.tog, Station.name] = textread(fileName, '%f%f%f%f%f%f%f%d%d%s');
 
% New (fast) code:
fid = fopen(fileName,'rt');
c = textscan(fid, '%f%f%f%f%f%f%f%d%d%s');
fclose(fid);
fn = {'lon', 'lat', 'eastVel', 'northVel', 'eastSig', 'northSig', 'corr', 'other1', 'tog', 'name'};
Station = cell2struct(c,fn,2);

To improve the performance of the str2num calls, I differentiated between two sub-cases:

In some cases, str2num was simply used to round numeric input data to a certain numeric precision. I improved this by changing the str2num calls with corresponding calls to round (which accepts an optional precision argument since R2014b):

% Old (slow) code
Station.lon = str2num(num2str(Station.lon, '%3.3f'));
 
% New (fast) code
Station.lon = round(Station.lon,3);        % R2014b or newer
Station.lon = round(Station.lon*1e3)/1e3;  % R2014a or older

In other cases, str2num was used to convert strings into numeric values. This should normally be done using a textscan parameter but in this specific case this was complicated due to the way the data was formatted, which required parsing iterative blocks. Still, converting strings into numbers is far faster using sscanf than str2num. The down side is that str2num also works in certain edge-cases where sscanf doesn’t. For this reason, I created a utility function (str2num_fast.m) that uses sscanf where possible, and falls back to str2num in case of problems. I then simply replaced all calls to str2num in the code to str2num_fast:

% str2num_fast - faster alternative to str2num
function data = str2num_fast(str, numCols)
    try
        % Fast code:
        str = char(str);
        str(:,end+1) = ' ';
        data = sscanf(str','%f');
        if nargin>1 && ~isempty(numCols)
            data = reshape(data,numCols,[])';
        end
    catch
        % This is much simpler but also much slower...
        data = str2num(str);
    end
end

The result: loading a medium-sized data set, which used to take 5-6 minutes (and much longer for larger data sets), now takes less than 1 second, a speedup of x500. When you continuously load and compare different data sets, it can mean the difference between a usable and an unusable program. Not bad for starters…

For many additional related techniques, read chapters 4 and 11 of my Accelerating MATLAB Performance book (string processing and I/O, respectively), or other performance-related articles on this website.

Displaying data

I now turned my attention to the graphic visualization aspects. As can be seen in the screenshot above, there are multiple layers of textual labels, arrows, lines and data points that can be added to the chart.

It turned out that in the interest of improved performance, the various checkboxes were designed such that they merely turned the visibility of the graphic components on and off. This does indeed improve performance in the specific use-case of checking and unchecking a specific checkbox. But in the general case, it significantly degrades performance by adding numerous graphic handles to the plot. By just checking 3 of these checkboxes (not all of them), I found that 37365 different graphic handles were created in the plot axes. That’s a HUGE number, and it’s no surprise that adding additional visualization layers, or zooming/panning the axes, became excruciatingly slow, even when the layers were turned off (i.e., made invisible). This is because Matlab’s internal graphics engine needs to manage all these handles, even when they are not visible.

The first rule of improving graphics performance is that except if the handles need to be frequently turned on/off, no graphic element should remain plotted if it is not visible. In our case, this meant that when a visualization layer’s checkbox is deselected, the corresponding handles are deleted, not made invisible (there is of course a throughput/latency tradeoff in the general case, between the recurring handle’s creation time and the performance impact of keeping numerous invisible handles):

% hCheckbox is the handle of the selected/deselected checkbox
% hPlotHandles is the list of corresponding plotted graphic handles
if get(hCheckbox, 'Value') == 0
   %set(hPlotHandles, 'Visible', 'off');  % Old (slow) code
   delete(hPlotHandles);  % Faster throughput in our use-case
else
   hPlotHandles = ...
end

A related aspect is that if the axes is zoomed-in (as is often the case in this specific GUI), then there is no need to plot any graphic element which is outside the axes limits:

% Old (slow) code:
text(lons, lats, labels);
 
% Much faster: limit the labels only to the visible axes area
hAxes = handle(gca);
validIdx = within(lons, hAxes.XLim) & within(lats, hAxes.YLim);
text(lons(validIdx), lats(validIdx), labels(validIdx,:));
 
function validIdx = within(data,limits)
    validIdx = data >= limits(1) & data <= limits(2);
end

Finally, in order to reduce the number of displayed graphic handles, we can unify the separate line segments into a single line that has NaN (or Inf) values interspaced between the segments. This is a very important technique, that enabled a reduction of ~7000 separate line handles into a single line, which improves both the line creation time and any subsequent axes action (e.g., zoom/pan). This is even faster than limiting the display to the axes limits (and yes, we could combine them by displaying a single line that has fewer data points that fit the axes limits, but the extra performance benefit would be negligible):

% Old (slow) code:
line([Segment.lon1'; Segment.lon2'], [Segment.lat1'; Segment.lat2']);
 
% Faster code: limit the display to the axes limits
hAxes = handle(gca);
lonLimits = hAxes.XLim;
latLimits = hAxes.YLim;
valid = (within(Segment.lon1,lonLimits) | within(Segment.lon2,lonLimits)) & ...
        (within(Segment.lat1,latLimits) | within(Segment.lat2,latLimits));
line([Segment.lon1(valid)', Segment.lon2(valid)'], ...
     [Segment.lat1(valid)', Segment.lat2(valid)']);
 
% New (fastest) code:
lons = [Segment.lon1'; Segment.lon2'; nan(1,numel(Segment.lon2))];
lats = [Segment.lat1'; Segment.lat2'; nan(1,numel(Segment.lat2))];
line(lons(:), lats(:));

The result: the time for displaying the slip-rate labels in the zoomed-in Australasia region in the screenshot above was reduced from 33 seconds to 0.6 secs; displaying the residual velocity vectors was reduced from 1.63 secs to 0.02 secs — speedups of x50-100. Again, not bad at all… The GUI is now fast enough to enable true interactivity. In Prof. Brendan Meade’s words:

[The GUI] is now tremendously fast with all arrow type visualizations! It’s amazing and enabling us to work with large scale data sets much more efficiently.​ Simply fantastic. Just awesome… It’s so fast on the load and showing the slip rates in zoom regions with large models almost instant! This is exactly what we were hoping for. This is really making a difference in terms of how fast we can do science!

The technique of converting multiple lines into a single line using NaNs was discussed last week by Mike Garrity. Users who are interested in additional ideas for improving Matlab graphics performance are encouraged to visit Mike’s blog. For many additional techniques, read chapter 10 of my Accelerating MATLAB Performance book, or other performance-related articles on this website.

Some other techniques for speeding up graphic objects creation that I’ve found useful over the years include:

  1. Avoid plotting non-visible elements, including elements that are outside the current axes limits, or have their Visible property set to ‘off’ or have a color that is the same as the axes background color.
  2. Avoid plotting overlapped elements, esp. those that are occluded by non-transparent patches, or lines having the same coordinates.
  3. Avoid using the scatter function with fewer than 100 data points – instead, duplicate these points so that scatter will work with more than 100 points, where vectorization kicks in (details), or even better: use line rather than scatter.
  4. Use low-level rather than high-level plotting functions – i.e., line instead of scatter/plot/plot3; surface instead of surf.
  5. Avoid creating straight line with multiple data points – instead, only keep the end-points for plotting such lines. I find that this is a very common use-case, which is often overlooked and could have a significant performance impact.
  6. Avoid using plot markers if possible, and use simple markers if this cannot be avoided. Various markers have different performance impacts in various situations, but ‘.’ and ‘o’ are typically faster than others.
  7. Use the plot function’s input triplets format, rather than multiple calls to plot. For example:
    plot(data1x,data1y,'r', data2x,data2y,'g', data3x,data3y,'b', ...);
  8. Set the axes properties to static values before plotting, in order to avoid run-time dynamic computation and update of things like the limits, tick-marks etc.
  9. Avoid creating legends or colorbars initially – let the user create them by clicking the corresponding toolbar icon if needed. Legends and colorbars take a second or more to create and can often be avoided in the initial display. If this cannot be avoided, then at least create static legends/colorbars.
  10. Only call drawnow once you’ve finished plotting. I’ve seen numerous cases where users call drawnow within the plotting loop and this has a horrendous performance impact. However, note that in some cases drawnow is very important (example1, example2).
  11. Generate the plot while the figure is hidden.
  12. Data-reduce the plotted data. We can program this ourselves, or use former MathWorker Tucker McClure’s reduce_plot utility (a POTW selection) to do it for us. Data reduction is especially important when displaying images that do not need to be zoomed-in.
  13. Cast image data to uint8 before using image or imagesc to display the image.
  14. Avoid clearing/deleting and then recreating the axes when we need to replot – instead, just delete the relevant axes children objects and replot in the existing axes.
  15. Avoid using the axes function to set the focus on a specific axes – instead, set the figure’s CurrentAxes property, or pass the axes handle directly to the plotting function.

Naturally, there are certain use-cases where applicative requirements might prevent using one or more of these techniques, but in the general case I find that following them improves performance.

Conclusions

Matlab is NOT inherently slow. It can be made to run much faster than many people assume, by simply using the built-in profiler tool, following several simple coding techniques and employing common sense. Too often I find that complains about Matlab’s speed stem from the fact that not even a minimal effort was invested in trying to follow these steps. The difference between an unoptimized and optimized Matlab code can be far larger in Matlab than in other programming languages, so Matlab users should invest more time optimizing their code than they would perhaps need to do in other programming environments. The potential benefits, as I’ve shown above and in my book, could be enormous.

MathWorks is constantly investing in making Matlab’s engine faster by default, and there is certainly room for improvement in certain aspects (e.g, OOP and HG2 performance). But there will always be room for human insight to optimize performance, and we should not neglect this. Moreover, we should not blame Matlab for our failure to invest even a minimal optimization effort. MathWorks cannot (of course) say this to their users, but I don’t have this limitation and I say it out loud: people should stop blaming MathWorks for everything. If you create a car with square wheels, don’t complain if it doesn’t drive as fast as you expect (even if its engine could indeed be improved). In this case, the customer is not always (or entirely) right.

Perhaps it’s just a matter of setting the user expectations straight: we do not expect Matlab to automatically solve our equations or generate the perfect engineering model. So too should we not expect Matlab to automatically run fast enough for our needs. Just as we expect to spend time to solve the scientific or engineering problem, so too should we spend a bit of time to optimize the code to run fast enough.

Luckily, there are numerous different ways in which we can improve Matlab’s performance. In fact, there are so many different ways to achieve our performance goals that we can take a pick based on aesthetic preferences and subjective experience: Some people use vectorization, others like parallelization, some others prefer to invest in smarter algorithms or faster hardware, others trade memory for performance or latency for throughput, still others display a GUI that just provides a faster impression with dynamic feedback. Even if one technique fails or is inapplicable, there are many other alternatives that we could try. Just use the profiler and some common sense and you are half-way there. Good luck!

Interested readers can find out more information about improving Matlab’s performance in my book “Accelerating MATLAB Performance” (CRC Press, 2014, ISBN 978-1482211290). If you already have this book, please be kind enough to post your feedback on it on Amazon (link), for the benefit of others.

]]>
https://undocumentedmatlab.com/blog/some-performance-tuning-tips/feed 4
copyobj behavior change in HG2https://undocumentedmatlab.com/blog/copyobj-behavior-change-in-hg2 https://undocumentedmatlab.com/blog/copyobj-behavior-change-in-hg2#respond Wed, 13 May 2015 16:00:49 +0000 http://undocumentedmatlab.com/?p=5797
 
Related posts:
  1. HG2 update HG2 appears to be nearing release. It is now a stable mature system. ...
  2. Performance: accessing handle properties Handle object property access (get/set) performance can be significantly improved using dot-notation. ...
  3. uicontextmenu performance Matlab uicontextmenus are not automatically deleted with their associated objects, leading to leaks and slow-downs. ...
  4. Graphic sizing in Matlab R2015b Matlab release R2015b's new "DPI-aware" nature broke some important functionality. Here's what can be done... ...
 
]]>
As a followup to last-week’s post on class-object and generic data copies, I would like to welcome back guest blogger Robert Cumming, who developed a commercial Matlab GUI framework. Today, Robert will highlight a behavior change of Matlab’s copyobj function in HG2.

One of the latest features that was introduced to the GUI Toolbox was the ability to undock or copy panels, that would be displayed in a standalone figure window, but remain connected to the underlying class object:

Panel copy in the GUI framework toolbox

Panel copy in the GUI framework toolbox

These panel copies had to remain fully functional, including all children and callbacks, and they needed to retain all connections back to the source data. In the example above I have altered the plot to show that it’s an actual copy of the data, but has separate behavior from the original panel.

To simply undock a uipanel to a new figure, we can simply re parent it by updating its Parent property to the new figure handle. To make a copy we need to utilize the copyobj function, rather than re-parenting. copyobj can be used to make a copy of all graphic objects that are “grouped” under a common parent, placing their copy in a new parent. In HG2 (R2014b onwards) the default operation of copyobj has changed.

When I started developing this feature everything looked okay and all the objects appeared copied. However, none of the callbacks were functional and all the information stored in the object’s ApplicationData was missing.

I had used copyobj in the past, so I knew that it originally worked ok, so I investigated what was happening. Matlab’s documentation for HG2 code transition suggests re-running the original code to create the second object to populate the callbacks. Unfortunately, this may not be suitable in all cases. Certainly in this case it would be much harder to do, than if the original callbacks had been copied directly. Another suggestion is to use the new ‘lagacy’ option’:

copyobj(___,’legacy’) copies object callback properties and object application data. This behavior is consistent with versions of copyobj before MATLAB® release R2014b.

So, instead of re-running the original code to create the second object to populate the callbacks, we can simply use the new ‘legacy’ option to copy all the callbacks and ApplicationData:

copyobj(hPanel, hNewParent, 'legacy')

Note: for some reason, this new ‘legacy’ option is mentioned in both the doc page and the above-mentioned HG2 code-transition page, but not in the often used help section (help copyobj). There is also no link to the relevant HG2 code-transition page in either the help section or the doc page. I find it unfortunate that for such a backward-incompatible behavior change, MathWorks has not seen fit to document the information more prominently.

Other things to note (this is probably not an exhaustive list…) when you are using copyobj:

  • Any event listeners won’t be copied
  • Any uicontextmenus will not be copied – it will in fact behave strangely due to the fact that it will have the uicontextmenu – but the parent is the original figure – and when you right-click on the object it will change the figure focus. For example:
    hFig= figure;
    ax = axes;
    uic = uicontextmenu ('parent', hFig);
    uim = uimenu('label','My Label', 'parent',uic);
    ax.UIContextMenu = uic;
     
    copyChildren = copyobj (ax, hFig, 'legacy');
     
    hFig2 = figure;
    copyChildren.Parent = hFig2;

Another note on undocked copies – you will need to manage your callbacks appropriately so that the callbacks manage whether they are being run by the original figure or in a new undocked figure.

Conclusions

  1. copyobj has changed in HG2 – but the “legacy” switch allows you to use it as before.
  2. It is unfortunate that backward compatibility was not fully preserved (nor documented enough) in HG2, but at least we have an escape hatch in this case.
  3. Take care with the legacy option as you may need to alter uicontextmenus and re-attach listeners as required.
]]>
https://undocumentedmatlab.com/blog/copyobj-behavior-change-in-hg2/feed 0
Undocumented view transformation matrixhttps://undocumentedmatlab.com/blog/undocumented-view-transformation-matrix https://undocumentedmatlab.com/blog/undocumented-view-transformation-matrix#comments Wed, 15 Apr 2015 21:21:51 +0000 http://undocumentedmatlab.com/?p=5711
 
Related posts:
  1. Multi-column (grid) legend This article explains how to use undocumented axes listeners for implementing multi-column plot legends...
  2. Draggable plot data-tips Matlab's standard plot data-tips can be customized to enable dragging, without being limitted to be adjacent to their data-point. ...
  3. Customizing axes part 3 – Backdrop Matlab HG2 axes can be customized in many different ways. This article explains some of the undocumented aspects. ...
  4. Customizing axes part 4 – additional properties Matlab HG2 axes can be customized in many different ways. This article explains some of the undocumented aspects. ...
 
]]>
Everyone knows Matlab’s view function, right? You know, the function that can set a 3D plot to the proper orientation angles and/or return the current plot’s azimuth/elevation angles. I’ve used it numerous times myself in the past two decades. It’s one of Matlab’s earliest functions, dating back to at least 1984. Still, as often as I’ve used it, it was not until I came across Bruce Elliott’s post on CSSM last week that I realized that this seamingly-innocent stock Matlab function holds a few interesting secrets.

view()’s transformation matrix output

First, while view‘s 2-output syntax ([az,el]=view()) is well known and documented, there is also a single-output syntax (T=view()) that is neither. To be exact, this syntax is not mentioned in the official documentation pages, but it does appear in the help section of view.m, which is viewable (no pun intended…) if you type the following in your Matlab console (R2014a or earlier, note the highlighted lines):

>> help view
 view   3-D graph viewpoint specification.
    view(AZ,EL) and view([AZ,EL]) set the angle of the view from which an
    observer sees the current 3-D plot.  AZ is the azimuth or horizontal
    rotation and EL is the vertical elevation (both in degrees). Azimuth
    revolves about the z-axis, with positive values indicating counter-
    clockwise rotation of the viewpoint. Positive values of elevation
    correspond to moving above the object; negative values move below.
    view([X Y Z]) sets the view angle in Cartesian coordinates. The
    magnitude of vector X,Y,Z is ignored.
 
    Here are some examples:
 
    AZ = -37.5, EL = 30 is the default 3-D view.
    AZ = 0, EL = 90 is directly overhead and the default 2-D view.
    AZ = EL = 0 looks directly up the first column of the matrix.
    AZ = 180 is behind the matrix.
 
    view(2) sets the default 2-D view, AZ = 0, EL = 90.
    view(3) sets the default 3-D view, AZ = -37.5, EL = 30.
 
    [AZ,EL] = view returns the current azimuth and elevation.
 
    T = view returns the current general 4-by-4 transformation matrix. 
    view(AX,...) uses axes AX instead of the current axes.
 
    See also viewmtx, the axes Properties view, Xform. 
    Reference page in Help browser
       doc view
 
>> surf(peaks); T=view
T =
      0.79335     -0.60876            0    -0.092296
      0.30438      0.39668      0.86603     -0.78354
       0.5272      0.68706         -0.5       8.3031
            0            0            0            1

Note that the extra highlighted information is probably a documentation oversight by some MathWorker many years ago, since it was removed from the help section in R2014b and does not appear in the doc pages (not even in R2014a). Perhaps it was documented in the early years but then someone for who-knows-what-reason decided that it shouldn’t be, and then forgot to remove all the loose ends until R2014b. Or maybe it was this way from the very beginning, I don’t know.

In any case, just to be clear on this, the transformation matrix out is still returned by view in the latest Matlab release (R2015a), just as it has for the past who-knows-how-many releases.

There are several interesting things to note here:

view()’s vs. viewmtx()’s transformation matrices

First, MathWorks have still not done a good job of removing all loose ends. Specifically, the T=view syntax is discussed in the doc page (and help section) of the viewmtx function.

To make things worse (and even more confusing), the usage example shown in that doc page is wrong: it says that view(az,el); T=view returns the same transformation matrix T as T=viewmtx(az,el). Close, but not the same:

>> view(30,60); T=view
T =
      0.86603          0.5            0     -0.68301
     -0.43301         0.75          0.5     -0.40849
        -0.25      0.43301     -0.86603       9.0018
            0            0            0            1
>> T2=viewmtx(30,60)
T2 =
      0.86603          0.5            0            0
     -0.43301         0.75          0.5            0
         0.25     -0.43301      0.86603            0
            0            0            0            1

Tough luck I guess for anyone who relies on viewmtx‘s output for complex 3D graphics…

T and T2 appear to be related via a transformation matrix (XT=[1,0,0,0; 0,1,0,0; 0,0,-1,0; 0,0,0,1], we’ll use it again below) that fixes the signs of the first 3 columns, and another translation matrix (camera viewpoint?) that provides the 4th column of T.

HG1’s undocumented axes transformation properties

Another tidbit that should never have been placed in view‘s help section in the first place, is the reference to the axes property Xform (read: “transform”, not “X-Form”). Xform is a hidden undocumented property, and as far as I can tell has always been this way. It is therefore surprising to see it mentioned in the official help section of a highly-visible function such as view. In fact, I don’t remember any other similar case.

In HG1 (R2014a and earlier), the axes’ Xform property held the transformation matrix that view returns. Alongside Xform, the HG1 axes contained several additional transformation vectors (x_RenderOffset, x_RenderScale) and matrices (x_NormRenderTransform, x_ProjectionTransform, x_RenderTransform, x_ViewPortTransform, x_ViewTransform – the latter (x_ViewTransform) is the same as Xform) that could be used for various purposes (example, technical details). All of these properties were removed in HG2 (R2014b or newer).

A complete usage example for some of these properties can be found in MathWorker Joe Conti’s select3d utility, which was removed from the File exchange, but can still be found online (note that it croacks on HG2=R2014b+ due to the removal of the hidden properties):

function [p] = local_Data2PixelTransform(ax,vert)
% Transform vertices from data space to pixel space.
 
% Get needed transforms
xform  = get(ax,'x_RenderTransform');
offset = get(ax,'x_RenderOffset');
scale  = get(ax,'x_RenderScale');
 
% Equivalent: nvert = vert/scale - offset;
nvert(:,1) = vert(:,1)./scale(1) - offset(1);
nvert(:,2) = vert(:,2)./scale(2) - offset(2);
nvert(:,3) = vert(:,3)./scale(3) - offset(3);
 
% Equivalent xvert = xform*xvert;
w = xform(4,1) * nvert(:,1) + xform(4,2) * nvert(:,2) + xform(4,3) * nvert(:,3) + xform(4,4);
xvert(:,1) = xform(1,1) * nvert(:,1) + xform(1,2) * nvert(:,2) + xform(1,3) * nvert(:,3) + xform(1,4);
xvert(:,2) = xform(2,1) * nvert(:,1) + xform(2,2) * nvert(:,2) + xform(2,3) * nvert(:,3) + xform(2,4);
 
% w may be 0 for perspective plots 
ind = find(w==0);
w(ind) = 1; % avoid divide by zero warning
xvert(ind,:) = 0; % set pixel to 0
 
p(:,1) = xvert(:,1) ./ w;
p(:,2) = xvert(:,2) ./ w;

We could even set these hidden properties directly, as Bruno Luong showed back in 2009 (the bug he reported in the R2009b prerelease was temporary, it still worked ok in R2014a):

set(gca,'Xform',eye(4))

HG2’s transformations

In HG2 (R2014b onward), we no longer have access to the hidden properties above. I’m still not exactly sure how to get all the transformations above, but at least the following can be used to replicate the transformation matrix T:

% "standard" way to get the transformation matrix
T = view;
 
% internal way
XT = [1,0,0,0; 0,1,0,0; 0,0,-1,0; 0,0,0,1];
hCamera = get(gca, 'Camera');
T = XT * GetViewMatrix(hCamera);

I’m guessing there are probably similar ways to get the other transformation matrices, but I’ll leave that as an exercise to the reader. Anyone who is up to the task is welcome to leave a comment below. Don’t come asking for my help here – I’m off to solve another puzzle. After all, there’s only a week left before my next blog post is due, so I better get started.

In summary, MathWorks have apparently done some cleanup for the new HG2 in R2014b, but I guess there’s still some work left to do (at least on the documentation). More importantly, much more work is needed to provide simple documented/supported ways of doing 3D transformations without banging our heads at all these hidden corners. Or maybe there already is such a way and I’m simply not aware of it, there’s always that possibility…

]]>
https://undocumentedmatlab.com/blog/undocumented-view-transformation-matrix/feed 5