Undocumented Matlab https://undocumentedmatlab.com Charting Matlab's unsupported hidden underbelly Fri, 20 Oct 2017 09:57:44 +0000 en-US hourly 1 https://wordpress.org/?v=4.4.1 Tips for accelerating Matlab performancehttps://undocumentedmatlab.com/blog/tips-for-accelerating-matlab-performance https://undocumentedmatlab.com/blog/tips-for-accelerating-matlab-performance#comments Thu, 05 Oct 2017 18:25:06 +0000 http://undocumentedmatlab.com/?p=7099
 
Related posts:
  1. Performance: scatter vs. line In many circumstances, the line function can generate visually-identical plots as the scatter function, much faster...
  2. Plot LimInclude properties The plot objects' XLimInclude, YLimInclude, ZLimInclude, ALimInclude and CLimInclude properties are an important feature, that has both functional and performance implications....
  3. Plot performance Undocumented inner plot mechanisms can significantly improve plotting performance ...
  4. Performance: accessing handle properties Handle object property access (get/set) performance can be significantly improved using dot-notation. ...
 
]]>
I’m proud to report that MathWorks has recently posted my article “Tips for Accelerating MATLAB Performance” in their latest newsletter digest (September 2017). This article is an updated and expanded version of my post about consulting work that I did for the Crustal Dynamics Research Group at Harvard University, where I helped speed-up a complex Matlab-based GUI by a factor of 50-500 (depending on the specific feature).

Crustal dynamics visualization GUI

Crustal dynamics visualization GUI

Featuring an article on the official newsletter by a non-MathWorker is rare. Doing this with someone like myself who has a reputation for undocumented aspects, and a consultancy business that potentially competes with theirs, is certainly not obvious. I take this to be a sign that despite the possible drawbacks of publishing my article, MathWorks felt that it provided enough value to the Matlab user community to merit the risk. I applaud MathWorks for this, and thank them for the opportunity of being featured in their official newsletter and conferences. I do not take it for granted in the least.

The newsletter article provides multiple ideas of improving the run-time performance for file I/O and graphics. Many additional techniques for improving Matlab’s performance can be found under the Performance tag in this blog, as well as in my book “Accelerating MATLAB Performance” (CRC Press, 2014, ISBN 978-1482211290).

Next week I will present live online webinars about various ways to improve Matlab’s run-time performance:

These live webinars will be 3.5 hours long, starting at 10am EDT (7am PDT, 3pm UK, 4pm CET, 7:30pm IST, time in your local timezone), with a short break in the middle. The presentations content will be based on onsite training courses that I presented at multiple client locations (details). A recording of the webinars will be available for anyone who cannot join the live events.

 Email me if you would like additional information on the webinars or my consulting, or to inquire regarding an onsite training course.

]]>
https://undocumentedmatlab.com/blog/tips-for-accelerating-matlab-performance/feed 2
Faster csvwrite/dlmwritehttps://undocumentedmatlab.com/blog/faster-csvwrite-dlmwrite https://undocumentedmatlab.com/blog/faster-csvwrite-dlmwrite#comments Tue, 03 Oct 2017 15:00:05 +0000 http://undocumentedmatlab.com/?p=7080
 
Related posts:
  1. Parsing mlint (Code Analyzer) output The Matlab Code Analyzer (mlint) has a lot of undocumented functionality just waiting to be used. ...
  2. Explicit multi-threading in Matlab part 3 Matlab performance can be improved by employing POSIX threads in C/C++ code. ...
  3. Plot LimInclude properties The plot objects' XLimInclude, YLimInclude, ZLimInclude, ALimInclude and CLimInclude properties are an important feature, that has both functional and performance implications....
  4. Preallocation performance Preallocation is a standard Matlab speedup technique. Still, it has several undocumented aspects. ...
 
]]>
Matlab’s builtin functions for exporting (saving) data to output files are quite sub-optimal (as in slowwwwww…). I wrote a few posts about this in the past (how to improve fwrite performance, and save performance). Today I extend the series by showing how we can improve the performance of delimited text output, for example comma-separated (CSV) or tab-separated (TSV/TXT) files.

The basic problem is that Matlab’s dlmwrite function, which can either be used directly, or via the csvwrite function which calls it internally, is extremely inefficient: It processes each input data value separately, in a non-vectorized loop. In the general (completely non-vectorized) case, each data value is separately converted into a string, and is separately sent to disk (using fprintf). In the specific case of real data values with simple delimiters and formatting, row values are vectorized, but in any case the rows are processed in a non-vectorized loop: A newline character is separately exported at the end of each row, using a separate fprintf call, and this has the effect of flushing the I/O to disk each and every row separately, which is of course disastrous for performance. The output file is indeed originally opened in buffered mode (as I explained in my fprintf performance post), but this only helps for outputs done within the row – the newline output at the end of each row forces an I/O flush regardless of how the file was opened. In general, when you read the short source-code of dlmwrite.m you’ll get the distinct feeling that it was written for correctness and maintainability, and some focus on performance (e.g., the vectorization edge-case). But much more could be done for performance it would seem.

This is where Alex Nazarovsky comes to the rescue.

Alex was so bothered by the slow performance of csvwrite and dlmwrite that he created a C++ (MEX) version that runs about enormously faster (30 times faster on my system). He explains the idea in his blog, and posted it as an open-source utility (mex-writematrix) on GitHub.

Usage of Alex’s utility is very easy:

mex_WriteMatrix(filename, dataMatrix, textFormat, delimiter, writeMode);

where the input arguments are:

  • filename – full path name for file to export
  • dataMatrix – matrix of numeric values to be exported
  • textFormat – format of output text (sprintf format), e.g. '%10.6f'
  • delimiter – delimiter, for example ',' or ';' or char(9) (=tab)
  • writeMode – 'w+' for rewriting file; 'a+' for appending (note the lowercase: uppercase will crash Matlab!)

Here is a sample run on my system, writing a simple CSV file containing 1K-by-1K data values (1M elements, ~12MB text files):

>> data = rand(1000, 1000);  % 1M data values, 8MB in memory, ~12MB on disk
 
>> tic, dlmwrite('temp1.csv', data, 'delimiter',',', 'precision','%10.10f'); toc
Elapsed time is 28.724937 seconds.
 
>> tic, mex_WriteMatrix('temp2.csv', data, '%10.10f', ',', 'w+'); toc   % 30 times faster!
Elapsed time is 0.957256 seconds.

Alex’s mex_WriteMatrix function is faster even in the edge case of simple formatting where dlmwrite uses vectorized mode (in that case, the file is exported in ~1.2 secs by dlmwrite and ~0.9 secs by mex_WriteMatrix, on my system).

Trapping Ctrl-C interrupts

Alex’s mex_WriteMatrix code includes another undocumented trick that could help anyone else who uses a long-running MEX function, namely the ability to stop the MEX execution using Ctrl-C. Using Ctrl-C is normally ignored in MEX code, but Wotao Yin showed how we can use the undocumented utIsInterruptPending() MEX function to monitor for user interrupts using Ctrl-C. For easy reference, here is a copy of Wotao Yin’s usage example (read his webpage for additional details):

/* A demo of Ctrl-C detection in mex-file by Wotao Yin. Jan 29, 2010. */
 
#include "mex.h"
 
#if defined (_WIN32)
    #include <windows.h>
#elif defined (__linux__)
    #include <unistd.h>
#endif
 
#ifdef __cplusplus 
    extern "C" bool utIsInterruptPending();
#else
    extern bool utIsInterruptPending();
#endif
 
void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[]) {
    int count = 0;    
    while(1) {
        #if defined(_WIN32)
            Sleep(1000);        /* Sleep one second */
        #elif defined(__linux__)
            usleep(1000*1000);  /* Sleep one second */
        #endif
 
        mexPrintf("Count = %d\n", count++);  /* print count and increase it by 1 */
        mexEvalString("drawnow;");           /* flush screen output */
 
        if (utIsInterruptPending()) {        /* check for a Ctrl-C event */
            mexPrintf("Ctrl-C Detected. END\n\n");
            return;
        }
        if (count == 10) {
            mexPrintf("Count Reached 10. END\n\n");
            return;
        }
    }
}

Matlab performance webinars

Next week I will present live online webinars about numerous other ways to improve Matlab’s run-time performance:

These live webinars will be 3.5 hours long, starting at 10am EDT (7am PDT, 3pm UK, 4pm CET, 7:30pm IST, time in your local timezone), with a short break in the middle. The presentations content will be based on onsite training courses that I presented at multiple client locations (details). A recording of the webinars will be available for anyone who cannot join the live events.

Additional Matlab performance tips can be found under the Performance tag in this blog, as well as in my book “Accelerating MATLAB Performance“.

 Email me if you would like additional information on the webinars, or an onsite training course, or about my consulting.

]]>
https://undocumentedmatlab.com/blog/faster-csvwrite-dlmwrite/feed 5
Runtime code instrumentationhttps://undocumentedmatlab.com/blog/runtime-code-instrumentation https://undocumentedmatlab.com/blog/runtime-code-instrumentation#comments Thu, 28 Sep 2017 13:36:17 +0000 http://undocumentedmatlab.com/?p=7063
 
Related posts:
  1. Creating a simple UDD class This article explains how to create and test custom UDD packages, classes and objects...
  2. Customizing print setup Matlab figures print-setup can be customized to automatically prepare the figure for printing in a specific configuration...
  3. UDD Events and Listeners UDD event listeners can be used to listen to property value changes and other important events of Matlab objects...
  4. Trapping warnings efficiently warnings can be trapped in a try-catch block just like regular errors. This articles explains how. ...
 
]]>
I regularly follow the MathWorks Pick-of-the-Week (POTW) blog. In a recent post, Jiro Doke highlighted Per Isakson’s tracer4m utility. Per is an accomplished Matlab programmer, who has a solid reputation in the Matlab user community for many years. His utility uses temporary conditional breakpoints to enable users to trace calls to their Matlab functions and class methods. This uses a little-known trick that I wish to highlight in this post.

tracer4m utility uses conditional breakpoints that evaluate but never become live

tracer4m utility uses conditional breakpoints that evaluate but never become live

Matlab breakpoints are documented and supported functionality, and yet their documented use is typically focused at interactive programming in the Matlab editor, or as interactive commands that are entered in the Matlab console using the set of db* functions: dbstop, dbclear, dbstatus, dbstack etc. However, nothing prevents us from using these db* functions directly within our code.

For example, the dbstack function can help us diagnose the calling tree for the current function, in order to do action A if one of the calling ancestors was FunctionX, or to do action B otherwise (for example, to avoid nested recursions).

Similarly, we could add a programmatic call to dbstop in order to stop at a certain code location downstream (for debugging), if a certain condition happens upstream.

Per extended this idea very cleverly in tracer4m: conditional breakpoints evaluate a string in run-time: if the result is true (non-zero) then the code run is stopped at that location, but if it’s false (or zero) then the code run continues normally. To instrument calls to specific functions, Per created a function tracer() that logs the function call (using dbstack) and always returns the value false. He then dynamically created a string that contains a call to this new function and used the dbstop function to create a conditional breakpoint based on this function, something similar to this:

dbstop('in', filename, 'at', location, 'if', 'tracer()');

We can use this same technique for other purposes. For example, if we want to do some action (not necessarily log – perhaps do something else) when a certain code point is reached. The benefit here is that we don’t need to modify the code at all – we’re adding ad-hoc code pieces using the conditional breakpoint mechanism without affecting the source code. This is particularly useful when we do not have access to the source code (such as when it’s compiled or write-protected). All you need to do is to ensure that the instrumentation function always returns false so that the breakpoint does not become live and for code execution to continue normally.

The tracer4m utility is quite sophisticated in the sense that it uses mlint and smart regexp to parse the code and know which functions/methods occur on which line numbers and have which type (more details). In this sense, Per used undocumented functionality. I’m certain that Jiro was not aware of the dependency on undocumented features when he posted about the utility, so please don’t take this to mean that Jiro or MathWorks officially support this or any other undocumented functionality. Undocumented aspects are often needed to achieve top functionality, and I’m happy that the POTW blog highlights utilities based on their importance and merit, even if they do happen to use some undocumented aspect.

tracer4m‘s code also contains references to the undocumented profiler option -history, but this is not in fact used by the code itself, only in comments. I use this feature in my profile_history utility, which displays the function call/timing history in an interactive GUI window. This utility complements tracer4m by providing a lot more information, but this can result in a huge amount of information for large and/or long-running programs. In addition, tracer4m has the benefit of only logging those functions/methods that the user finds useful, rather than all the function call, which enables easier debugging when the relevant code area is known. In short, I wish I had known about tracer4m when I created profile_history. Now that I know about it, maybe I’ll incorporate some of its ideas into profile_history in order to make it more useful. Perhaps another moral of this is that we should actively monitor the POTW blog, because true gems are quite often highlighted there.

Function call timeline profiling
Function call timeline profiling

For anyone who missed the announcement in my previous post, I’m hosting a series of live webinars on advanced Matlab topics in the upcoming 2 weeks – I’ll be happy if you would join.

]]>
https://undocumentedmatlab.com/blog/runtime-code-instrumentation/feed 3
Advanced Matlab online webinarshttps://undocumentedmatlab.com/blog/advanced-matlab-online-webinars https://undocumentedmatlab.com/blog/advanced-matlab-online-webinars#respond Thu, 07 Sep 2017 08:37:06 +0000 http://undocumentedmatlab.com/?p=7034
 
Related posts:
  1. Matlab GUI training seminars – Zurich, 29-30 August 2017 Advanced Matlab training courses on Matlab User Interfaces (GUI) will be presented in Zurich Switzerland on 29-30 August, 2017...
  2. Upcoming public Matlab presentations I will be presenting a professional pairs-trading and analysis program in two upcoming Matlab conferences in Tel Aviv and Munich. ...
  3. Adding a search box to figure toolbar An interactive search-box can easily be added to a Matlab figure toolbar for enhanced user experience. ...
  4. A few parfor tips The parfor (parallel for) loops can be made faster using a few simple tips. ...
 
]]>
Advanced Matlab training webinars at the comfort of your desk I have prepared the following online webinars on advanced Matlab topics I (click the webinar titles for a detailed description):

All webinars are highly technical, concise and to the point, making very effective use of your time. They are based on onsite training courses that I presented at multiple client locations (details).

These webinars could be a great way for you to improve your Matlab proficiency and efficiency. You will quickly learn how to produce higher quality, better looking, faster working, and more robust applications. Your effectiveness in writing Matlab programs will improve, saving you development time while improving the quality. And all this at the comfort and convenience of your office or home.

 Email me if you would like additional information or a group discount, or to inquire regarding an onsite training course, or for any other related query/suggestion.

]]>
https://undocumentedmatlab.com/blog/advanced-matlab-online-webinars/feed 0
Matlab GUI training seminars – Zurich, 29-30 August 2017https://undocumentedmatlab.com/blog/matlab-gui-training-seminars-zurich-29-30-august-2017 https://undocumentedmatlab.com/blog/matlab-gui-training-seminars-zurich-29-30-august-2017#respond Fri, 04 Aug 2017 09:37:52 +0000 http://undocumentedmatlab.com/?p=6992
 
Related posts:
  1. Customizing Matlab labels Matlab's text uicontrol is not very customizable, and does not support HTML or Tex formatting. This article shows how to display HTML labels in Matlab and some undocumented customizations...
  2. GUI integrated browser control A fully-capable browser component is included in Matlab and can easily be incorporated in regular Matlab GUI applications. This article shows how....
  3. FindJObj GUI – display container hierarchy The FindJObj utility can be used to present a GUI that displays a Matlab container's internal Java components, properties and callbacks....
  4. Inactive Control Tooltips & Event Chaining Inactive Matlab uicontrols cannot normally display their tooltips. This article shows how to do this with a combination of undocumented Matlab and Java hacks....
 
]]>
Advanced Matlab training, Zurich 29-30 August 2017
Advanced Matlab training courses/seminars will be presented by me (Yair) in Zürich, Switzerland on 29-30 August, 2017:

  • August 29 (full day) – Interactive Matlab GUI
  • August 30 (full day) – Advanced Matlab GUI

The seminars are targeted at Matlab users who wish to improve their program’s usability and professional appearance. Basic familiarity with the Matlab environment and coding/programming is assumed. The courses will present a mix of both documented and undocumented aspects, which is not available anywhere else. The curriculum is listed below.

This is a unique opportunity to enhance your Matlab coding skills and improve your program’s usability in a couple of days.

If you are interested in either or both of these seminars, please Email me (altmany at gmail dot com).

I can also schedule a dedicated visit to your location, for onsite Matlab training customized to your organization’s specific needs. Additional information can be found on my Training page.

Around the time of the training, I will be traveling to various locations around Switzerland. If you wish to meet me in person to discuss how I could bring value to your project, then please email me (altmany at gmail):

  • Geneva: Aug 22 – 27
  • Bern: Aug 27 – 28
  • Zürich: Aug 28 – 30
  • Stuttgart: Aug 30 – 31
  • Basel: Sep 1 – 3

 Email me


Interactive Matlab GUI – 29 August, 2017

  1. Introduction to Matlab Graphical User Interfaces (GUI)
    • Design principles and best practices
    • Matlab GUI alternatives
    • Typical evolution of Matlab GUI developers
  2. GUIDE – MATLAB’s GUI Design Editor
    • Using GUIDE to design a custom GUI
    • Available built-in MATLAB uicontrols
    • Customizing uicontrols
    • Important figure and uicontrol properties
    • GUIDE utility windows
    • The GUIDE-generated file-duo
  3. Customizing GUI appearance and behavior
    • Programmatic GUI creation and control
    • GUIDE vs. m-programming
    • Attaching callback functionality to GUI components
    • Sharing data between GUI components
    • The handles data struct
    • Using handle visibility
    • Position, size and units
    • Formatting GUI using HTML
  4. Uitable
    • Displaying data in a MATLAB GUI uitable
    • Controlling column data type
    • Customizing uitable appearance
    • Reading uitable data
    • Uitable callbacks
    • Additional customizations using Java
  5. Matlab’s new App Designer and web-based GUI
    • App Designer environment, widgets and code
    • The web-based future of Matlab GUI and assumed roadmap
    • App Designer vs. GUIDE – pros and cons comparison
  6. Performance and interactivity considerations
    • Speeding up the initial GUI generation
    • Improving GUI responsiveness
    • Actual vs. perceived performance
    • Continuous interface feedback
    • Avoiding common performance pitfalls
    • Tradeoff considerations

At the end of this seminar, you will have learned how to:

  • apply GUI design principles in Matlab
  • create simple Matlab GUIs
  • manipulate and customize graphs, images and GUI components
  • display Matlab data in a variety of GUI manners, including data tables
  • decide between using GUIDE, App Designer and/or programmatic GUI
  • understand tradeoffs in design and run-time performance
  • comprehend performance implications, to improve GUI speed and responsiveness

Advanced Matlab GUI – 30 August, 2017

  1. Advanced topics in Matlab GUI
    • GUI callback interrupts and re-entrancy
    • GUI units and resizing
    • Advanced HTML formatting
    • Using hidden (undocumented) properties
    • Listening to action and property-change events
    • Uitab, uitree, uiundo and other uitools
  2. Customizing the figure window
    • Creating and customizing the figure’s main menu
    • Creating and using context menus
    • Creating and customizing figure toolbars
  3. Using Java with Matlab GUI
    • Matlab and Java Swing
    • Integrating Java controls in Matlab GUI
    • Handling Java events as Matlab callbacks
    • Integrating built-in Matlab controls/widgets
    • Integrating JIDE’s advanced features and professional controls
    • Integrating 3rd-party Java components: charts/graphs/widgets/reports
  4. Advanced Matlab-Java GUI
    • Customizing standard Matlab uicontrols
    • Figure-level customization (maximize/minimize, disable etc.)
    • Containers and position – Matlab vs. Java
    • Compatibility aspects and trade-offs
    • Safe programming with Java in Matlab
    • Java’s EDT and timing considerations
    • Deployment (compiler) aspects

At the end of this seminar, you will have learned how to:

  • customize the figure toolbar and main menu
  • use HTML to format GUI appearance
  • integrate Java controls in Matlab GUI
  • customize your Matlab GUI to a degree that you never knew was possible
  • create a modern-looking professional GUI in Matlab
]]>
https://undocumentedmatlab.com/blog/matlab-gui-training-seminars-zurich-29-30-august-2017/feed 0
Sending HTML emails from Matlabhttps://undocumentedmatlab.com/blog/sending-html-emails-from-matlab https://undocumentedmatlab.com/blog/sending-html-emails-from-matlab#respond Wed, 02 Aug 2017 21:19:42 +0000 http://undocumentedmatlab.com/?p=6986
 
Related posts:
  1. Uitable sorting Matlab's uitables can be sortable using simple undocumented features...
  2. Uitable customization report Matlab's uitable can be customized in many different ways. A detailed report explains how. ...
  3. Types of undocumented Matlab aspects This article lists the different types of undocumented/unsupported/hidden aspects in Matlab...
  4. Multi-line uitable column headers Matlab uitables can present long column headers in multiple lines, for improved readability. ...
 
]]>
A few months ago I wrote about various tricks for sending email/text messages from Matlab. Unfortunately, Matlab only sends text emails by default and provides no documented way to send HTML-formatted emails. Text-only emails are naturally very bland and all mail clients in the past 2 decades support HTML-formatted emails. Today I will show how we can send such HTML emails from Matlab.

A quick recap: Matlab’s sendmail function uses Java (specifically, the standard javax.mail package) to prepare and send emails. The Java classes are extremely powerful and so there is no wonder that Mathworks chose to use them rather than reinventing the wheel. However, Matlab’s sendmail function only uses part of the functionality exposed by these classes (admittedly, the most important parts that deal with the basic mail-sending mechanism), and does not expose external hooks or input args that would enable the user to take full advantage of the more advanced features, HTML formatting included.

Only two small changes are needed in sendmail.m to support HTML formatting:

  1. HTML formatting required calling the message-object’s setContent() method, rather than setText().
  2. We need to specify 'text/html' as part of the message’s encoding

To implement these features, change the following (lines #119-130 in the original sendmail.m file of R2017a, changed lines highlighted):

% Construct the body of the message and attachments.
body = formatText(theMessage);
if numel(attachments) == 0    if ~isempty(charset)        msg.setText(body, charset);
    else
        msg.setText(body);
    end
else
    % Add body text.
    messageBodyPart = MimeBodyPart;
    if ~isempty(charset)        messageBodyPart.setText(body, charset);
    ...

to this (changed lines highlighted):

% Construct the body of the message and attachments.
body = formatText(theMessage);
isHtml = ~isempty(body) && body(1) == '<';  % msg starting with '<' indicates HTMLif isHtml    if isempty(charset)        charset = 'text/html; charset=utf-8';    else        charset = ['text/html; charset=' charset];    endendif numel(attachments) == 0  && ~isHtml    if isHtml        msg.setContent(body, charset);    elseif ~isempty(charset)        msg.setText(body, charset);
    else
        msg.setText(body);
    end
    else
        % Add body text.
        messageBodyPart = MimeBodyPart;
        if isHtml            messageBodyPart.setContent(body, charset);        elseif ~isempty(charset)            messageBodyPart.setText(body, charset);
        ...

In addition, I also found it useful to remove the hard-coded 75-character line-wrapping in text messages. This can be done by changing the following (line #291 in the original sendmail.m file of R2017a):

maxLineLength = 75;

to this:

maxLineLength = inf;  % or some other large numeric value

Deployment

It’s useful to note two alternatives for making these fixes:

  • Making the changes directly in %matlabroot%/toolbox/matlab/iofun/sendmail.m. You will need administrator rights to edit this file. You will also need to redo the fix whenever you install Matlab, either installation on a different machine, or installing a new Matlab release. In general, I discourage changing Matlab’s internal files because it is simply not very maintainable.
  • Copying %matlabroot%/toolbox/matlab/iofun/sendmail.m into a dedicated wrapper function (e.g., sendEmail.m) that has a similar function signature and exists on the Matlab path. This has the benefit of working on multiple Matlab releases, and being copied along with the rest of our m-files when we install our Matlab program on a different computer. The downside is that our wrapper function will be stuck with the version of sendmail.m that we copied into it, and we’d lose any possible improvements that Mathworks may implement in future Matlab releases.

The basic idea for the second alternative, the sendEmail.m wrapper, is something like this (the top highlighted lines are the additions made to the original sendmail.m, with everything placed in sendEmail.m on the Matlab path):

function sendEmail(to,subject,theMessage,attachments)%SENDEMAIL Send e-mail wrapper (with HTML formatting)   sendmail(to,subject,theMessage,attachments); 
% The rest of this file is copied from %matlabroot%/toolbox/matlab/iofun/sendmail.m (with the modifications mentioned above):
function sendmail(to,subject,theMessage,attachments)
%SENDMAIL Send e-mail.
%   SENDMAIL(TO,SUBJECT,MESSAGE,ATTACHMENTS) sends an e-mail.  TO is either a
%   character vector specifying a single address, or a cell array of character vector
...

We would then call the wrapper function as follows:

sendEmail('abc@gmail.com', 'email subject', 'regular text message');     % will send a regular text message
sendEmail('abc@gmail.com', 'email subject', '<b><font color="blue">HTML-formatted</font> <i>message');  % HTML-formatted message

In this case, the code automatically infers HTML formatting based on whether the first character in the message body is a ‘<‘ character. Instead, we could just as easily have passed an additional input argument (isHtml) to our sendEmail wrapper function.

Hopefully, in some future Matlab release Mathworks will be kind enough to enable sending 21st-century HTML-formatted emails without needing such hacks. Until then, note that sendmail.m relies on standard non-GUI Java networking classes, which are expected to be supported far into the future, well after Java-based GUI may cease to be supported in Matlab. For this reason I believe that while it seems a bit tricky, the changes that I outlined in today’s post actually have a low risk of breaking in a future Matlab release.

Do you have some other advanced email feature that you use in your Matlab program by some crafty customization to sendmail? If so, please share it in a comment below.

]]>
https://undocumentedmatlab.com/blog/sending-html-emails-from-matlab/feed 0
User-defined tab completions – take 2https://undocumentedmatlab.com/blog/user-defined-tab-completions-take-2 https://undocumentedmatlab.com/blog/user-defined-tab-completions-take-2#comments Wed, 12 Jul 2017 13:00:30 +0000 http://undocumentedmatlab.com/?p=6961
 
Related posts:
  1. Setting desktop tab completions The Matlab desktop's Command-Window tab-completion can be customized for user-defined functions...
  2. Class object tab completion & improper field names Tab completions and property access can be customized for user-created Matlab classes. ...
  3. Recovering previous editor state Recovering the previous state of the Matlab editor and its loaded documents is possible using a built-in backup config file. ...
  4. setPrompt – Setting the Matlab Desktop prompt The Matlab Desktop's Command-Window prompt can easily be modified using some undocumented features...
 
]]>
Back in 2010, I posted about Matlab’s undocumented mechanism for setting Matlab desktop tab-completions. That mechanism used a couple of internal files (TC.xml and TC.xsd) to describe the various possible options that are auto-completed (or displayed in a small tooltip window) when the user clicks the <Tab> key on partially-entered function input parameters.

Using TabComplete for user-defined functions

Using TabComplete for user-defined functions

Unfortunately, this mechanism apparently broke in R2016a and was replaced with a new mechanism, as explained below.

The new mechanism relies on a file called functionSignatures.json which exists in every single folder that contains Matlab files that have functions whose input parameters ought to be tab-completable.

The new mechanism offers far greater versatility and flexability in defining the input types and inter-relationsships compared to the old TC.*-based mechanism. Another important benefit is that we can now add custom user-defined functionSignatures.json files to our user folders, next to our m-files, without having to modify any Matlab system file.

Note that you may need to restart Matlab the first time that you create a functionSignatures.json file. But once it’s created, you can modify it within a Matlab session and the changes take effect immediately.

Note: Credit for first posting about this new mechanism goes to Nicholas Mati. I’ve known about this new mechanism for over a year, but I never found the time to write about it until now, so Nicholas gets credit for breaking the scoop. The discussion below uses and expands Nicholas’ original post.

Syntax

The functionSignatures.json file has the general form:

{
	"FunctionName1":
	{
		"key1":"val1",
		"key2":"val2",
		"keyn":"valn"
	},
	"FunctionName2":
	{
		"key1":"val1",
		"key2":"val2",
		"keyn":"valn"
	}
}

A number of keys are supported including “platform“, “setsAns“, “inputs“, and “outputs“, although inputs and outputs are by far the most common (and presumably only inputs are relevant for tab-completion). These keys take an array of (or a single) object value(s). The objects typically take one of the following forms:

{"name":"variable_name", "kind":"kind_option", "type":"string_or_array_of_type"}
{"mutuallyExclusiveGroup":
	[
		...
	]
}
{"name":"varargin", "kind":"optional", "multiplicity":"append"}

The value for “kind” can be “required”, “optional”, “positional”, “flag”, “namevalue” or “platform” (and perhaps a few other lesser-used kinds):

  • required” means that the specified input is mandatory
  • optional” means that it can be added or omitted
  • positional” means that it’s an optional input but if it is specified then it must appear at the specified position relative to the previous (earlier) inputs
  • flag” means that it’s an optional input flag, from a predefined list of one or more single-token strings. For example, in regexp(s1,s2,'once') the last input arg ('once') is such a flag.
  • namevalue” means that it follows Matlab’s standard practice of using P-V pairs (parameter name followed by its value). For example, func('propName',propValue)
  • platform” indicates that this input is only available on the specified platform(s)

These “kind”s are all explained below.

The value for “type” can be a string such as “char” or “numeric” or “filepath”, or a more complicated JSON array (see below).

In addition to “name”, “kind” and “type”, we can also define a “default” value (e.g. "default":"false") and a “display” string. While these are [currently] not used by Desktop tab-completion, they might be used by other components such as the JIT compiler or the Editor, if not today then perhaps in a future release.

Note that while pure JSON format does not accept comments, Matlab’s functionSignatures.json does accept C++-style comments, as discovered by Heiko in a comment below. To add a comment, simply add // comment text at the end of any line, or /* comment text */ anywhere within a line.

Usage examples

Multiple examples of functionSignatures.json files can be found in subfolders of %matlabroot%/toolbox/matlab. For example, here’s the tab-completion definition for the visdiff function, which displays a visual comparison between two files, and resides in %matlabroot%/toolbox/shared/comparisons/functionSignatures.json:

{
"visdiff":
{
    "inputs":
    [
        {"name":"filename1", "kind":"required",   "type":"filepath"},
        {"name":"filename2", "kind":"required",   "type":"filepath"},
        {"name":"type",      "kind":"positional", "type":"choices={'text', 'binary'}"}
    ]
}
}

As can be seen in this example, the first and second inputs are expected to be a filename, whereas the third input is one of the two predefined strings ‘text’ or ‘binary’. This third input has “kind”:”positional”, meaning that it is optional, but if it is provided then it must be in the 3rd position and cannot appear sooner. Moreover, if the user specifies any input argument to the “right” of a positional input, then the positional argument becomes required, not optional.

Whereas a “positional” parameter has a specific position in the args list (#3 in the case of visdiff above), an “optional” parameter may appear anywhere in the list of inputs.

Here’s a more complex example, for the built-in regexprep function (in %matlabroot%/toolbox/matlab/strfun/functionSignatures.json). This example shows how to limit the input to certain data types and how to specify optional input flags with pre-defined choices:

"regexprep":
{
	"inputs":
	[
		{"name":"str",               "kind":"required",  "type":[["char"], ["cell"], ["string"]]},
		{"name":"expression",        "kind":"required",  "type":[["char"], ["cell"], ["string"]]},
		{"name":"replace",           "kind":"required",  "type":[["char"], ["cell"], ["string"]]},
		{"name":"optMatch",          "kind":"flag",      "display":"", "type":[["char", "choices={'all','once'}"], ["numeric", "scalar"]],   "default":"'all'"},
		{"name":"optWarnings",       "kind":"flag",      "display":"", "type":["char", "choices={'nowarnings','warnings'}"],                 "default":"'nowarnings'"},
		{"name":"optCase",           "kind":"flag",      "display":"", "type":["char", "choices={'matchcase','ignorecase','preservecase'}"], "default":"'matchcase'"},
		{"name":"optEmptyMatch",     "kind":"flag",      "display":"", "type":["char", "choices={'noemptymatch','emptymatch'}"],             "default":"'noemptymatch'"},
		{"name":"optDotAll",         "kind":"flag",      "display":"", "type":["char", "choices={'dotall','dotexceptnewline'}"],             "default":"'dotall'"},
		{"name":"optStringAnchors",  "kind":"flag",      "display":"", "type":["char", "choices={'stringanchors','lineanchors'}"],           "default":"'stringanchors'"},
		{"name":"optSpacing",        "kind":"flag",      "display":"", "type":["char", "choices={'literalspacing','freespacing'}"],          "default":"'literalspacing'"}
	],
	"outputs":
	[
		{"name":"newStr", "type":[["char"], ["cell"], ["string"]]}
	]
},

Here’s an even more complex example, this time for the codegen function (in %matlabroot%/toolbox/coder/matlabcoder/functionSignatures.json, part of the Matlab Coder toolbox). This example shows how to limit the filenames to certain extensions and how to specify name-value input pairs:

"codegen":
{
	"inputs":
	[
		{"name":"compile_only",  "kind":"flag",       "type":"choices={'-c'}"},
		{"name":"config_flag",   "kind":"flag",       "type":"choices={'-config:mex','-config:lib','-config:dll','-config:exe','-config:hdl'}"},
		{"name":"debug",         "kind":"flag",       "type":"choices={'-g'}"},
		{"name":"report",        "kind":"flag",       "type":"choices={'-report'}"},
		{"name":"launchreport",  "kind":"flag",       "type":"choices={'-launchreport'}"},
		{"name":"file",          "kind":"flag",       "type":"filepath=*.m,*.mlx,*.c,*.cpp,*.h,*.o,*.obj,*.a,*.so,*.lib,*.tmf", "multiplicity":"append"},
		{"name":"-d",            "kind":"namevalue",  "type":"folderpath"},
		{"name":"-I",            "kind":"namevalue",  "type":"folderpath"},
		{"name":"-globals",      "kind":"namevalue"},
		{"name":"-o",            "kind":"namevalue",  "type":[["char"], ["filepath"]]},
		{"name":"-O",            "kind":"namevalue",  "type":"choices={'enable:inline','disable:inline','enable:blas','disable:blas','enable:openmp','disable:openmp'}"},
		{"name":"-args",         "kind":"namevalue",  "type":[["identifier=variable"], ["char"]]},
		{"name":"-config",       "kind":"namevalue",  "type":[["identifier=variable"], ["char"]]},
		{"name":"verbose",       "kind":"flag",       "type":"choices={'-v'}"},
		{"name":"singleC",       "kind":"flag",       "type":"choices={'-singleC'}"},
		{"name":"-test",         "kind":"namevalue",  "type":"identifier=function"}
	]
},

Argument types

As noted above, we use "type":... to specify the expected data type of each parameter. This can be a simple string such as “char”, “cellstr”, “numeric”, “table”, “categorical”, “filepath”, “folderpath”, “matlabpath”, class name, or a more complicated JSON array. For example:

  • "type":["numeric","scalar"]
  • "type":["numeric","numel=3",">=4"]
  • "type":[["char"], ["cellstr"], ["numeric"], ["logical","vector"]]
  • "type":[["char", "choices={'-ascii'}"]]
  • "type":[["filepath"], ["matlabpath=*.m,*.mlx"], ["char"]]
  • "type":"identifier=variable,function,localfunction,package,classdef"
  • "type":"matlab.graphics.axis.Axes"
  • "type":"choices={'yes','no','maybe'}"

We can even specify on-the-fly Matlab computation that returns a cell-array of values, for example a list of available fonts via "type":"choices=listfonts". A more complex example is the definition of the rmfield function, where the possible input choices for the second input arg (highlighted) depend on the struct that is provided in the first input arg (by running the fieldnames function on it):

"rmfield":
{
	"inputs":
	[
		{"name":"s",     "kind":"required", "type":"struct"},
		{"name":"field", "kind":"required", "type":"choices=fieldnames(s)"}	],
	"outputs":
	[
		{"name":"s", "type":"struct"}
	]
},

Alternative inputs

Multiple alternative inputs can be specified in the functionSignatures.json file. The easiest way to do so is to simply create multiple different definitions for the same function, one beneath the other. Matlab’s tab-completion parser is smart enough to combine those definitions and proceed with the most appropriate one based on the user-entered inputs.

For example, in the same Coder file above we find 6 alternative definitions. If (for example) we start typing coder('-ecoder', and click <Tab>, Matlab would automatically auto-complete the second input to “false”, and then another <Tab> click would set the third input to the required ‘-new’ parameter (see highlighted lines below):

...
"coder":
{
	"inputs":
	[
		{"name":"projectname", "kind":"required", "type":"filepath=*.prj"}
	]
},
"coder":
{
	"inputs":
	[
		{"name":"-open", "kind":"namevalue", "type":"filepath=*.prj"}
	]
},
"coder":
{
	"inputs":
	[
		{"name":"-build", "kind":"namevalue", "type":"filepath=*.prj"}
	]
},
"coder":
{
	"inputs":
	[
		{"name":"-new", "kind":"namevalue", "type":[["filepath=*.prj"], ["char"]]}
	]
},
"coder":
{
	"inputs":
	[
		{"name":"ecoderFlag",  "kind":"required", "type":"choices={'-ecoder'}"},		{"name":"ecoderValue", "kind":"required", "type":[["logical"], ["choices={'false'}"]]},		{"name":"newFlag",     "kind":"required", "type":"choices={'-new'}"},		{"name":"newvalue",    "kind":"required", "type":[["filepath=*.prj"], ["char"]]}	]
},
"coder":
{
	"inputs":
	[
		{"name":"tocodeFlag",  "kind":"required", "type":"choices={'-tocode'}"},
		{"name":"tocodevalue", "kind":"required", "type":"filepath=*.prj"},
		{"mutuallyExclusiveGroup":
			[
				[],
				[
					{"name":"scriptFlag", "kind":"required", "type":"choices={'-script'}"},
					{"name":"scriptname", "kind":"required", "type":[["filepath=*.m"], ["char"]]}
				]
			]
		}
	]
}

This example also shows, in the last definition for the coder function, another mechanism for specifying alternative inputs, using “mutuallyExclusiveGroup” (aka “MEGs”). A MEG is defined using an array of options, enclosed in square brackets ([]). Each of the MEG options is exclusive to each of the others, meaning that we can only work with one of them and not the others. This is equivalent to duplicating the definition as we saw above, and saves us some copy-paste (in some cases a lot of copy-pastes, especially with multiple and/or nested MEGs). However, MEGs have a major drawback of reduced readability. I believe that in most cases we only have a single MEG and few input args, and in such cases it makes more sense to use repeated function defs rather than a MEG. The Matlab signature files contain numerous usage examples for either of these two mechanisms.

Platform dependencies

If a specific function (or a specific signature variant) depends on the running platform, this can be specified via the “platform” directive. For example, the winopen function only works on Windows, but not on Linux/Mac. Its corresponding signature definition is:

"winopen":
{
	"platform":"win32,win64",	"inputs":
	[
		{"name":"filename", "kind":"required", "type":"filepath"},
		{"name":"varargin", "kind":"optional", "multiplicity":"append"}
	]
}

Platform dependence could also be specified at the parameter level, not just the entire function level. For example, in the xlsread function (defined in %matlabroot%/toolbox/matlab/iofun/functionSignatures.json), we see that the usage variant xlsread(filename,-1) is only available on Windows (note that the numeric value is defined as "<=-1", not necessarily -1), and so is the “functionHandle” input (which is called processFcn in the documentation – for some reason that escapes me the names of many input args do not match in the documentation and functionSignature):

"xlsread":
{
	"inputs":
	[
		{"name":"filename", "kind":"required", "type":"filepath=*.xls,*.xlsx,*.xlsb,*.csv"},
		{"mutuallyExclusiveGroup":
			[
				{"name":"openExcel", "kind":"required", "display":"", "type":["numeric", "<=-1"], "platform":"win64,win32"},				{"name":"xlRange",   "kind":"required", "type":["char", "@(x) isempty(x) || ~isempty(strfind(x, ':'))"], "default":"''"},
				[
					{"name":"sheet",          "kind":"positional", "type":[["char", "choices=matlab.internal.language.introspective.tabcompletion.xlsread_vsheet(filename)"], ["numeric", ">=1"]], "default":"1"},
					{"name":"xlRange",        "kind":"positional", "type":"char", "default":"''"},
					{"name":"basic",          "kind":"positional", "display":"", "type":["char", "choices={'basic',''}"]},
					{"name":"functionHandle", "kind":"positional", "type":"function_handle", "platform":"win64,win32"}				]
			]
		}
	],
        ...

Parsing errors

The new mechanism is not very user-friendly when you get something wrong. In the best case, it issues a cryptic error message (see below), and in the worst case it simply ignores the changes and the user has no idea why the new custom tab-completion is not working as intended.

The most likely causes of such problems are:

  • The most common problem is that you placed the functionSignatures.json file in a different folder than the Matlab function. For example, if the myFunction() function is defined in myFunction.m, then the tab-completion of this function MUST be located in a functionSignatures.json file that resides in the same folder, not anywhere else on the Matlab path. In other words, the Matlab path is NOT relevant for tab-completion.
  • Your functionSignatures.json file does not follow the [extremely strict] syntax rules above, to the letter. For example, forgetting the top or final curly braces, forgetting a comma or adding an extra one, or not closing all brackets/braces properly.
  • You mistyped one or more of the input parameters, types or options.

In case of a parsing error, you’d see a red error message on the Matlab console the next time that you try to use tab-completion:

Error parsing JSON data; Boost reports "(189): expected ',' or ']'".

Unfortunately the error message only tells us the problematic line location within the functionSignatures.json file, but not the file’s location, so if we haven’t recently edited this file we’d need to find it in the relevant folder. For example:

edit(fullfile(fileparts(which('myFunction')), 'functionSignatures.json')

Moreover, when a JSON syntax error (such as the one above) occurs, the entire file is not parsed, not just the definition that caused the error.

Another limitation of tab-completion is that it does not work while the main Matlab thread is working (e.g., during a uiwait or waitfor). This may be somewhat misleading since most editor/debugging actions do work.

Arguably, this new tab-completion mechanism could be made more programmer-friendly. Perhaps this will improve in a future Matlab release.

For a related mechanism, see my article on tab-completion for class properties and methods from 2014, which is apparently still relevant and functional.

]]>
https://undocumentedmatlab.com/blog/user-defined-tab-completions-take-2/feed 5
Matlab Expo – Bern, 22 June 2017https://undocumentedmatlab.com/blog/matlab-expo-bern-22-june-2017 https://undocumentedmatlab.com/blog/matlab-expo-bern-22-june-2017#comments Sun, 11 Jun 2017 13:26:55 +0000 http://undocumentedmatlab.com/?p=6927
 
Related posts:
  1. Upcoming public Matlab presentations I will be presenting a professional pairs-trading and analysis program in two upcoming Matlab conferences in Tel Aviv and Munich. ...
  2. Adding a search box to figure toolbar An interactive search-box can easily be added to a Matlab figure toolbar for enhanced user experience. ...
  3. A few parfor tips The parfor (parallel for) loops can be made faster using a few simple tips. ...
  4. Matlab GUI training seminars – Zurich, 29-30 August 2017 Advanced Matlab training courses on Matlab User Interfaces (GUI) will be presented in Zurich Switzerland on 29-30 August, 2017...
 
]]>
Matlab Expo Bern - 22 June, 2017
Munich Germany Expo video, 10 May, 2016
My Matlab Expo 2016 keynote presentation (32:45)
(Matlab Expo 2017 presentation will be different)

MathWorks were very kind to invite me to speak at the upcoming annual Matlab Expo in Bern, Switzerland, on June 22, 2017 at 15:30. My presentation will be about “MATLAB Tricks You Need to Know“.

I also presented at last year’s Expo in Munich (you can see the video on the right). So in order not to bore the audience, my presentation this year will be completely different – it will not focus on any single program or industry, but instead provide content that should be relevant to a large portion of Matlab users.

My presentation will highlight several simple-to-use tips and tricks that can improve Matlab program usability and performance, and Matlab programming productivity in general. My aim is to show that Matlab can be used to create professional-quality applications, without sacrificing Matlab’s benefits (RAD, functionality, reliability), and that Matlab is certainly relevant for serious user-facing applications, not just for prototyping and internal organizational use.

I am targeting the presentation at anyone who uses Matlab, with any level of experience. Many of the tricks will be easy enough to use that even novice users could benefit, and some tricks might be useful even to advanced users. All these tricks are simple to understand, and yet very effective for improving run-time performance and visualization quality.

Participation in the Bern Expo is free, please don’t hesitate to come. If you’re considering it, then you might also be interested in my Advanced Matlab seminars in Zurich earlier that same week, on June 19-20.

If you are in the area and wish to meet me to discuss how I could bring value to your work, then please email me (altmany at gmail) to coordinate a meeting. We could meet either at the Expo, or in a dedicated (private) meeting.

Update June 23, 2017: I am extremely disappointed to report that my presentation at the Matlab Expo in Bern yesterday was not video-recorded. I thought that it went quite well so this makes me very sad. Anyway, you can see my presentation slides here. It doesn’t contain all the explanations and extra details that I communicated verbally, but I think that it might still be useful as-is. I hope you find it beneficial!

Matlab Expo Bern - 22 June, 2017 Matlab Expo Bern - 22 June, 2017

]]>
https://undocumentedmatlab.com/blog/matlab-expo-bern-22-june-2017/feed 3
Matlab compilation quirks – take 2https://undocumentedmatlab.com/blog/matlab-compilation-quirks-take-2 https://undocumentedmatlab.com/blog/matlab-compilation-quirks-take-2#respond Wed, 31 May 2017 18:00:42 +0000 http://undocumentedmatlab.com/?p=6919
 
Related posts:
  1. Matlab compiler bug and workaround Both the Matlab compiler and the publish function have errors when parsing block-comments in Matlab m-code. ...
  2. UDD Properties UDD provides a very convenient way to add customizable properties to existing Matlab object handles...
  3. Disabling menu entries in deployed docked figures Matlab's standard menu items can and should be removed from deployed docked figures. This article explains how. ...
  4. Handle Graphics Behavior HG behaviors are an important aspect of Matlab graphics that enable custom control of handle functionality. ...
 
]]>
Once again I would like to welcome guest blogger Hanan Kavitz of Applied Materials. Hanan posted a couple of guest posts here over the past few years, including a post last year about quirks with Matlab-compiled DLLs. Today Hanan will follow up on that post by discussing several additional quirks that they have encountered with Matlab compilations/deployment.

Don’t fix it, if it ain’t broke…

In Applied Materials Israel (PDC) we use Matlab code for both algorithm development and deployment (production). As part of the dev-ops build system, which builds our product software versions, we build Matlab artifacts (binaries) from the Matlab source code.

A typical software version has several hundreds Matlab artifacts that are automatically rebuilt on a daily basis, and we have many such versions – totaling many thousands of compilations each day.

This process takes a long time, so we were looking for a way to make it more efficient.

The idea that we chose to implement sounds simple – take a single binary module in any software version (Ex. foo.exe – Matlab-compiled exe) and check it: if the source code for this module has not changed since the last compilation then simply don’t compile it, just copy it from previous software version repository. Since most of our code doesn’t change daily (some of it hasn’t changed in years), we can skip the compilation time of most binaries and just copy them from some repository of previously compiled binaries.

In a broader look, avoiding lengthy compilations cycles by not compiling unchanged code is a common programming practice, implemented by all modern compilers. For example, the ‘make’ utility uses a ‘makefile’ to check the time stamps of all dependencies of every object file in order to decide which object requires recompilation. In reality, this is not always the best solution as time stamps may be incorrect, but it works well in the vast majority of cases.

Coming back to Matlab, now comes the hard part – how could our build system know that nothing has changed in module X and that something has changed in module Y? How does it even know which source files it needs to ensure didn’t change?

The credit for the idea goes to my manager, Lior Cohen, as follows: You can actually check the dependency of a given binary after compilation. The basis of the solution is that a Matlab executable is in fact a compressed (zip) file. The idea is then to:

  1. Compile the binary once
  2. Unzip the binary and “see” all your dependencies (source files are encrypted and resources are not, but we only need the list of file names – not their content).
  3. Now build a list of all your dependency files and compute the CRC value of each from the source control. Save it for the next time you are required to compile this module.
  4. In the next compilation cycle, find this dependency list, review it, dependency source file at a time and make sure CRC of the dependency hasn’t changed since last time.
  5. If no dependency CRC has changed, then copy the binary from the repository of previous software version, without compiling.
  6. Otherwise, recompile the binary and rebuild the CRC list of all dependencies again, in preparation for the next compilation cycle.

That’s it! That simple? Well… not really – the reality is a bit more complex since there are many other dependencies that need to be checked. Some of them are:

  1. Did the requested Matlab version of the binary change since the last compilation?
  2. Did the compilation instructions themselves (we have a sort of ‘makefile’) change?

Basically, I implemented a policy that if anything changed, or if the dependency check itself failed, then we don’t take any chances and just compile this binary. Keeping in mind that this dependencies check and file copying is much faster than a Matlab compilation, we save a lot of actual compilation time using this method.

Bottom line: Given a software version containing hundreds of compilation instructions to execute and assuming not much has changed in the version (which is often the case), we skip over 90% of compilations altogether and only rebuild what really changed. The result is a version build that takes about half an hour, instead of many hours. Moreover, since the compilation process is working significantly less, we get fewer failures, fewer stuck or crashed mcc processes, and [not less importantly] less maintenance required by me.

Note that in our implementation we rely on the undocumented fact that Matlab binaries are in fact compressed zip archives. If and when a future Matlab release will change the implementation such that the binaries will no longer be zip archives, another way will need to be devised in order to ensure the consistency of the target executable with its dependent source files.

Don’t kill it, if it ain’t bad…

I want to share a very weird issue I investigated over a year ago when using Matlab compiled exe. It started with a user showed me a Matlab compiled exe that didn’t run – I’m not talking about a regular Matlab exception: the process was crashing with an MS Windows popup window popping, stating something very obscure.

It was a very weird behavior that I couldn’t explain – the compiler seemed to work well but the compiled executable process kept crashing. Compiling completely different code showed the same behavior.

This issue has to do with the system compiler configuration that is being used. As you might know, when installing the Matlab compiler, before the first compilation is ever made, the user has to state the C compiler that the Matlab compiler should use in its compilation process. This is done by command ‘mbuild –setup’. This command asks the users to choose the C compiler and saves the configuration (batch file back then, xml in the newer versions of Matlab) in the user’s prefdir folder. At the time we were using Microsoft Visual C++ compiler 9.0 SP1.

The breakthrough in the investigation came when I ran mcc command with –verbose flag, which outputs much more compilation info than I would typically ever want… I discovered that although the target executable file had been created, a post compilation step failed to execute, while issuing a very cryptic error message:

mt.exe : general error c101008d: Failed to write the updated manifest to the resource of file “…”. Access is denied.

cryptic compilation error (click to zoom)

cryptic compilation error (click to zoom)

The failure was in one of the ‘post link’ commands in the configuration batch file – something obscure such as this:

set POSTLINK_CMDS2=mt.exe -outputresource: %MBUILD_OUTPUT_FILE_NAME%;%MANIFEST_RESOURCE% -manifest "%MANIFEST_FILE_NAME%"

This line of code takes an XML manifest file and inserts it into the generated binary file (additional details).

If you open a valid R2010a (and probably other old versions as well) Matlab-generated exe in a text editor you can actually see a small XML code embedded in it, while in a non-functioning exe I could not see this XML code.

So why would this command fail?

It turned out, as funny as it sounds, to be an antivirus issue – our IT department updated its antivirus policies and this ‘post link’ command suddenly became an illegal operation. Once our IT eased the policy, this command worked well again and the compiled executables stopped crashing, to our great joy.

]]>
https://undocumentedmatlab.com/blog/matlab-compilation-quirks-take-2/feed 0
GUI formatting using HTMLhttps://undocumentedmatlab.com/blog/gui-formatting-using-html https://undocumentedmatlab.com/blog/gui-formatting-using-html#comments Wed, 05 Apr 2017 20:26:44 +0000 http://undocumentedmatlab.com/?p=6877
 
Related posts:
  1. Icon images & text in Matlab uicontrols HTML can be used to add image icons to Matlab listbox and popup (drop-down) controls. ...
  2. Spicing up Matlab uicontrol tooltips Matlab uicontrol tooltips can be spiced-up using HTML and CSS, including fonts, colors, tables and images...
  3. Multi-line uitable column headers Matlab uitables can present long column headers in multiple lines, for improved readability. ...
  4. Rich-contents log panel Matlab listboxes and editboxes can be used to display rich-contents HTML-formatted strings, which is ideal for log panels. ...
 
]]>
As I’ve mentioned several times in the past, HTML can be used for simple formatting of GUI controls, including font colors/sizes/faces/angles. With a bit of thought, HTML (and some CSS) can also be used for non-trivial formatting, that would otherwise require the use of Java, such as text alignment, background color, and using a combination of text and icons in the GUI control’s contents.

Alignment

For example, a question that I am often asked (latest example) is whether it is possible to left/center/right align the label within a Matlab button, listbox or table. While Matlab does not (yet) have properties that control alignment in uicontrols, we can indeed use HTML for this. There’s a catch though: if we simply tried to use <div align="left">…, it will not work. No error will be generated but we will not see any visible left-alignment. The reason is that internally, the text is contained within a snugly-fitting box. Aligning anything within a tight-fitting box obviously has no effect.

To solve the problem, we need to tell Matlab (or rather, the HTML interpreter used by the underlying Java control) to widen this internal box. One way to do this is to specify the width of the div tag, which can be enormous in order to span the entire available apace (<div width="999px" align="left">…). Another method is to simulate a simple HTML table that contains a single cell that holds the text, and then tell HTML the table cell’s width:

hButton.String   = '<html><tr><td width=9999 align=left>Left-aligned';  % left-align within a button
hTable.Data{2,1} = '<html><tr><td width=9999 align=right>And right';   % right-align within a specific uitable cell

centered (default) button label   right-aligned button label

Centered (default) and right-aligned button labels

Non-default alignment of uitable cells

Non-default alignment of uitable cells

I discussed the specific aspect of uicontrol content alignment in another post last year.

Background color

The same problem (and solution) applies to background colors: if we don’t enlarge the snugly-fitting internal bounding-box, any HTML bgcolor that we specify would only be shown under the text (i.e., within the internal box’s confines). In order to display bgcolor across the entire control/cell width, we need to enlarge the internal box’s width (the align and bgcolor tags can of course be used together):

hButton.String   = '<html><tr><td width=9999 bgcolor=#ffff00>Yellow';  % bgcolor within a button
hTable.Data{2,1} = '<html><tr><td width=9999 bgcolor=#ffff00>Yellow';  % bgcolor within a specific uitable cell

CSS

We can also use simple CSS, which provides more formatting customizability than plain HTML:

hTable.Data{2,1} = '<html><tr><td width=9999 style="background-color:yellow">Yellow';

HTML/CSS formatting is a poor-man’s hack. It is very crude compared to the numerous customization options available via Java. However, it does provide a reasonable solution for many use-cases, without requiring any Java. I discussed the two approaches for uitable cell formatting in this post.

[Non-]support in uifigures

Important note: HTML formatting is NOT [yet] supported by the new web-based uifigures. While uifigures can indeed be hacked with HTML/CSS content (details), this is not an easy task. Since it should be trivially easy for MathWorks to enable HTML content in the new web-based uifigures, I implore anyone who uses HTML in their Matlab GUI to let MathWorks know about it so that they could prioritize this R&D effort into an upcoming Matlab release. You can send an email to George.Caia at mathworks.com, who apparently handles such aspects in MathWorks’ R&D efforts to transition from Java-based GUIs to web-based ones. In my previous post I spotlit MathWorks user-feedback surveys about users’ use of Java GUI aspects, aimed in order to migrate as many of the use-cases as possible onto the new web-based framework. HTML/CSS support is a natural by-product of the fact that Matlab’s non-web-based GUI is based on Java Swing components (that inherently support HTML/CSS). But unfortunately the MathWorks surveys are specific to the javacomponent function and the figure’s JavaFrame property. In other words, many users might be using undocumented Java aspects by simply using HTML content in their GUI, without ever realizing it or using javacomponent. So I think that in this case a simple email to George.Caia at mathworks.com to let him know how you’re using HTML would be more useful. Maybe one day MathWorks will be kind enough to post a similar survey specific to HTML support, or maybe one day they’s just add the missing HTML support, if only to be done with my endless nagging. :-)

p.s. – I am well aware that we can align and bgcolor buttons in AppDesigner. But we can’t do this with individual table/listbox cells, and in general we can’t use HTML within uifigures without extensive hacks. I merely used the simple examples of button and uitable cell formatting in today’s post to illustrate the issue. So please don’t get hung up on the specifics, but rather on the broader issue of HTML support in uifigures.

And in the meantime, for as long as non-web-based GUI is still supported in Matlab, keep on enjoying the benefits that HTML/CSS provides.

Automated bug-fix emails

In an unrelated matter, I wish to express my Kudos to the nameless MathWorkers behind the scenes who, bit by bit, improve Matlab and the user experience: Over the years I’ve posted a few times my frustrations with the opaqueness of MathWorks’ bug-reporting mechanism. One of my complaints was that users who file bugs are not notified when a fix or workaround becomes available. That at least seems to have been fixed now. I just received a seemingly-automated email notifying me that one of the bugs that I reported a few years ago has been fixed. This is certainly a good step in the right direction, so thank you!

Happy Passover/Easter to all!

]]>
https://undocumentedmatlab.com/blog/gui-formatting-using-html/feed 1
MathWorks-solicited Java surveyhttps://undocumentedmatlab.com/blog/mathworks-solicited-java-survey https://undocumentedmatlab.com/blog/mathworks-solicited-java-survey#comments Wed, 22 Mar 2017 22:05:34 +0000 http://undocumentedmatlab.com/?p=6866
 
Related posts:
  1. Minimize/maximize figure window Matlab figure windows can easily be maximized, minimized and restored using a bit of undocumented magic powder...
  2. Types of undocumented Matlab aspects This article lists the different types of undocumented/unsupported/hidden aspects in Matlab...
  3. FindJObj – find a Matlab component’s underlying Java object The FindJObj utility can be used to access and display the internal components of Matlab controls and containers. This article explains its uses and inner mechanism....
  4. Matlab callbacks for Java events Events raised in Java code can be caught and handled in Matlab callback functions - this article explains how...
 
]]>
Over the years I’ve reported numerous uses for integrating Java components and functionality in Matlab. As I’ve also recently reported, MathWorks is apparently making a gradual shift away from standalone Java-based figures, toward browser-based web-enabled figures. As I surmised a few months ago, MathWorks has created dedicated surveys to solicit user feedbacks on the most important (and undocumented) non-compatible aspects of this paradigm change: one regarding users’ use of the javacomponent function, the other regarding the use of the figure’s JavaFrame property:

In MathWorks’ words:

In order to extend your ability to build MATLAB apps, we understand you sometimes need to make use of undocumented Java UI technologies, such as the JavaFrame property. In response to your needs, we are working to develop documented alternatives that address gaps in our app building offerings.

To help inform our work and plans, we would like to understand how you are using the JavaFrame property. Based on your understanding of how it is being used within your app, please take a moment to fill out the following survey. The survey will take approximately 1-2 minutes to finish.

I urge anyone who uses one or both of these features to let MathWorks know how you’re using them, so that they could incorporate that functionality into the core (documented) Matlab. The surveys are really short and to the point. If you wish to send additional information, please email George.Caia at mathworks.com.

The more feedback responses that MathWorks will get, the better it will be able to prioritize its R&D efforts for the benefit of all users, and the more likely are certain features to get a documented solution at some future release. If you don’t take the time now to tell MathWorks how you use these features in your code, don’t complain if and when they break in the future…

My personal uses of these features

  • Functionality:
    • Figure: maximize/minimize/restore, enable/disable, always-on-top, toolbar controls, menu customizations (icons, tooltips, font, shortcuts, colors)
    • Table: sorting, filtering, grouping, column auto-sizing, cell-specific behavior (tooltip, context menu, context-sensitive editor, merging cells)
    • Tree control
    • Listbox: cell-specific behavior (tooltip, context menu)
    • Tri-state checkbox
    • uicontrols in general: various event callbacks (e.g. mouse hover/unhover, focus gained/lost)
    • Ability to add Java controls e.g. color/font/date/file selector panel or dropdown, spinner, slider, search box, password field
    • Ability to add 3rd-party components e.g. JFreeCharts, JIDE controls/panels

  • Appearance:
    • Figure: undecorated (frameless), other figure frame aspects
    • Table: column/cell-specific rendering (alignment, icons, font, fg/bg color, string formatting)
    • Listbox: auto-hide vertical scrollbar as needed, cell-specific renderer (icon, font, alignment, fg/bg color)
    • Button/checkbox/radio: icons, text alignment, border customization, Look & Feel
    • Right-aligned checkbox (button to the right of label)
    • Panel: border customization (rounded/matte/…)

You can find descriptions/explanations of many of these in posts I made on this website over the years.

]]>
https://undocumentedmatlab.com/blog/mathworks-solicited-java-survey/feed 2
I am hiring experienced Matlab programmers (Tel Aviv)https://undocumentedmatlab.com/blog/i-am-hiring-experienced-matlab-programmers-tel-aviv https://undocumentedmatlab.com/blog/i-am-hiring-experienced-matlab-programmers-tel-aviv#respond Mon, 20 Feb 2017 09:14:49 +0000 http://undocumentedmatlab.com/?p=6857
 
Related posts:
  1. About This Blog is the fruit of passion. In over 20 years of programming in a multitude of different programming languages and environments, I have found Matlab to be second to...
  2. I am hiring a Matlab programmer (Tel Aviv) I am hiring a Matlab programmer in the Tel Aviv area...
  3. USA visit, 22-31 July 2014 I will be visiting some US cities on July 2014. ...
  4. New training courses I am now offering a new service of professional Matlab training, at your location. ...
 
]]>
I am hiring experienced Matlab programmers for work in Tel Aviv, to join a growing team of Matlab experts working under my supervision. Very interesting work, good salary, and flexible worhours. This job opening is only applicable to candidates who live in central Israel. If you live in the area and are interested, or if you know someone who could be a good fit, please email me: altmany at gmail.

-אני מגייס מתכנת/ת מטלב מנוסה לעבודה בתל אביב בחברת הייעוץ שלי. המשרה בהיקף ובימים/שעות גמישים, העבודה מעניינת והשכר טוב. לפרטים נא לפנות ל
altmany at gmail

]]>
https://undocumentedmatlab.com/blog/i-am-hiring-experienced-matlab-programmers-tel-aviv/feed 0
Additional license datahttps://undocumentedmatlab.com/blog/additional-license-data https://undocumentedmatlab.com/blog/additional-license-data#comments Wed, 15 Feb 2017 18:01:55 +0000 http://undocumentedmatlab.com/?p=6852
 
Related posts:
  1. uiundo – Matlab’s undocumented undo/redo manager The built-in uiundo function provides easy yet undocumented access to Matlab's powerful undo/redo functionality. This article explains its usage....
  2. Undocumented Profiler options part 3 An undocumented feature of the Matlab Profiler can report call history timeline - part 3 of series. ...
  3. Undocumented Profiler options part 4 Several undocumented features of the Matlab Profiler can make it much more useful - part 4 of series. ...
  4. Pinning annotations to graphs Annotation object can be programmatically set at, and pinned-to, plot axes data points. ...
 
]]>
Matlab’s license function returns the primary license number/ID used by Matlab, but no information about the various toolboxes that may be installed. The ver function returns a bit more information, listing the version number and installation date of installed toolboxes (even user toolboxes, such as my IB-Matlab toolbox). However, no additional useful information is provided beyond that:

>> license
ans =
123456   % actual number redacted
 
>> ver
----------------------------------------------------------------------------------------------------
MATLAB Version: 9.1.0.441655 (R2016b)
MATLAB License Number: 123456
Operating System: Microsoft Windows 7 Professional  Version 6.1 (Build 7601: Service Pack 1)
Java Version: Java 1.7.0_60-b19 with Oracle Corporation Java HotSpot(TM) 64-Bit Server VM mixed mode
----------------------------------------------------------------------------------------------------
MATLAB                                                Version 9.1         (R2016b)           
Curve Fitting Toolbox                                 Version 3.5.4       (R2016b)           
Database Toolbox                                      Version 7.0         (R2016b)           
Datafeed Toolbox                                      Version 5.4         (R2016b)           
Financial Instruments Toolbox                         Version 2.4         (R2016b)           
Financial Toolbox                                     Version 5.8         (R2016b)           
GUI Layout Toolbox                                    Version 2.2.1       (R2015b)           
Global Optimization Toolbox                           Version 3.4.1       (R2016b)           
IB-Matlab - Matlab connector to InteractiveBrokers    Version 1.89        Expires: 1-Apr-2018
Image Processing Toolbox                              Version 9.5         (R2016b)           
MATLAB Coder                                          Version 3.2         (R2016b)           
MATLAB Report Generator                               Version 5.1         (R2016b)           
Optimization Toolbox                                  Version 7.5         (R2016b)           
Parallel Computing Toolbox                            Version 6.9         (R2016b)           
Statistical Graphics Toolbox                          Version 1.2                            
Statistics and Machine Learning Toolbox               Version 11.0        (R2016b)           
 
>> v = ver
v = 
  1×16 struct array with fields:
    Name
    Version
    Release
    Date
 
>> v(1)
ans = 
  struct with fields:
 
       Name: 'Curve Fitting Toolbox'
    Version: '3.5.4'
    Release: '(R2016b)'
       Date: '25-Aug-2016'
 
>> v(8)
ans = 
  struct with fields:
 
       Name: 'IB-Matlab - Matlab connector to InteractiveBrokers'
    Version: '1.89'
    Release: 'Expires: 1-Apr-2018'
       Date: '02-Feb-2017'

It is sometimes useful to know which license number “owns” which product/toolbox, and the expiration date is associated with each of them. Unfortunately, there is no documented way to retrieve this information in Matlab – the only documented way is to go to your account section on the MathWorks website and check there.

Luckily, there is a simpler way that can be used to retrieve additional information, from right inside Matlab, using matlab.internal.licensing.getFeatureInfo:

>> all_data = matlab.internal.licensing.getFeatureInfo
all_data = 
  23×1 struct array with fields:
    feature
    expdate
    keys
    license_number
    entitlement_id
 
>> all_data(20)
ans = 
  struct with fields:
 
           feature: 'optimization_toolbox'
           expdate: '31-mar-2018'
              keys: 0
    license_number: '123456'
    entitlement_id: '1409891'
 
>> all_data(21)
ans = 
  struct with fields:
 
           feature: 'optimization_toolbox'
           expdate: '07-mar-2017'
              keys: 0
    license_number: 'DEMO'
    entitlement_id: '3749959'

As can be seen in this example, I have the Optimization toolbox licensed under my main Matlab license (123456 [actual number redacted]) until 31-mar-2018, and also licensed under a trial (DEMO) license that expires in 3 weeks. As long as a toolbox has any future expiration date, it will continue to function, so in this case I’m covered until March 2018.

We can also request information about a specific toolbox (“feature”):

>> data = matlab.internal.licensing.getFeatureInfo('matlab')
data = 
  3×1 struct array with fields:
    feature
    expdate
    keys
    license_number
    entitlement_id
 
>> data(1)
data = 
  struct with fields:
 
           feature: 'matlab'
           expdate: '31-mar-2018'
              keys: 0
    license_number: '123456'
    entitlement_id: '1409891'

The drawback of this functionality is that it only provides information about MathWorks’ toolbox, not any user-provided toolboxes (such as my IB-Matlab connector, or MathWorks’ own GUI Layout toolbox). Also, some of the toolbox names may be difficult to understand (“gads_toolbox” apparently stands for the Global Optimization Toolbox, for example):

>> {all_data.feature}
ans =
  1×23 cell array
  Columns 1 through 4
    'curve_fitting_toolbox'    'database_toolbox'    'datafeed_toolbox'    'distrib_computing_toolbox'
  Columns 5 through 8
    'distrib_computing_toolbox'    'excel_link'    'fin_instruments_toolbox'    'financial_toolbox'
  Columns 9 through 15
    'gads_toolbox'    'gads_toolbox'    'image_toolbox'    'image_toolbox'    'matlab'    'matlab'    'matlab'
  Columns 16 through 20
    'matlab_coder'    'matlab_coder'    'matlab_report_gen'    'matlab_report_gen'    'optimization_toolbox'
  Columns 21 through 23
    'optimization_toolbox'    'optimization_toolbox'    'statistics_toolbox'

A related undocumented builtin function is matlab.internal.licensing.getLicInfo:

% Information on a single toolbox/product:
>> matlab.internal.licensing.getLicInfo('matlab')
ans = 
  struct with fields:
 
     license_number: {'123456'  'Prerelease'  'T3749959'}
    expiration_date: {'31-mar-2018'  '30-sep-2016'  '07-mar-2017'}
 
% Information on multiple toolboxes/products:
>> matlab.internal.licensing.getLicInfo({'matlab', 'image_toolbox'})  % cell array of toolbox/feature names
ans = 
  1×2 struct array with fields:
    license_number
    expiration_date
 
% The full case-insensitive names of the toolboxes can also be used:
>> matlab.internal.licensing.getLicInfo({'Matlab', 'Image Processing toolbox'})
ans = 
  1×2 struct array with fields:
    license_number
    expiration_date
 
% And here's how to get the full list (MathWorks products only):
>> v=ver; data=matlab.internal.licensing.getLicInfo({v.Name})
data = 
  1×16 struct array with fields:
    license_number
    expiration_date

I have [still] not found any way to associate a user toolbox/product (such as my IB-Matlab) in a way that will report it in a unified manner with the MathWorks products. If anyone finds a way to do this, please do let me know.

p.s. – don’t even think of asking questions or posting comments on this website related to illegal uses or hacks of the Matlab license…

]]>
https://undocumentedmatlab.com/blog/additional-license-data/feed 5
Parsing XML stringshttps://undocumentedmatlab.com/blog/parsing-xml-strings https://undocumentedmatlab.com/blog/parsing-xml-strings#comments Wed, 01 Feb 2017 09:52:45 +0000 http://undocumentedmatlab.com/?p=6838
 
Related posts:
  1. Undocumented XML functionality Matlab's built-in XML-processing functions have several undocumented features that can be used by Java-savvy users...
  2. Matlab-Java memory leaks, performance Internal fields of Java objects may leak memory - this article explains how to avoid this without sacrificing performance. ...
  3. Types of undocumented Matlab aspects This article lists the different types of undocumented/unsupported/hidden aspects in Matlab...
  4. Pause for the better Java's thread sleep() function is much more accurate than Matlab's pause() function. ...
 
]]>
I have recently consulted in a project where data was provided in XML strings and needed to be parsed in Matlab memory in an efficient manner (in other words, as quickly as possible). Now granted, XML is rather inefficient in storing data (JSON would be much better for this, for example). But I had to work with the given situation, and that required processing the XML.

I basically had two main alternatives:

  • I could either create a dedicated string-parsing function that searches for a particular pattern within the XML string, or
  • I could use a standard XML-parsing library to create the XML model and then parse its nodes

The first alternative is quite error-prone, since it relies on the exact format of the data in the XML. Since the same data can be represented in multiple equivalent XML ways, making the string-parsing function robust as well as efficient would be challenging. I was lazy expedient, so I chose the second alternative.

Unfortunately, Matlab’s xmlread function only accepts input filenames (of *.xml files), it cannot directly parse XML strings. Yummy!

The obvious and simple solution is to simply write the XML string into a temporary *.xml file, read it with xmlread, and then delete the temp file:

% Store the XML data in a temp *.xml file
filename = [tempname '.xml'];
fid = fopen(filename,'Wt');
fwrite(fid,xmlString);
fclose(fid);
 
% Read the file into an XML model object
xmlTreeObject = xmlread(filename);
 
% Delete the temp file
delete(filename);
 
% Parse the XML model object
...

This works well and we could move on with our short lives. But cases such as this, where a built-in function seems to have a silly limitation, really fire up the investigative reporter in me. I decided to drill into xmlread to discover why it couldn’t parse XML strings directly in memory, without requiring costly file I/O. It turns out that xmlread accepts not just file names as input, but also Java object references (specifically, java.io.File, java.io.InputStream or org.xml.sax.InputSource). In fact, there are quite a few other inputs that we could use, to specify a validation parser etc. – I wrote about this briefly back in 2009 (along with other similar semi-documented input altermatives in xmlwrite and xslt).

In our case, we could simply send xmlread as input a java.io.StringBufferInputStream(xmlString) object (which is an instance of java.io.InputStream) or org.xml.sax.InputSource(java.io.StringReader(xmlString)):

% Read the xml string directly into an XML model object
inputObject = java.io.StringBufferInputStream(xmlString);                % alternative #1
inputObject = org.xml.sax.InputSource(java.io.StringReader(xmlString));  % alternative #2
 
xmlTreeObject = xmlread(inputObject);
 
% Parse the XML model object
...

If we don’t want to depend on undocumented functionality (which might break in some future release, although it has remained unchanged for at least the past decade), and in order to improve performance even further by passing xmlread‘s internal validity checks and processing, we can use xmlread‘s core functionality to parse our XML string directly. We can add a fallback to the standard (fully-documented) functionality, just in case something goes wrong (which is good practice whenever using any undocumented functionality):

try
    % The following avoids the need for file I/O:
    inputObject = java.io.StringBufferInputStream(xmlString);  % or: org.xml.sax.InputSource(java.io.StringReader(xmlString))
    try
        % Parse the input data directly using xmlread's core functionality
        parserFactory = javaMethod('newInstance','javax.xml.parsers.DocumentBuilderFactory');
        p = javaMethod('newDocumentBuilder',parserFactory);
        xmlTreeObject = p.parse(inputObject);
    catch
        % Use xmlread's semi-documented inputObject input feature
        xmlTreeObject = xmlread(inputObject);
    end
catch
    % Fallback to standard xmlread usage, using a temporary XML file:
 
    % Store the XML data in a temp *.xml file
    filename = [tempname '.xml'];
    fid = fopen(filename,'Wt');
    fwrite(fid,xmlString);
    fclose(fid);
 
    % Read the file into an XML model object
    xmlTreeObject = xmlread(filename);
 
    % Delete the temp file
    delete(filename);
end
 
% Parse the XML model object
...
]]>
https://undocumentedmatlab.com/blog/parsing-xml-strings/feed 4
Quirks with parfor vs. forhttps://undocumentedmatlab.com/blog/quirks-with-parfor-vs-for https://undocumentedmatlab.com/blog/quirks-with-parfor-vs-for#comments Thu, 05 Jan 2017 17:15:48 +0000 http://undocumentedmatlab.com/?p=6821
 
Related posts:
  1. Matlab mex in-place editing Editing Matlab arrays in-place can be an important technique for optimizing calculations. This article shows how to do it using Mex. ...
  2. Preallocation performance Preallocation is a standard Matlab speedup technique. Still, it has several undocumented aspects. ...
  3. Array resizing performance Several alternatives are explored for dynamic array growth performance in Matlab loops. ...
  4. Matlab’s internal memory representation Matlab's internal memory structure is explored and discussed. ...
 
]]>
A few months ago, I discussed several tips regarding Matlab’s parfor command, which is used by the Parallel Computing Toolbox (PCT) for parallelizing loops. Today I wish to extend that post with some unexplained oddities when using parfor, compared to a standard for loop.

Data serialization quirks

Dimitri Shvorob may not appear at first glance to be a prolific contributor on Matlab Central, but from the little he has posted over the years I regard him to be a Matlab power-user. So when Dimitri reports something, I take it seriously. Such was the case several months ago, when he contacted me regarding very odd behavior that he saw in his code: the for loop worked well, but the parfor version returned different (incorrect) results. Eventually, Dimitry traced the problem to something originally reported by Dan Austin on his Fluffy Nuke It blog.

The core issue is that if we have a class object that is used within a for loop, Matlab can access the object directly in memory. But with a parfor loop, the object needs to be serialized in order to be sent over to the parallel workers, and deserialized within each worker. If this serialization/deserialization process involves internal class methods, the workers might see a different version of the class object than the one seen in the serial for loop. This could happen, for example, if the serialization/deserialization method croaks on an error, or depends on some dynamic (or random) conditions to create data.

In other words, when we use data objects in a parfor loop, the data object is not necessarily sent “as-is”: additional processing may be involved under the hood that modify the data in a way that may be invisible to the user (or the loop code), resulting in different processing results of the parallel (parfor) vs. serial (for) loops.

For additional aspects of Matlab serialization/deserialization, see my article from 2 years ago (and its interesting feedback comments).

Data precision quirks

The following section was contributed by guest blogger Lior Perlmuter-Shoshany, head algorithmician at a private equity fund.

In my work, I had to work with matrixes in the order of 109 cells. To reduce the memory footprint (and hopefully also improve performance), I decided to work with data of type single instead of Matlab’s default double. Furthermore, in order to speed up the calculation I use parfor rather than for in the main calculation. In the end of the run I am running a mini for-loop to see the best results.

What I discovered to my surprise is that the results from the parfor and for loop variants is not the same!

The following simplified code snippet illustrate the problem by calculating a simple standard-deviation (std) over the same data, in both single– and double-precision. Note that the loops are ran with only a single iteration, to illustrate the fact that the problem is with the parallelization mechanism (probably the serialization/deserialization parts once again), not with the distribution of iterations among the workers.

clear
rng('shuffle','twister');
 
% Prepare the data in both double and single precision
arr_double = rand(1,100000000);
arr_single = single(arr_double);
 
% No loop - direct computation
std_single0 = std(arr_single);
std_double0 = std(arr_double);
 
% Loop #1 - serial for loop
std_single = 0;
std_double = 0;
for i=1
    std_single(i) = std(arr_single);
    std_double(i) = std(arr_double);
end
 
% Loop #2 - parallel parfor loop
par_std_single = 0;
par_std_double = 0;
parfor i=1
    par_std_single(i) = std(arr_single);
    par_std_double(i) = std(arr_double);
end
 
% Compare results of for loop vs. non-looped computation
isForSingleOk = isequal(std_single, std_single0)
isForDoubleOk = isequal(std_double, std_double0)
 
% Compare results of single-precision data (for vs. parfor)
isParforSingleOk = isequal(std_single, par_std_single)
parforSingleAccuracy = std_single / par_std_single
 
% Compare results of double-precision data (for vs. parfor)
isParforDoubleOk = isequal(std_double, par_std_double)
parforDoubleAccuracy = std_double / par_std_double

Output example :

isForSingleOk = 
    1                   % <= true (of course!)
isForDoubleOk =
    1                   % <= true (of course!)
 
isParforSingleOk =
    0                   % <= false (odd!)
parforSingleAccuracy =
    0.73895227413361    % <= single-precision results are radically different in parfor vs. for
 
isParforDoubleOk =
    0                   % <= false (odd!)
parforDoubleAccuracy =
    1.00000000000021    % <= double-precision results are almost [but not exactly] the same in parfor vs. for

From my testing, the larger the data array, the bigger the difference is between the results of single-precision data when running in for vs. parfor.

In other words, my experience has been that if you have a huge data matrix, it’s better to parallelize it in double-precision if you wish to get [nearly] accurate results. But even so, I find it deeply disconcerting that the results are not exactly identical (at least on R2015a-R2016b on which I tested) even for the native double-precision .

Hmmm… bug?

Upcoming travels – Zürich & Geneva

I will shortly be traveling to clients in Zürich and Geneva, Switzerland. If you are in the area and wish to meet me to discuss how I could bring value to your work with some advanced Matlab consulting or training, then please email me (altmany at gmail):

  • Zürich: January 15-17
  • Geneva: January 18-21

Happy new year everybody!

]]>
https://undocumentedmatlab.com/blog/quirks-with-parfor-vs-for/feed 7
Checking status of warning messages in MEXhttps://undocumentedmatlab.com/blog/checking-status-of-warning-messages-in-mex https://undocumentedmatlab.com/blog/checking-status-of-warning-messages-in-mex#respond Wed, 21 Dec 2016 15:24:06 +0000 http://undocumentedmatlab.com/?p=6797
 
Related posts:
  1. Inactive Control Tooltips & Event Chaining Inactive Matlab uicontrols cannot normally display their tooltips. This article shows how to do this with a combination of undocumented Matlab and Java hacks....
  2. Introduction to UDD UDD classes underlie many of Matlab's handle-graphics objects and functionality. This article introduces these classes....
  3. Creating a simple UDD class This article explains how to create and test custom UDD packages, classes and objects...
  4. Undocumented Matlab MEX API Matlab's MEX API contains numerous undocumented functions, that can be extremely useful. ...
 
]]>
Once again I would like to welcome guest blogger Pavel Holoborodko, the developer of the Advanpix Multiprecision Computing Toolbox. Pavel has already posted here as a guest blogger about undocumented Matlab MEX functions. Today he will discuss another little-known aspect of advanced MEX programming with Matlab, a repost of an article that was originally posted on his own blog. Happy holidays everybody!

Matlab allows flexible adjustment of visibility of warning messages. Some, or even all, messages can be disabled from showing on the screen by warning command.

The little known fact is that status of some warnings may be used to change the execution path in algorithms. For example, if warning 'Matlab:nearlySingularMatrix' is disabled, then the linear system solver (mldivide operator) might skip estimation of reciprocal condition number which is used exactly for the purpose of detection of nearly singular matrices. If the trick is used, it allows 20%-50% boost in solver performance, since rcond estimation is a time consuming process.

Therefore it is important to be able to retrieve status of warnings in Matlab. Especially in MEX libraries targeted for improved performance. Unfortunately Matlab provides no simple way to check status of warning message from MEX module.

Today’s article outlines two workarounds for the issue:

  1. Using mexCallMATLABWithTrap (documented)
  2. Using utGetWarningStatus (undocumented)

Using mexCallMATLABWithTrap (documented)

The first idea is to use documented mexCallMATLABWithTrap function to execute warning(‘query’,…) command using Matlab’s interpreter and then parse the returned result:

bool mxIsWarningEnabled(const char* warningId)
{
    bool enabled = true;
 
    if (NULL != warningId)
    {
        mxArray *mxCommandResponse = NULL, *mxException = NULL;
        mxArray *args[2];
 
        /* warning('query', warningId); */
        args[0] = mxCreateString("query");
        args[1] = mxCreateString(warningId);
        mxException = mexCallMATLABWithTrap(1,&mxCommandResponse,2,args,"warning");
        if (NULL == mxException && NULL != mxCommandResponse)
        {
            if (mxIsStruct(mxCommandResponse))
            {
                const mxArray* state_field = mxGetField(mxCommandResponse, 0, "state");
                if (mxIsChar(state_field))
                {
                    char state_value[8] = {0};
                    enabled = (0 == mxGetString(state_field, state_value, 8)) &&
                              (0 == strcmp(state_value,"on"));
                }
            }
            mxDestroyArray(mxCommandResponse);
        }
        else
        {
            /* 'warning' returned with error */
            mxDestroyArray(mxException);
        }
        mxDestroyArray(args[0]);
        mxDestroyArray(args[1]);
    }
    return enabled;
}

This approach is slow, but works fine in most standard situations. See the bottom of this post for a usage example.

However, this approach has an important drawback – we should be careful with recursive calls to the Matlab interpreter (Matlab -> MEX -> Matlab) and with handling Matlab errors in MEX. It is safe only if we use identical standard libraries and compiler to build both MEX and Matlab.

In other cases, for example when MEX is targeted to work with different versions of Matlab, or was built with a different standard library and compiler, etc. – cross boundary handling of errors (which are just C++ exceptions) might lead to unpredictable results, most likely segfaults.

Using utGetWarningStatus (undocumented)

To avoid all the overhead of calling Matlab interpreter and unsafe error handling, we can use some undocumented internal Matlab functions:

/* Link with libut library to pick-up undocumented functions: */
extern "C" void* utGetWarningManagerContext(void);
extern "C" bool  utIsValidMessageIdentifier(const char *warningId);
extern "C" bool  utGetWarningStatus(void* context, const char *warningId);
 
/* 
   Returns true if warning with warningId enabled 
   Matlab versions supported/tested: R2008b - R2016b
*/
bool mxIsWarningEnabled(const char *warningId)
{
    bool enabled = true;
 
    if (NULL != warningId && utIsValidMessageIdentifier(warningId))
    {
        void* context = utGetWarningManagerContext();
        enabled = (NULL != context) && utGetWarningStatus(context, warningId);
    }
    return enabled;
}

Now the code is clean, fast and safe – we bypass the interpreter and work directly with Matlab kernel. All the undocumented functions involved are present in Matlab for at least 10 years and do simple logical checks under the hood.

The standard function mexWarnMsgIdAndTxt uses similar code to check if it should display the warning or just suppress it, and that code remains unchanged since R2008b. This is a good indication of code stability and makes us believe that it will not be changed in future versions of Matlab.

For both workarounds, usage is simple:

if (mxIsWarningEnabled("Matlab:nearlySingularMatrix"))
{
   /* compute rcond */
}
else
{
   /* do something else */
}
]]>
https://undocumentedmatlab.com/blog/checking-status-of-warning-messages-in-mex/feed 0
Password & spinner controls in Matlab GUIhttps://undocumentedmatlab.com/blog/password-and-spinner-controls-in-matlab-gui https://undocumentedmatlab.com/blog/password-and-spinner-controls-in-matlab-gui#comments Wed, 14 Dec 2016 17:28:09 +0000 http://undocumentedmatlab.com/?p=6775
 
Related posts:
  1. Uitable sorting Matlab's uitables can be sortable using simple undocumented features...
  2. Figure toolbar components Matlab's toolbars can be customized using a combination of undocumented Matlab and Java hacks. This article describes how to access existing toolbar icons and how to add non-button toolbar components....
  3. Tab panels – uitab and relatives This article describes several undocumented Matlab functions that support tab-panels...
  4. The javacomponent function Matlab's built-in javacomponent function can be used to display Java components in Matlab application - this article details its usages and limitations...
 
]]>
I often include configuration panels in my programs, to enable the user to configure various program aspects, such as which emails should automatically be sent by the program to alert when certain conditions occur. Last week I presented such a configuration panel, which is mainly composed of standard documented Matlab controls (sub-panels, uitables and uicontrols). As promised, today’s post will discuss two undocumented controls that are often useful in similar configuration panels (not necessarily for emails): password fields and spinners.

Matlab GUI configuration panel including password and spinner controls (click to zoom-in)
Matlab GUI configuration panel including password and spinner controls (click to zoom-in)

Password fields are basically editboxes that hide the typed text with some generic echo character (such as * or a bullet); spinners are editboxes that only enable typing certain preconfigured values (e.g., numbers in a certain range). Both controls are part of the standard Java Swing package, on which the current (non-web-based) Matlab GUIs relies. In both cases, we can use the javacomponent function to place the built-in Swing component in our Matlab GUI.

Password field

The relevant Java Swing control for password fields is javax.swing.JPasswordField. JPasswordField is basically an editbox that hides any typed key with a * or bullet character.

Here’s a basic code snippet showing how to display a simple password field:

jPasswordField = javax.swing.JPasswordField('defaultPassword');  % default password arg is optional
jPasswordField = javaObjectEDT(jPasswordField);  % javaObjectEDT is optional but recommended to avoid timing-related GUI issues
jhPasswordField = javacomponent(jPasswordField, [10,10,70,20], gcf);

Password control

Password control

We can set/get the password string programmatically via the Text property; the displayed (echo) character can be set/get using the EchoChar property.

To attach a data-change callback, set jhPasswordField’s ActionPerformedCallback property.

Spinner control

detailed post on using spinners in Matlab GUI

The relevant Java Swing control for spinners is javax.swing.JSpinner. JSpinner is basically an editbox with two tiny adjacent up/down buttons that visually emulate a small round spinning knob. Spinners are similar in functionality to a combo-box (a.k.a. drop-down or pop-up menu), where a user can switch between several pre-selected values. They are often used when the list of possible values is too large to display in a combo-box menu. Like combo-boxes, spinners too can be editable (meaning that the user can type a value in the editbox) or not (the user can only “spin” the value using the up/down buttons).

JSpinner uses an internal data model. The default model is SpinnerNumberModel, which defines a min/max value (unlimited=[] by default) and step-size (1 by default). Additional predefined models are SpinnerListModel (which accepts a cell array of possible string values) and SpinnerDateModel (which defines a date range and step unit).

Here’s a basic code snippet showing how to display a simple numeric spinner for numbers between 20 and 35, with an initial value of 24 and increments of 0.1:

jModel = javax.swing.SpinnerNumberModel(24,20,35,0.1);
jSpinner = javax.swing.JSpinner(jModel);
jSpinner = javaObjectEDT(jSpinner);  % javaObjectEDT is optional but recommended to avoid timing-related GUI issues
jhSpinner = javacomponent(jSpinner, [10,10,70,20], gcf);

The spinner value can be set using the edit-box or by clicking on one of the tiny arrow buttons, or programmatically by setting the Value property. The spinner object also has related read-only properties NextValue and PreviousValue. The spinner’s model object has the corresponding Value (settable), NextValue (read-only) and PreviousValue (read-only) properties. In addition, the various models have specific properties. For example, SpinnerNumberModel has the settable Maximum, Minimum and StepSize properties.

To attach a data-change callback, set jhSpinner’s StateChangedCallback property.

I have created a small Matlab demo, SpinnerDemo, which demonstrates usage of JSpinner in Matlab figures. Each of the three predefined models (number, list, and date) is presented, and the spinner values are inter-connected via their callbacks. The Matlab code is modeled after the Java code that is used to document JSpinner in the official Java documentation. Readers are welcome to download this demo from the Matlab File Exchange and reuse its source code.

Matlab SpinnerDemo

Matlab SpinnerDemo

The nice thing about spinners is that you can set a custom display format without affecting the underlying data model. For example, the following code snippet update the spinner’s display format without affecting its underlying numeric data model:

formatStr = '$ #,##0.0 Bn';
jEditor = javaObject('javax.swing.JSpinner$NumberEditor', jhSpinner, formatStr);
jhSpinner.setEditor(jEditor);

Formatted spinner control

Formatted spinner control

For more information, refer to my detailed post on using spinners in Matlab GUI.

Caveat emptor

MathWorks’ new web-based GUI paradigm will most probably not directly support the Java components presented in today’s post, or more specifically the javacomponent function that enables placing them in Matlab GUIs. The new web-based GUI-building application (AppDesigner, aka AD) does contain a spinner, although it is [currently] limited to displaying numeric values (not dates/lists as in my SpinnerDemo). Password fields are not currently supported by AppDesigner at all, and it is unknown whether they will ever be.

All this means that users of Java controls who wish to transition to the new web-based GUIs will need to develop programmatic workarounds, that would presumably appear and behave less professional. It’s a tradeoff: AppDesigner does include features that improve GUI usability, not to mention the presumed future ability to post Matlab GUIs online (hopefully without requiring a monstrous Matlab Production Server license/installation).

In the past, MathWorks has posted a dedicated webpage to solicit user feedback on how they are using the figure’s JavaFrame property. MathWorks will presumably prepare a similar webpage to solicit user feedback on uses of the javacomponent function, so they could add the top items to AppDesigner, making the transition to web-based GUIs less painful. When such a survey page becomes live, I will post about it on this website so that you could tell MathWorks about your specific use-cases and help them prioritize their R&D efforts.

In any case, regardless of whether the functionality eventually makes it into AppDesigner, my hope is that when the time comes MathWorks will not pull the plug from non-web GUIs, and will still enable running them on desktops for backward compatibility (“legacy mode”). Users of existing GUIs will then not need to choose between upgrading their Matlab (and redeveloping their GUI as a web-based app) and running their existing programs. Instead, users will face the much less painful choice between keeping the existing Java-based programs and developing a web-based variant at some later time, separate from the choice of whether or not to upgrade Matlab. The increased revenue from license upgrades and SMS (maintenance plan) renewals might well offset the R&D effort that would be needed to keep supporting the old Java-based figures. The traumatic* release of HG2 in R2014b, where a less-than-perfect version was released with no legacy mode, resulting in significant user backlash/disappointment, is hopefully still fresh in the memory of decision makers and would hopefully not be repeated.

*well, traumatic for some at least. I really don’t wish to make this a debate on HG2’s release; I’d rather focus on making the transition to web-based GUIs as seamless as possible.

]]>
https://undocumentedmatlab.com/blog/password-and-spinner-controls-in-matlab-gui/feed 4
Sending email/text messages from Matlabhttps://undocumentedmatlab.com/blog/sending-email-text-messages-from-matlab https://undocumentedmatlab.com/blog/sending-email-text-messages-from-matlab#comments Wed, 07 Dec 2016 21:24:03 +0000 http://undocumentedmatlab.com/?p=6765
 
Related posts:
  1. Types of undocumented Matlab aspects This article lists the different types of undocumented/unsupported/hidden aspects in Matlab...
  2. Legend ‘-DynamicLegend’ semi-documented feature The built-in Matlab legend function has a very useful semi-documented feature for automatic dynamic update, which is explained here....
  3. Undocumented XML functionality Matlab's built-in XML-processing functions have several undocumented features that can be used by Java-savvy users...
  4. Inactive Control Tooltips & Event Chaining Inactive Matlab uicontrols cannot normally display their tooltips. This article shows how to do this with a combination of undocumented Matlab and Java hacks....
 
]]>
In this day and age, applications are expected to communicate with users by sending email/text messages to alert them about applicative events (“IBM stock purchased @$99.99” or “House is on fire!”). Matlab has included the sendmail function to handle this for many years. Unfortunately, sendmail requires some tweaking to be useful on all but the most basic/insecure mail servers. Today’s post will hopefully fill the missing gaps.

None of the information I’ll present today is really new – it was all there already if you just knew what to search for online. But hopefully today’s post will concentrate all these loose ends in a single place, so it may have some value:

Using a secure mail server

All modern mail servers use end-to-end TLS/SSL encryption. The sendmail function needs extra configuration to handle such connections, since it is configured for a non-encrypted connection by default. Here’s the code that does this for gmail, using SMTP server smtp.gmail.com and default port #465 (for other SMTP servers, see here):

setpref('Internet', 'SMTP_Server',   'smtp.gmail.com');
setpref('Internet', 'SMTP_Username', username);
setpref('Internet', 'SMTP_Password', password);
 
props = java.lang.System.getProperties;
props.setProperty('mail.smtp.auth',                'true');  % Note: 'true' as a string, not a logical value!
props.setProperty('mail.smtp.starttls.enable',     'true');  % Note: 'true' as a string, not a logical value!
props.setProperty('mail.smtp.socketFactory.port',  '465');   % Note: '465'  as a string, not a numeric value!
props.setProperty('mail.smtp.socketFactory.class', 'javax.net.ssl.SSLSocketFactory');
 
sendmail(recipient, title, body, attachments);  % e.g., sendmail('recipient@gmail.com', 'Hello world', 'What a nice day!', 'C:\images\sun.jpg')

All this is not enough to enable Matlab to connect to gmail’s SMTP servers. In addition, we need to set the Google account to allow access from “less secure apps” (details, direct link). Without this, Google will not allow Matlab to relay emails. Other mail servers may require similar server-side account configurations to enable Matlab’s access.

Note: This code snippet uses a bit of Java as you can see. Under the hood, all networking code in Matlab relies on Java, and sendmail is no exception. For some reason that I don’t fully understand, MathWorks chose to label the feature of using sendmail with secure mail servers as a feature that relies on “undocumented commands” and is therefore not listed in sendmail‘s documentation. Considering the fact that all modern mail servers are secure, this seems to make sendmail rather useless without the undocumented extension. I assume that TMW are well aware of this, which is the reason they posted a partial documentation in the form of an official tech-support answer. I hope that one day MathWorks will incorporate it into sendmail as optional input args, so that using sendmail with secure servers would become fully documented and officially supported.

Emailing multiple recipients

To specify multiple email recipients, it is not enough to set sendmail‘s recipient input arg to a string with , or ; delimiters. Instead, we need to provide a cell array of individual recipient strings. For example:

sendmail({'recipient1@gmail.com','recipient2@gmail.com'}, 'Hello world', 'What a nice day!')

Note: this feature is actually fully documented in sendmail‘s doc-page, but for some reason I see that some users are not aware of it (to which it might be said: RTFM!).

Sending text messages

With modern smartphones, text (SMS) messages have become rather outdated, as most users get push notifications of incoming emails. Still, for some users text messages may still be a useful. To send such messages, all we need is to determine our mobile carrier’s email gateway for SMS messages, and send a simple text message to that email address. For example, to send a text message to T-Mobile number 123-456-7890 in the US, simply email the message to 1234567890@tmomail.net (details).

Ke Feng posted a nice Matlab File Exchange utility that wraps this messaging for a wide variety of US carriers.

User configuration panel

Many GUI programs contain configuration panels/tabs/windows. Enabling the user to set up their own email provider is a typical use-case for such a configuration. Naturally, you’d want your config panel not to display plain-text password, nor non-integer port numbers. You’d also want the user to be able to test the email connection.

Here’s a sample implementation for such a panel that I implemented for a recent project – I plan to discuss the implementation details of the password and port (spinner) controls in my next post, so stay tuned:

User configuration of emails in Matlab GUI (click to zoom-in)
User configuration of emails in Matlab GUI (click to zoom-in)

]]>
https://undocumentedmatlab.com/blog/sending-email-text-messages-from-matlab/feed 5
Afterthoughts on implicit expansionhttps://undocumentedmatlab.com/blog/afterthoughts-on-implicit-expansion https://undocumentedmatlab.com/blog/afterthoughts-on-implicit-expansion#comments Wed, 30 Nov 2016 20:28:44 +0000 http://undocumentedmatlab.com/?p=6750
 
Related posts:
  1. tic / toc – undocumented option Matlab's built-in tic/toc functions have an undocumented option enabling multiple nested clockings...
  2. Plot LimInclude properties The plot objects' XLimInclude, YLimInclude, ZLimInclude, ALimInclude and CLimInclude properties are an important feature, that has both functional and performance implications....
  3. Matrix processing performance Matrix operations performance is affected by internal subscriptions in a counter-intuitive way....
  4. Performance: accessing handle properties Handle object property access (get/set) performance can be significantly improved using dot-notation. ...
 
]]>
Matlab release R2016b introduced implicit arithmetic expansion, which is a great and long-awaited natural expansion of Matlab’s arithmetic syntax (if you are still unaware of this or what it means, now would be a good time to read about it). This is a well-documented new feature. The reason for today’s post is that this new feature contains an undocumented aspect that should very well have been documented and even highlighted.

The undocumented aspect that I’m referring to is the fact that code that until R2016a produced an error, in R2016b produces a valid result:

% R2016a
>> [1:5] + [1:3]'
Error using  + 
Matrix dimensions must agree.
 
% R2016b
>> [1:5] + [1:3]'
ans =
     2     3     4     5     6
     3     4     5     6     7
     4     5     6     7     8

This incompatibility is indeed documented, but not where it matters most (read on).

I first discovered this feature by chance when trying to track down a very strange phenomenon with client code that produced different numeric results on R2015b and earlier, compared to R2016a Pre-release. After some debugging the problem was traced to a code snippet in the client’s code that looked something like this (simplified):

% Ensure compatible input data
try
    dataA + dataB;  % this will (?) error if dataA, dataB are incompatible
catch
    dataB = dataB';
end

The code snippet relied on the fact that incompatible data (row vs. col) would error when combined, as it did up to R2015b. But in R2016a Pre-release it just gave a valid numeric matrix, which caused numerically incorrect results downstream in the code. The program never crashed, so everything appeared to be in order, it just gave different numeric results. I looked at the release notes and none of the mentioned release incompatibilities appeared relevant. It took me quite some time, using side-by-side step-by-step debugging on two separate instances of Matlab (R2015b and R2016aPR) to trace the problem to this new feature.

This implicit expansion feature was removed from the official R2016a release for performance reasons. This was apparently fixed in time for R2016b’s release.

I’m totally in favor of this great new feature, don’t get me wrong. I’ve been an ardent user of bsxfun for many years and (unlike many) have even grown fond of it, but I still find the new feature to be better. I use it wherever there is no significant performance penalty, a need to support older Matlab releases, or a possibility of incorrect results due to dimensional mismatch.

So what’s my point?

What I am concerned about is that I have not seen the new feature highlighted as a potential backward compatibility issue in the documentation or the release notes. Issues of far lesser importance are clearly marked for their backward incompatibility in the release notes, but not this important major change. A simple marking of the new feature with the warning icon () and in the “Functionality being removed or changed” section would have saved my client and me a lot of time and frustration.

MathWorks are definitely aware of the potential problems that the new feature might cause in rare use cases such as this. As Steve Eddins recently noted, there were plenty of internal discussions about this very thing. MathWorks were careful to ensure that the feature’s benefits far outweigh its risks (and I concur). But this also highlights the fact that MathWorks were fully aware that in some rare cases it might indeed break existing code. For those cases, I believe that they should have clearly marked the incompatibility implications in the release notes and elsewhere.

I have several clients who scour Matlab’s release notes before each release, trying to determine the operational risk of a Matlab upgrade. Having a program that returns different results in R2016b compared to R2016a, without being aware of this risk, is simply unacceptable to them, and leaves users with a disinclination to upgrade Matlab, to MathWorks’ detriment.

MathWorks in general are taking a very serious methodical approach to compatibility issues, and are clearly investing a lot of energy in this (a recent example). It’s too bad that sometimes this chain is broken. I find it a pity, and think that this can still be corrected in the online doc pages. If and when this is fixed, I’ll be happy to post an addendum here.

In my humble opinion from the backbenches, increasing the transparency on compatibility issues and open bugs will increase user confidence and result in greater adoption and upgrades of Matlab. Just my 2 cents…

Addendum December 27, 2016:

Today MathWorks added the following compatibility warning to the release notes (R2016b, Mathematics section, first item) – thanks for listening MathWorks :-)

MathWorks compatibility warning

]]>
https://undocumentedmatlab.com/blog/afterthoughts-on-implicit-expansion/feed 12
Speeding up Matlab-JDBC SQL querieshttps://undocumentedmatlab.com/blog/speeding-up-matlab-jdbc-sql-queries https://undocumentedmatlab.com/blog/speeding-up-matlab-jdbc-sql-queries#comments Wed, 16 Nov 2016 11:43:17 +0000 http://undocumentedmatlab.com/?p=6742
 
Related posts:
  1. Javacomponent background color This article explains how to align Java component background color with a Matlab color....
  2. Types of undocumented Matlab aspects This article lists the different types of undocumented/unsupported/hidden aspects in Matlab...
  3. Pause for the better Java's thread sleep() function is much more accurate than Matlab's pause() function. ...
  4. Explicit multi-threading in Matlab part 1 Explicit multi-threading can be achieved in Matlab by a variety of simple means. ...
 
]]>
Many of my consulting projects involve interfacing a Matlab program to an SQL database. In such cases, using MathWorks’ Database Toolbox is a viable solution. Users who don’t have the toolbox can also easily connect directly to the database using either the standard ODBC bridge (which is horrible for performance and stability), or a direct JDBC connection (which is also what the Database Toolbox uses under the hood). I explained this Matlab-JDBC interface in detail in chapter 2 of my Matlab-Java programming book. A bare-bones implementation of an SQL SELECT query follows (data update queries are a bit different and will not be discussed here):

% Load the appropriate JDBC driver class into Matlab's memory
% (but not directly, to bypass JIT pre-processing - we must do it in run-time!)
driver = eval('com.mysql.jdbc.Driver');  % or com.microsoft.sqlserver.jdbc.SQLServerDriver or whatever
 
% Connect to DB
dbPort = '3306'; % mySQL=3306; SQLServer=1433; Oracle=...
connectionStr = ['jdbc:mysql://' dbURL ':' dbPort '/' schemaName];  % or ['jdbc:sqlserver://' dbURL ':' dbPort ';database=' schemaName ';'] or whatever
dbConnObj = java.sql.DriverManager.getConnection(connectionStr, username, password);
 
% Send an SQL query statement to the DB and get the ResultSet
stmt = dbConnObj.createStatement(java.sql.ResultSet.TYPE_SCROLL_INSENSITIVE, java.sql.ResultSet.CONCUR_READ_ONLY);
try stmt.setFetchSize(1000); catch, end  % the default fetch size is ridiculously small in many DBs
rs = stmt.executeQuery(sqlQueryStr);
 
% Get the column names and data-types from the ResultSet's meta-data
MetaData = rs.getMetaData;
numCols = MetaData.getColumnCount;
data = cell(0,numCols);  % initialize
for colIdx = numCols : -1 : 1
    ColumnNames{colIdx} = char(MetaData.getColumnLabel(colIdx));
    ColumnType{colIdx}  = char(MetaData.getColumnClassName(colIdx));  % http://docs.oracle.com/javase/7/docs/api/java/sql/Types.html
end
ColumnType = regexprep(ColumnType,'.*\.','');
 
% Get the data from the ResultSet into a Matlab cell array
rowIdx = 1;
while rs.next  % loop over all ResultSet rows (records)
    for colIdx = 1 : numCols  % loop over all columns in the row
        switch ColumnType{colIdx}
            case {'Float','Double'}
                data{rowIdx,colIdx} = rs.getDouble(colIdx);
            case {'Long','Integer','Short','BigDecimal'}
                data{rowIdx,colIdx} = double(rs.getDouble(colIdx));
            case 'Boolean'
                data{rowIdx,colIdx} = logical(rs.getBoolean(colIdx));
            otherwise %case {'String','Date','Time','Timestamp'}
                data{rowIdx,colIdx} = char(rs.getString(colIdx));
        end
    end
    rowIdx = rowIdx + 1;
end
 
% Close the connection and clear resources
try rs.close();   catch, end
try stmt.close(); catch, end
try dbConnObj.closeAllStatements(); catch, end
try dbConnObj.close(); catch, end  % comment this to keep the dbConnObj open and reuse it for subsequent queries

Naturally, in a real-world implementation you also need to handle database timeouts and various other errors, handle data-manipulation queries (not just SELECTs), etc.

Anyway, this works well in general, but when you try to fetch a ResultSet that has many thousands of records you start to feel the pain – The SQL statement may execute much faster on the DB server (the time it takes for the stmt.executeQuery call), yet the subsequent double-loop processing to fetch the data from the Java ResultSet object into a Matlab cell array takes much longer.

In one of my recent projects, performance was of paramount importance, and the DB query speed from the code above was simply not good enough. You might think that this was due to the fact that the data cell array is not pre-allocated, but this turns out to be incorrect: the speed remains nearly unaffected when you pre-allocate data properly. It turns out that the main problem is due to Matlab’s non-negligible overhead in calling methods of Java objects. Since the JDBC interface only enables retrieving a single data item at a time (in other words, bulk retrieval is not possible), we have a double loop over all the data’s rows and columns, in each case calling the appropriate Java method to retrieve the data based on the column’s type. The Java methods themselves are extremely efficient, but when you add Matlab’s invocation overheads the total processing time is much much slower.

So what can be done? As Andrew Janke explained in much detail, we basically need to push our double loop down into the Java level, so that Matlab receives arrays of primitive values, which can then be processed in a vectorized manner in Matlab.

So let’s create a simple Java class to do this:

// Copyright (c) Yair Altman UndocumentedMatlab.com
import java.sql.ResultSet;
import java.sql.ResultSetMetaData;
import java.sql.SQLException;
import java.sql.Types;
 
public class JDBC_Fetch {
 
	public static int DEFAULT_MAX_ROWS = 100000;   // default cache size = 100K rows (if DB does not support non-forward-only ResultSets)
 
	public static Object[] getData(ResultSet rs) throws SQLException {
		try {
			if (rs.last()) {  // data is available
				int numRows = rs.getRow();    // row # of the last row
				rs.beforeFirst();             // get back to the top of the ResultSet
				return getData(rs, numRows);  // fetch the data
			} else {  // no data in the ResultSet
				return null;
			}
		} catch (Exception e) {
			return getData(rs, DEFAULT_MAX_ROWS);
		}
	}
 
	public static Object[] getData(ResultSet rs, int maxRows) throws SQLException {
		// Read column number and types from the ResultSet's meta-data
		ResultSetMetaData metaData = rs.getMetaData();
		int numCols = metaData.getColumnCount();
		int[] colTypes = new int[numCols+1];
		int numDoubleCols = 0;
		int numBooleanCols = 0;
		int numStringCols = 0;
		for (int colIdx = 1; colIdx <= numCols; colIdx++) {
			int colType = metaData.getColumnType(colIdx);
			switch (colType) {
				case Types.FLOAT:
				case Types.DOUBLE:
				case Types.REAL:
					colTypes[colIdx] = 1;  // double
					numDoubleCols++;
					break;
				case Types.DECIMAL:
				case Types.INTEGER:
				case Types.TINYINT:
				case Types.SMALLINT:
				case Types.BIGINT:
					colTypes[colIdx] = 1;  // double
					numDoubleCols++;
					break;
				case Types.BIT:
				case Types.BOOLEAN:
					colTypes[colIdx] = 2;  // boolean
					numBooleanCols++;
					break;
				default: // 'String','Date','Time','Timestamp',...
					colTypes[colIdx] = 3;  // string
					numStringCols++;
			}
		}
 
		// Loop over all ResultSet rows, reading the data into the 2D matrix caches
		int rowIdx = 0;
		double [][] dataCacheDouble  = new double [numDoubleCols] [maxRows];
		boolean[][] dataCacheBoolean = new boolean[numBooleanCols][maxRows];
		String [][] dataCacheString  = new String [numStringCols] [maxRows];
		while (rs.next() && rowIdx < maxRows) {
			int doubleColIdx = 0;
			int booleanColIdx = 0;
			int stringColIdx = 0;
			for (int colIdx = 1; colIdx <= numCols; colIdx++) {
				try {
					switch (colTypes[colIdx]) {
						case 1:  dataCacheDouble[doubleColIdx++][rowIdx]   = rs.getDouble(colIdx);   break;  // numeric
						case 2:  dataCacheBoolean[booleanColIdx++][rowIdx] = rs.getBoolean(colIdx);  break;  // boolean
						default: dataCacheString[stringColIdx++][rowIdx]   = rs.getString(colIdx);   break;  // string
					}
				} catch (Exception e) {
					System.out.println(e);
					System.out.println(" in row #" + rowIdx + ", col #" + colIdx);
				}
			}
			rowIdx++;
		}
 
		// Return only the actual data in the ResultSet
		int doubleColIdx = 0;
		int booleanColIdx = 0;
		int stringColIdx = 0;
		Object[] data = new Object[numCols];
		for (int colIdx = 1; colIdx <= numCols; colIdx++) {
			switch (colTypes[colIdx]) {
				case 1:   data[colIdx-1] = dataCacheDouble[doubleColIdx++];    break;  // numeric
				case 2:   data[colIdx-1] = dataCacheBoolean[booleanColIdx++];  break;  // boolean
				default:  data[colIdx-1] = dataCacheString[stringColIdx++];            // string
			}
		}
		return data;
	}
}

So now we have a JDBC_Fetch class that we can use in our Matlab code, replacing the slow double loop with a single call to JDBC_Fetch.getData(), followed by vectorized conversion into a Matlab cell array (matrix):

% Get the data from the ResultSet using the JDBC_Fetch wrapper
data = cell(JDBC_Fetch.getData(rs));
for colIdx = 1 : numCols
   switch ColumnType{colIdx}
      case {'Float','Double'}
          data{colIdx} = num2cell(data{colIdx});
      case {'Long','Integer','Short','BigDecimal'}
          data{colIdx} = num2cell(data{colIdx});
      case 'Boolean'
          data{colIdx} = num2cell(data{colIdx});
      otherwise %case {'String','Date','Time','Timestamp'}
          %data{colIdx} = cell(data{colIdx});  % no need to do anything here!
   end
end
data = [data{:}];

On my specific program the resulting speedup was 15x (this is not a typo: 15 times faster). My fetches are no longer limited by the Matlab post-processing, but rather by the DB’s processing of the SQL statement (where DB indexes, clustering, SQL tuning etc. come into play).

Additional speedups can be achieved by parsing dates at the Java level (rather than returning strings), as well as several other tweaks in the Java and Matlab code (refer to Andrew Janke’s post for some ideas). But certainly the main benefit (the 80% of the gain that was achieved in 20% of the worktime) is due to the above push of the main double processing loop down into the Java level, leaving Matlab with just a single Java call to JDBC_Fetch.

Many additional ideas of speeding up database queries and Matlab programs in general can be found in my second book, Accelerating Matlab Performance.

If you’d like me to help you speed up your Matlab program, please email me (altmany at gmail), or fill out the query form on my consulting page.

]]>
https://undocumentedmatlab.com/blog/speeding-up-matlab-jdbc-sql-queries/feed 6