Low risk of breaking in future versions – Undocumented Matlab https://undocumentedmatlab.com Charting Matlab's unsupported hidden underbelly Fri, 20 Oct 2017 09:57:44 +0000 en-US hourly 1 https://wordpress.org/?v=4.4.1 Tips for accelerating Matlab performancehttps://undocumentedmatlab.com/blog/tips-for-accelerating-matlab-performance https://undocumentedmatlab.com/blog/tips-for-accelerating-matlab-performance#comments Thu, 05 Oct 2017 18:25:06 +0000 http://undocumentedmatlab.com/?p=7099
 
Related posts:
  1. Performance: scatter vs. line In many circumstances, the line function can generate visually-identical plots as the scatter function, much faster...
  2. Plot LimInclude properties The plot objects' XLimInclude, YLimInclude, ZLimInclude, ALimInclude and CLimInclude properties are an important feature, that has both functional and performance implications....
  3. Plot performance Undocumented inner plot mechanisms can significantly improve plotting performance ...
  4. Performance: accessing handle properties Handle object property access (get/set) performance can be significantly improved using dot-notation. ...
 
]]>
I’m proud to report that MathWorks has recently posted my article “Tips for Accelerating MATLAB Performance” in their latest newsletter digest (September 2017). This article is an updated and expanded version of my post about consulting work that I did for the Crustal Dynamics Research Group at Harvard University, where I helped speed-up a complex Matlab-based GUI by a factor of 50-500 (depending on the specific feature).

Crustal dynamics visualization GUI

Crustal dynamics visualization GUI

Featuring an article on the official newsletter by a non-MathWorker is rare. Doing this with someone like myself who has a reputation for undocumented aspects, and a consultancy business that potentially competes with theirs, is certainly not obvious. I take this to be a sign that despite the possible drawbacks of publishing my article, MathWorks felt that it provided enough value to the Matlab user community to merit the risk. I applaud MathWorks for this, and thank them for the opportunity of being featured in their official newsletter and conferences. I do not take it for granted in the least.

The newsletter article provides multiple ideas of improving the run-time performance for file I/O and graphics. Many additional techniques for improving Matlab’s performance can be found under the Performance tag in this blog, as well as in my book “Accelerating MATLAB Performance” (CRC Press, 2014, ISBN 978-1482211290).

Next week I will present live online webinars about various ways to improve Matlab’s run-time performance:

These live webinars will be 3.5 hours long, starting at 10am EDT (7am PDT, 3pm UK, 4pm CET, 7:30pm IST, time in your local timezone), with a short break in the middle. The presentations content will be based on onsite training courses that I presented at multiple client locations (details). A recording of the webinars will be available for anyone who cannot join the live events.

 Email me if you would like additional information on the webinars or my consulting, or to inquire regarding an onsite training course.

]]>
https://undocumentedmatlab.com/blog/tips-for-accelerating-matlab-performance/feed 2
Faster csvwrite/dlmwritehttps://undocumentedmatlab.com/blog/faster-csvwrite-dlmwrite https://undocumentedmatlab.com/blog/faster-csvwrite-dlmwrite#comments Tue, 03 Oct 2017 15:00:05 +0000 http://undocumentedmatlab.com/?p=7080
 
Related posts:
  1. Parsing mlint (Code Analyzer) output The Matlab Code Analyzer (mlint) has a lot of undocumented functionality just waiting to be used. ...
  2. Explicit multi-threading in Matlab part 3 Matlab performance can be improved by employing POSIX threads in C/C++ code. ...
  3. Plot LimInclude properties The plot objects' XLimInclude, YLimInclude, ZLimInclude, ALimInclude and CLimInclude properties are an important feature, that has both functional and performance implications....
  4. Preallocation performance Preallocation is a standard Matlab speedup technique. Still, it has several undocumented aspects. ...
 
]]>
Matlab’s builtin functions for exporting (saving) data to output files are quite sub-optimal (as in slowwwwww…). I wrote a few posts about this in the past (how to improve fwrite performance, and save performance). Today I extend the series by showing how we can improve the performance of delimited text output, for example comma-separated (CSV) or tab-separated (TSV/TXT) files.

The basic problem is that Matlab’s dlmwrite function, which can either be used directly, or via the csvwrite function which calls it internally, is extremely inefficient: It processes each input data value separately, in a non-vectorized loop. In the general (completely non-vectorized) case, each data value is separately converted into a string, and is separately sent to disk (using fprintf). In the specific case of real data values with simple delimiters and formatting, row values are vectorized, but in any case the rows are processed in a non-vectorized loop: A newline character is separately exported at the end of each row, using a separate fprintf call, and this has the effect of flushing the I/O to disk each and every row separately, which is of course disastrous for performance. The output file is indeed originally opened in buffered mode (as I explained in my fprintf performance post), but this only helps for outputs done within the row – the newline output at the end of each row forces an I/O flush regardless of how the file was opened. In general, when you read the short source-code of dlmwrite.m you’ll get the distinct feeling that it was written for correctness and maintainability, and some focus on performance (e.g., the vectorization edge-case). But much more could be done for performance it would seem.

This is where Alex Nazarovsky comes to the rescue.

Alex was so bothered by the slow performance of csvwrite and dlmwrite that he created a C++ (MEX) version that runs about enormously faster (30 times faster on my system). He explains the idea in his blog, and posted it as an open-source utility (mex-writematrix) on GitHub.

Usage of Alex’s utility is very easy:

mex_WriteMatrix(filename, dataMatrix, textFormat, delimiter, writeMode);

where the input arguments are:

  • filename – full path name for file to export
  • dataMatrix – matrix of numeric values to be exported
  • textFormat – format of output text (sprintf format), e.g. '%10.6f'
  • delimiter – delimiter, for example ',' or ';' or char(9) (=tab)
  • writeMode – 'w+' for rewriting file; 'a+' for appending (note the lowercase: uppercase will crash Matlab!)

Here is a sample run on my system, writing a simple CSV file containing 1K-by-1K data values (1M elements, ~12MB text files):

>> data = rand(1000, 1000);  % 1M data values, 8MB in memory, ~12MB on disk
 
>> tic, dlmwrite('temp1.csv', data, 'delimiter',',', 'precision','%10.10f'); toc
Elapsed time is 28.724937 seconds.
 
>> tic, mex_WriteMatrix('temp2.csv', data, '%10.10f', ',', 'w+'); toc   % 30 times faster!
Elapsed time is 0.957256 seconds.

Alex’s mex_WriteMatrix function is faster even in the edge case of simple formatting where dlmwrite uses vectorized mode (in that case, the file is exported in ~1.2 secs by dlmwrite and ~0.9 secs by mex_WriteMatrix, on my system).

Trapping Ctrl-C interrupts

Alex’s mex_WriteMatrix code includes another undocumented trick that could help anyone else who uses a long-running MEX function, namely the ability to stop the MEX execution using Ctrl-C. Using Ctrl-C is normally ignored in MEX code, but Wotao Yin showed how we can use the undocumented utIsInterruptPending() MEX function to monitor for user interrupts using Ctrl-C. For easy reference, here is a copy of Wotao Yin’s usage example (read his webpage for additional details):

/* A demo of Ctrl-C detection in mex-file by Wotao Yin. Jan 29, 2010. */
 
#include "mex.h"
 
#if defined (_WIN32)
    #include <windows.h>
#elif defined (__linux__)
    #include <unistd.h>
#endif
 
#ifdef __cplusplus 
    extern "C" bool utIsInterruptPending();
#else
    extern bool utIsInterruptPending();
#endif
 
void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[]) {
    int count = 0;    
    while(1) {
        #if defined(_WIN32)
            Sleep(1000);        /* Sleep one second */
        #elif defined(__linux__)
            usleep(1000*1000);  /* Sleep one second */
        #endif
 
        mexPrintf("Count = %d\n", count++);  /* print count and increase it by 1 */
        mexEvalString("drawnow;");           /* flush screen output */
 
        if (utIsInterruptPending()) {        /* check for a Ctrl-C event */
            mexPrintf("Ctrl-C Detected. END\n\n");
            return;
        }
        if (count == 10) {
            mexPrintf("Count Reached 10. END\n\n");
            return;
        }
    }
}

Matlab performance webinars

Next week I will present live online webinars about numerous other ways to improve Matlab’s run-time performance:

These live webinars will be 3.5 hours long, starting at 10am EDT (7am PDT, 3pm UK, 4pm CET, 7:30pm IST, time in your local timezone), with a short break in the middle. The presentations content will be based on onsite training courses that I presented at multiple client locations (details). A recording of the webinars will be available for anyone who cannot join the live events.

Additional Matlab performance tips can be found under the Performance tag in this blog, as well as in my book “Accelerating MATLAB Performance“.

 Email me if you would like additional information on the webinars, or an onsite training course, or about my consulting.

]]>
https://undocumentedmatlab.com/blog/faster-csvwrite-dlmwrite/feed 5
Sending HTML emails from Matlabhttps://undocumentedmatlab.com/blog/sending-html-emails-from-matlab https://undocumentedmatlab.com/blog/sending-html-emails-from-matlab#respond Wed, 02 Aug 2017 21:19:42 +0000 http://undocumentedmatlab.com/?p=6986
 
Related posts:
  1. Uitable sorting Matlab's uitables can be sortable using simple undocumented features...
  2. Uitable customization report Matlab's uitable can be customized in many different ways. A detailed report explains how. ...
  3. Types of undocumented Matlab aspects This article lists the different types of undocumented/unsupported/hidden aspects in Matlab...
  4. Multi-line uitable column headers Matlab uitables can present long column headers in multiple lines, for improved readability. ...
 
]]>
A few months ago I wrote about various tricks for sending email/text messages from Matlab. Unfortunately, Matlab only sends text emails by default and provides no documented way to send HTML-formatted emails. Text-only emails are naturally very bland and all mail clients in the past 2 decades support HTML-formatted emails. Today I will show how we can send such HTML emails from Matlab.

A quick recap: Matlab’s sendmail function uses Java (specifically, the standard javax.mail package) to prepare and send emails. The Java classes are extremely powerful and so there is no wonder that Mathworks chose to use them rather than reinventing the wheel. However, Matlab’s sendmail function only uses part of the functionality exposed by these classes (admittedly, the most important parts that deal with the basic mail-sending mechanism), and does not expose external hooks or input args that would enable the user to take full advantage of the more advanced features, HTML formatting included.

Only two small changes are needed in sendmail.m to support HTML formatting:

  1. HTML formatting required calling the message-object’s setContent() method, rather than setText().
  2. We need to specify 'text/html' as part of the message’s encoding

To implement these features, change the following (lines #119-130 in the original sendmail.m file of R2017a, changed lines highlighted):

% Construct the body of the message and attachments.
body = formatText(theMessage);
if numel(attachments) == 0    if ~isempty(charset)        msg.setText(body, charset);
    else
        msg.setText(body);
    end
else
    % Add body text.
    messageBodyPart = MimeBodyPart;
    if ~isempty(charset)        messageBodyPart.setText(body, charset);
    ...

to this (changed lines highlighted):

% Construct the body of the message and attachments.
body = formatText(theMessage);
isHtml = ~isempty(body) && body(1) == '<';  % msg starting with '<' indicates HTMLif isHtml    if isempty(charset)        charset = 'text/html; charset=utf-8';    else        charset = ['text/html; charset=' charset];    endendif numel(attachments) == 0  && ~isHtml    if isHtml        msg.setContent(body, charset);    elseif ~isempty(charset)        msg.setText(body, charset);
    else
        msg.setText(body);
    end
    else
        % Add body text.
        messageBodyPart = MimeBodyPart;
        if isHtml            messageBodyPart.setContent(body, charset);        elseif ~isempty(charset)            messageBodyPart.setText(body, charset);
        ...

In addition, I also found it useful to remove the hard-coded 75-character line-wrapping in text messages. This can be done by changing the following (line #291 in the original sendmail.m file of R2017a):

maxLineLength = 75;

to this:

maxLineLength = inf;  % or some other large numeric value

Deployment

It’s useful to note two alternatives for making these fixes:

  • Making the changes directly in %matlabroot%/toolbox/matlab/iofun/sendmail.m. You will need administrator rights to edit this file. You will also need to redo the fix whenever you install Matlab, either installation on a different machine, or installing a new Matlab release. In general, I discourage changing Matlab’s internal files because it is simply not very maintainable.
  • Copying %matlabroot%/toolbox/matlab/iofun/sendmail.m into a dedicated wrapper function (e.g., sendEmail.m) that has a similar function signature and exists on the Matlab path. This has the benefit of working on multiple Matlab releases, and being copied along with the rest of our m-files when we install our Matlab program on a different computer. The downside is that our wrapper function will be stuck with the version of sendmail.m that we copied into it, and we’d lose any possible improvements that Mathworks may implement in future Matlab releases.

The basic idea for the second alternative, the sendEmail.m wrapper, is something like this (the top highlighted lines are the additions made to the original sendmail.m, with everything placed in sendEmail.m on the Matlab path):

function sendEmail(to,subject,theMessage,attachments)%SENDEMAIL Send e-mail wrapper (with HTML formatting)   sendmail(to,subject,theMessage,attachments); 
% The rest of this file is copied from %matlabroot%/toolbox/matlab/iofun/sendmail.m (with the modifications mentioned above):
function sendmail(to,subject,theMessage,attachments)
%SENDMAIL Send e-mail.
%   SENDMAIL(TO,SUBJECT,MESSAGE,ATTACHMENTS) sends an e-mail.  TO is either a
%   character vector specifying a single address, or a cell array of character vector
...

We would then call the wrapper function as follows:

sendEmail('abc@gmail.com', 'email subject', 'regular text message');     % will send a regular text message
sendEmail('abc@gmail.com', 'email subject', '<b><font color="blue">HTML-formatted</font> <i>message');  % HTML-formatted message

In this case, the code automatically infers HTML formatting based on whether the first character in the message body is a ‘<‘ character. Instead, we could just as easily have passed an additional input argument (isHtml) to our sendEmail wrapper function.

Hopefully, in some future Matlab release Mathworks will be kind enough to enable sending 21st-century HTML-formatted emails without needing such hacks. Until then, note that sendmail.m relies on standard non-GUI Java networking classes, which are expected to be supported far into the future, well after Java-based GUI may cease to be supported in Matlab. For this reason I believe that while it seems a bit tricky, the changes that I outlined in today’s post actually have a low risk of breaking in a future Matlab release.

Do you have some other advanced email feature that you use in your Matlab program by some crafty customization to sendmail? If so, please share it in a comment below.

]]>
https://undocumentedmatlab.com/blog/sending-html-emails-from-matlab/feed 0
Parsing XML stringshttps://undocumentedmatlab.com/blog/parsing-xml-strings https://undocumentedmatlab.com/blog/parsing-xml-strings#comments Wed, 01 Feb 2017 09:52:45 +0000 http://undocumentedmatlab.com/?p=6838
 
Related posts:
  1. Undocumented XML functionality Matlab's built-in XML-processing functions have several undocumented features that can be used by Java-savvy users...
  2. Matlab-Java memory leaks, performance Internal fields of Java objects may leak memory - this article explains how to avoid this without sacrificing performance. ...
  3. Types of undocumented Matlab aspects This article lists the different types of undocumented/unsupported/hidden aspects in Matlab...
  4. Pause for the better Java's thread sleep() function is much more accurate than Matlab's pause() function. ...
 
]]>
I have recently consulted in a project where data was provided in XML strings and needed to be parsed in Matlab memory in an efficient manner (in other words, as quickly as possible). Now granted, XML is rather inefficient in storing data (JSON would be much better for this, for example). But I had to work with the given situation, and that required processing the XML.

I basically had two main alternatives:

  • I could either create a dedicated string-parsing function that searches for a particular pattern within the XML string, or
  • I could use a standard XML-parsing library to create the XML model and then parse its nodes

The first alternative is quite error-prone, since it relies on the exact format of the data in the XML. Since the same data can be represented in multiple equivalent XML ways, making the string-parsing function robust as well as efficient would be challenging. I was lazy expedient, so I chose the second alternative.

Unfortunately, Matlab’s xmlread function only accepts input filenames (of *.xml files), it cannot directly parse XML strings. Yummy!

The obvious and simple solution is to simply write the XML string into a temporary *.xml file, read it with xmlread, and then delete the temp file:

% Store the XML data in a temp *.xml file
filename = [tempname '.xml'];
fid = fopen(filename,'Wt');
fwrite(fid,xmlString);
fclose(fid);
 
% Read the file into an XML model object
xmlTreeObject = xmlread(filename);
 
% Delete the temp file
delete(filename);
 
% Parse the XML model object
...

This works well and we could move on with our short lives. But cases such as this, where a built-in function seems to have a silly limitation, really fire up the investigative reporter in me. I decided to drill into xmlread to discover why it couldn’t parse XML strings directly in memory, without requiring costly file I/O. It turns out that xmlread accepts not just file names as input, but also Java object references (specifically, java.io.File, java.io.InputStream or org.xml.sax.InputSource). In fact, there are quite a few other inputs that we could use, to specify a validation parser etc. – I wrote about this briefly back in 2009 (along with other similar semi-documented input altermatives in xmlwrite and xslt).

In our case, we could simply send xmlread as input a java.io.StringBufferInputStream(xmlString) object (which is an instance of java.io.InputStream) or org.xml.sax.InputSource(java.io.StringReader(xmlString)):

% Read the xml string directly into an XML model object
inputObject = java.io.StringBufferInputStream(xmlString);                % alternative #1
inputObject = org.xml.sax.InputSource(java.io.StringReader(xmlString));  % alternative #2
 
xmlTreeObject = xmlread(inputObject);
 
% Parse the XML model object
...

If we don’t want to depend on undocumented functionality (which might break in some future release, although it has remained unchanged for at least the past decade), and in order to improve performance even further by passing xmlread‘s internal validity checks and processing, we can use xmlread‘s core functionality to parse our XML string directly. We can add a fallback to the standard (fully-documented) functionality, just in case something goes wrong (which is good practice whenever using any undocumented functionality):

try
    % The following avoids the need for file I/O:
    inputObject = java.io.StringBufferInputStream(xmlString);  % or: org.xml.sax.InputSource(java.io.StringReader(xmlString))
    try
        % Parse the input data directly using xmlread's core functionality
        parserFactory = javaMethod('newInstance','javax.xml.parsers.DocumentBuilderFactory');
        p = javaMethod('newDocumentBuilder',parserFactory);
        xmlTreeObject = p.parse(inputObject);
    catch
        % Use xmlread's semi-documented inputObject input feature
        xmlTreeObject = xmlread(inputObject);
    end
catch
    % Fallback to standard xmlread usage, using a temporary XML file:
 
    % Store the XML data in a temp *.xml file
    filename = [tempname '.xml'];
    fid = fopen(filename,'Wt');
    fwrite(fid,xmlString);
    fclose(fid);
 
    % Read the file into an XML model object
    xmlTreeObject = xmlread(filename);
 
    % Delete the temp file
    delete(filename);
end
 
% Parse the XML model object
...
]]>
https://undocumentedmatlab.com/blog/parsing-xml-strings/feed 4
Afterthoughts on implicit expansionhttps://undocumentedmatlab.com/blog/afterthoughts-on-implicit-expansion https://undocumentedmatlab.com/blog/afterthoughts-on-implicit-expansion#comments Wed, 30 Nov 2016 20:28:44 +0000 http://undocumentedmatlab.com/?p=6750
 
Related posts:
  1. tic / toc – undocumented option Matlab's built-in tic/toc functions have an undocumented option enabling multiple nested clockings...
  2. Plot LimInclude properties The plot objects' XLimInclude, YLimInclude, ZLimInclude, ALimInclude and CLimInclude properties are an important feature, that has both functional and performance implications....
  3. Matrix processing performance Matrix operations performance is affected by internal subscriptions in a counter-intuitive way....
  4. Performance: accessing handle properties Handle object property access (get/set) performance can be significantly improved using dot-notation. ...
 
]]>
Matlab release R2016b introduced implicit arithmetic expansion, which is a great and long-awaited natural expansion of Matlab’s arithmetic syntax (if you are still unaware of this or what it means, now would be a good time to read about it). This is a well-documented new feature. The reason for today’s post is that this new feature contains an undocumented aspect that should very well have been documented and even highlighted.

The undocumented aspect that I’m referring to is the fact that code that until R2016a produced an error, in R2016b produces a valid result:

% R2016a
>> [1:5] + [1:3]'
Error using  + 
Matrix dimensions must agree.
 
% R2016b
>> [1:5] + [1:3]'
ans =
     2     3     4     5     6
     3     4     5     6     7
     4     5     6     7     8

This incompatibility is indeed documented, but not where it matters most (read on).

I first discovered this feature by chance when trying to track down a very strange phenomenon with client code that produced different numeric results on R2015b and earlier, compared to R2016a Pre-release. After some debugging the problem was traced to a code snippet in the client’s code that looked something like this (simplified):

% Ensure compatible input data
try
    dataA + dataB;  % this will (?) error if dataA, dataB are incompatible
catch
    dataB = dataB';
end

The code snippet relied on the fact that incompatible data (row vs. col) would error when combined, as it did up to R2015b. But in R2016a Pre-release it just gave a valid numeric matrix, which caused numerically incorrect results downstream in the code. The program never crashed, so everything appeared to be in order, it just gave different numeric results. I looked at the release notes and none of the mentioned release incompatibilities appeared relevant. It took me quite some time, using side-by-side step-by-step debugging on two separate instances of Matlab (R2015b and R2016aPR) to trace the problem to this new feature.

This implicit expansion feature was removed from the official R2016a release for performance reasons. This was apparently fixed in time for R2016b’s release.

I’m totally in favor of this great new feature, don’t get me wrong. I’ve been an ardent user of bsxfun for many years and (unlike many) have even grown fond of it, but I still find the new feature to be better. I use it wherever there is no significant performance penalty, a need to support older Matlab releases, or a possibility of incorrect results due to dimensional mismatch.

So what’s my point?

What I am concerned about is that I have not seen the new feature highlighted as a potential backward compatibility issue in the documentation or the release notes. Issues of far lesser importance are clearly marked for their backward incompatibility in the release notes, but not this important major change. A simple marking of the new feature with the warning icon () and in the “Functionality being removed or changed” section would have saved my client and me a lot of time and frustration.

MathWorks are definitely aware of the potential problems that the new feature might cause in rare use cases such as this. As Steve Eddins recently noted, there were plenty of internal discussions about this very thing. MathWorks were careful to ensure that the feature’s benefits far outweigh its risks (and I concur). But this also highlights the fact that MathWorks were fully aware that in some rare cases it might indeed break existing code. For those cases, I believe that they should have clearly marked the incompatibility implications in the release notes and elsewhere.

I have several clients who scour Matlab’s release notes before each release, trying to determine the operational risk of a Matlab upgrade. Having a program that returns different results in R2016b compared to R2016a, without being aware of this risk, is simply unacceptable to them, and leaves users with a disinclination to upgrade Matlab, to MathWorks’ detriment.

MathWorks in general are taking a very serious methodical approach to compatibility issues, and are clearly investing a lot of energy in this (a recent example). It’s too bad that sometimes this chain is broken. I find it a pity, and think that this can still be corrected in the online doc pages. If and when this is fixed, I’ll be happy to post an addendum here.

In my humble opinion from the backbenches, increasing the transparency on compatibility issues and open bugs will increase user confidence and result in greater adoption and upgrades of Matlab. Just my 2 cents…

Addendum December 27, 2016:

Today MathWorks added the following compatibility warning to the release notes (R2016b, Mathematics section, first item) – thanks for listening MathWorks :-)

MathWorks compatibility warning

]]>
https://undocumentedmatlab.com/blog/afterthoughts-on-implicit-expansion/feed 12
Speeding up Matlab-JDBC SQL querieshttps://undocumentedmatlab.com/blog/speeding-up-matlab-jdbc-sql-queries https://undocumentedmatlab.com/blog/speeding-up-matlab-jdbc-sql-queries#comments Wed, 16 Nov 2016 11:43:17 +0000 http://undocumentedmatlab.com/?p=6742
 
Related posts:
  1. Javacomponent background color This article explains how to align Java component background color with a Matlab color....
  2. Types of undocumented Matlab aspects This article lists the different types of undocumented/unsupported/hidden aspects in Matlab...
  3. Pause for the better Java's thread sleep() function is much more accurate than Matlab's pause() function. ...
  4. Explicit multi-threading in Matlab part 1 Explicit multi-threading can be achieved in Matlab by a variety of simple means. ...
 
]]>
Many of my consulting projects involve interfacing a Matlab program to an SQL database. In such cases, using MathWorks’ Database Toolbox is a viable solution. Users who don’t have the toolbox can also easily connect directly to the database using either the standard ODBC bridge (which is horrible for performance and stability), or a direct JDBC connection (which is also what the Database Toolbox uses under the hood). I explained this Matlab-JDBC interface in detail in chapter 2 of my Matlab-Java programming book. A bare-bones implementation of an SQL SELECT query follows (data update queries are a bit different and will not be discussed here):

% Load the appropriate JDBC driver class into Matlab's memory
% (but not directly, to bypass JIT pre-processing - we must do it in run-time!)
driver = eval('com.mysql.jdbc.Driver');  % or com.microsoft.sqlserver.jdbc.SQLServerDriver or whatever
 
% Connect to DB
dbPort = '3306'; % mySQL=3306; SQLServer=1433; Oracle=...
connectionStr = ['jdbc:mysql://' dbURL ':' dbPort '/' schemaName];  % or ['jdbc:sqlserver://' dbURL ':' dbPort ';database=' schemaName ';'] or whatever
dbConnObj = java.sql.DriverManager.getConnection(connectionStr, username, password);
 
% Send an SQL query statement to the DB and get the ResultSet
stmt = dbConnObj.createStatement(java.sql.ResultSet.TYPE_SCROLL_INSENSITIVE, java.sql.ResultSet.CONCUR_READ_ONLY);
try stmt.setFetchSize(1000); catch, end  % the default fetch size is ridiculously small in many DBs
rs = stmt.executeQuery(sqlQueryStr);
 
% Get the column names and data-types from the ResultSet's meta-data
MetaData = rs.getMetaData;
numCols = MetaData.getColumnCount;
data = cell(0,numCols);  % initialize
for colIdx = numCols : -1 : 1
    ColumnNames{colIdx} = char(MetaData.getColumnLabel(colIdx));
    ColumnType{colIdx}  = char(MetaData.getColumnClassName(colIdx));  % http://docs.oracle.com/javase/7/docs/api/java/sql/Types.html
end
ColumnType = regexprep(ColumnType,'.*\.','');
 
% Get the data from the ResultSet into a Matlab cell array
rowIdx = 1;
while rs.next  % loop over all ResultSet rows (records)
    for colIdx = 1 : numCols  % loop over all columns in the row
        switch ColumnType{colIdx}
            case {'Float','Double'}
                data{rowIdx,colIdx} = rs.getDouble(colIdx);
            case {'Long','Integer','Short','BigDecimal'}
                data{rowIdx,colIdx} = double(rs.getDouble(colIdx));
            case 'Boolean'
                data{rowIdx,colIdx} = logical(rs.getBoolean(colIdx));
            otherwise %case {'String','Date','Time','Timestamp'}
                data{rowIdx,colIdx} = char(rs.getString(colIdx));
        end
    end
    rowIdx = rowIdx + 1;
end
 
% Close the connection and clear resources
try rs.close();   catch, end
try stmt.close(); catch, end
try dbConnObj.closeAllStatements(); catch, end
try dbConnObj.close(); catch, end  % comment this to keep the dbConnObj open and reuse it for subsequent queries

Naturally, in a real-world implementation you also need to handle database timeouts and various other errors, handle data-manipulation queries (not just SELECTs), etc.

Anyway, this works well in general, but when you try to fetch a ResultSet that has many thousands of records you start to feel the pain – The SQL statement may execute much faster on the DB server (the time it takes for the stmt.executeQuery call), yet the subsequent double-loop processing to fetch the data from the Java ResultSet object into a Matlab cell array takes much longer.

In one of my recent projects, performance was of paramount importance, and the DB query speed from the code above was simply not good enough. You might think that this was due to the fact that the data cell array is not pre-allocated, but this turns out to be incorrect: the speed remains nearly unaffected when you pre-allocate data properly. It turns out that the main problem is due to Matlab’s non-negligible overhead in calling methods of Java objects. Since the JDBC interface only enables retrieving a single data item at a time (in other words, bulk retrieval is not possible), we have a double loop over all the data’s rows and columns, in each case calling the appropriate Java method to retrieve the data based on the column’s type. The Java methods themselves are extremely efficient, but when you add Matlab’s invocation overheads the total processing time is much much slower.

So what can be done? As Andrew Janke explained in much detail, we basically need to push our double loop down into the Java level, so that Matlab receives arrays of primitive values, which can then be processed in a vectorized manner in Matlab.

So let’s create a simple Java class to do this:

// Copyright (c) Yair Altman UndocumentedMatlab.com
import java.sql.ResultSet;
import java.sql.ResultSetMetaData;
import java.sql.SQLException;
import java.sql.Types;
 
public class JDBC_Fetch {
 
	public static int DEFAULT_MAX_ROWS = 100000;   // default cache size = 100K rows (if DB does not support non-forward-only ResultSets)
 
	public static Object[] getData(ResultSet rs) throws SQLException {
		try {
			if (rs.last()) {  // data is available
				int numRows = rs.getRow();    // row # of the last row
				rs.beforeFirst();             // get back to the top of the ResultSet
				return getData(rs, numRows);  // fetch the data
			} else {  // no data in the ResultSet
				return null;
			}
		} catch (Exception e) {
			return getData(rs, DEFAULT_MAX_ROWS);
		}
	}
 
	public static Object[] getData(ResultSet rs, int maxRows) throws SQLException {
		// Read column number and types from the ResultSet's meta-data
		ResultSetMetaData metaData = rs.getMetaData();
		int numCols = metaData.getColumnCount();
		int[] colTypes = new int[numCols+1];
		int numDoubleCols = 0;
		int numBooleanCols = 0;
		int numStringCols = 0;
		for (int colIdx = 1; colIdx <= numCols; colIdx++) {
			int colType = metaData.getColumnType(colIdx);
			switch (colType) {
				case Types.FLOAT:
				case Types.DOUBLE:
				case Types.REAL:
					colTypes[colIdx] = 1;  // double
					numDoubleCols++;
					break;
				case Types.DECIMAL:
				case Types.INTEGER:
				case Types.TINYINT:
				case Types.SMALLINT:
				case Types.BIGINT:
					colTypes[colIdx] = 1;  // double
					numDoubleCols++;
					break;
				case Types.BIT:
				case Types.BOOLEAN:
					colTypes[colIdx] = 2;  // boolean
					numBooleanCols++;
					break;
				default: // 'String','Date','Time','Timestamp',...
					colTypes[colIdx] = 3;  // string
					numStringCols++;
			}
		}
 
		// Loop over all ResultSet rows, reading the data into the 2D matrix caches
		int rowIdx = 0;
		double [][] dataCacheDouble  = new double [numDoubleCols] [maxRows];
		boolean[][] dataCacheBoolean = new boolean[numBooleanCols][maxRows];
		String [][] dataCacheString  = new String [numStringCols] [maxRows];
		while (rs.next() && rowIdx < maxRows) {
			int doubleColIdx = 0;
			int booleanColIdx = 0;
			int stringColIdx = 0;
			for (int colIdx = 1; colIdx <= numCols; colIdx++) {
				try {
					switch (colTypes[colIdx]) {
						case 1:  dataCacheDouble[doubleColIdx++][rowIdx]   = rs.getDouble(colIdx);   break;  // numeric
						case 2:  dataCacheBoolean[booleanColIdx++][rowIdx] = rs.getBoolean(colIdx);  break;  // boolean
						default: dataCacheString[stringColIdx++][rowIdx]   = rs.getString(colIdx);   break;  // string
					}
				} catch (Exception e) {
					System.out.println(e);
					System.out.println(" in row #" + rowIdx + ", col #" + colIdx);
				}
			}
			rowIdx++;
		}
 
		// Return only the actual data in the ResultSet
		int doubleColIdx = 0;
		int booleanColIdx = 0;
		int stringColIdx = 0;
		Object[] data = new Object[numCols];
		for (int colIdx = 1; colIdx <= numCols; colIdx++) {
			switch (colTypes[colIdx]) {
				case 1:   data[colIdx-1] = dataCacheDouble[doubleColIdx++];    break;  // numeric
				case 2:   data[colIdx-1] = dataCacheBoolean[booleanColIdx++];  break;  // boolean
				default:  data[colIdx-1] = dataCacheString[stringColIdx++];            // string
			}
		}
		return data;
	}
}

So now we have a JDBC_Fetch class that we can use in our Matlab code, replacing the slow double loop with a single call to JDBC_Fetch.getData(), followed by vectorized conversion into a Matlab cell array (matrix):

% Get the data from the ResultSet using the JDBC_Fetch wrapper
data = cell(JDBC_Fetch.getData(rs));
for colIdx = 1 : numCols
   switch ColumnType{colIdx}
      case {'Float','Double'}
          data{colIdx} = num2cell(data{colIdx});
      case {'Long','Integer','Short','BigDecimal'}
          data{colIdx} = num2cell(data{colIdx});
      case 'Boolean'
          data{colIdx} = num2cell(data{colIdx});
      otherwise %case {'String','Date','Time','Timestamp'}
          %data{colIdx} = cell(data{colIdx});  % no need to do anything here!
   end
end
data = [data{:}];

On my specific program the resulting speedup was 15x (this is not a typo: 15 times faster). My fetches are no longer limited by the Matlab post-processing, but rather by the DB’s processing of the SQL statement (where DB indexes, clustering, SQL tuning etc. come into play).

Additional speedups can be achieved by parsing dates at the Java level (rather than returning strings), as well as several other tweaks in the Java and Matlab code (refer to Andrew Janke’s post for some ideas). But certainly the main benefit (the 80% of the gain that was achieved in 20% of the worktime) is due to the above push of the main double processing loop down into the Java level, leaving Matlab with just a single Java call to JDBC_Fetch.

Many additional ideas of speeding up database queries and Matlab programs in general can be found in my second book, Accelerating Matlab Performance.

If you’d like me to help you speed up your Matlab program, please email me (altmany at gmail), or fill out the query form on my consulting page.

]]>
https://undocumentedmatlab.com/blog/speeding-up-matlab-jdbc-sql-queries/feed 6
Working with non-standard DPI displayshttps://undocumentedmatlab.com/blog/working-with-non-standard-dpi-displays https://undocumentedmatlab.com/blog/working-with-non-standard-dpi-displays#comments Wed, 09 Nov 2016 21:47:27 +0000 http://undocumentedmatlab.com/?p=6736
 
Related posts:
  1. FindJObj – find a Matlab component’s underlying Java object The FindJObj utility can be used to access and display the internal components of Matlab controls and containers. This article explains its uses and inner mechanism....
  2. FindJObj GUI – display container hierarchy The FindJObj utility can be used to present a GUI that displays a Matlab container's internal Java components, properties and callbacks....
  3. Blurred Matlab figure window Matlab figure windows can be blurred using a semi-transparent overlaid window - this article explains how...
  4. Customizing figure toolbar background Setting the figure toolbar's background color can easily be done using just a tiny bit of Java magic powder. This article explains how. ...
 
]]>
With high-density displays becoming increasingly popular, some users set their display’s DPI to a higher-than-standard (i.e., >100%) value, in order to compensate for the increased pixel density to achieve readable interfaces. This OS setting tells the running applications that there are fewer visible screen pixels, and these are spread over a larger number of physical pixels. This works well for most cases (at least on recent OSes, it was a bit buggy in non-recet ones). Unfortunately, in some cases we might actually want to know the screen size in physical, rather than logical, pixels. Apparently, Matlab root’s ScreenSize property only reports the logical (scaled) pixel size, not the physical (unscaled) one:

>> get(0,'ScreenSize')   % with 100% DPI (unscaled standard)
ans =
        1       1      1366       768
 
>> get(0,'ScreenSize')   % with 125% DPI (scaled)
ans =
        1       1      1092.8     614.4

The same phenomenon also affects other related properties, for example MonitorPositions.

Raimund Schlüßler, a reader on this blog, was kind enough to point me to this problem and its workaround, which I thought worthy to share here: To get the physical screen-size, use the following builtin Java command:

>> jScreenSize = java.awt.Toolkit.getDefaultToolkit.getScreenSize
jScreenSize =
java.awt.Dimension[width=1366,height=768]
 
>> width = jScreenSize.getWidth
width =
        1366
 
>> height = jScreenSize.getHeight
height =
        768

Also see the related recent article on an issue with the DPI-aware feature starting with R2015b.

Upcoming travels – London/Belfast, Zürich & Geneva

I will shortly be traveling to consult some clients in Belfast (via London), Zürich and Geneva. If you are in the area and wish to meet me to discuss how I could bring value to your work, then please email me (altmany at gmail):

  • Belfast: Nov 28 – Dec 1 (flying via London)
  • Zürich: Dec 11-12
  • Geneva: Dec 13-15
]]>
https://undocumentedmatlab.com/blog/working-with-non-standard-dpi-displays/feed 6
Aligning uicontrol contentshttps://undocumentedmatlab.com/blog/aligning-uicontrol-contents https://undocumentedmatlab.com/blog/aligning-uicontrol-contents#comments Thu, 22 Sep 2016 13:10:18 +0000 http://undocumentedmatlab.com/?p=6663
 
Related posts:
  1. Spicing up Matlab uicontrol tooltips Matlab uicontrol tooltips can be spiced-up using HTML and CSS, including fonts, colors, tables and images...
  2. Rich-contents log panel Matlab listboxes and editboxes can be used to display rich-contents HTML-formatted strings, which is ideal for log panels. ...
  3. Multi-line uitable column headers Matlab uitables can present long column headers in multiple lines, for improved readability. ...
  4. Undocumented button highlighting Matlab button uicontrols can easily be highlighted by simply setting their Value property. ...
 
]]>
Matlab automatically aligns the text contents of uicontrols: button labels are centered, listbox contents are left-aligned, and table cells align depending on their contents (left-aligned for strings, centered for logical values, and right-aligned for numbers). Unfortunately, the control’s HorizontalAlignment property is generally ignored by uicontrols. So how can we force Matlab buttons (for example) to have right-aligned labels, or for listbox/table cells to be centered? Undocumented Matlab has the answer, yet again…

It turns out that there are at least two distinct ways to set uicontrol alignment, using HTML and using Java. Today I will only discuss the HTML variant.

The HTML method relies on the fact that Matlab uicontrols accept and process HTML strings. This was true ever since Matlab GUI started relying on Java Swing components (which inherently accept HTML labels) over a decade ago. This is expected to remain true even in Matlab’s upcoming web-based GUI system, since Matlab would need to consciously disable HTML in its web components, and I see no reason for MathWorks to do so. In short, HTML parsing of GUI control strings is here to stay for the foreseeable future.

% note: no need to close HTML tags, e.g. </font></html>
uicontrol('Style','list', 'Position',[10,10,70,70], 'String', ...
          {'<HTML><FONT color="red">Hello</Font></html>', 'world', ...
           '<html><font style="font-family:impact;color:green"><i>What a', ...
           '<Html><FONT color="blue" face="Comic Sans MS">nice day!'});

Listbox with HTML items

Listbox with HTML items

While HTML formatting is generally frowned-upon compared to the alternatives, it provides a very quick and easy way to format text labels in various different manners, including using a combination of font faces, sizes, colors and other aspects (bold, italic, super/sub-script, underline etc.) within a single text label. This is naturally impossible to do with Matlab’s standard properties, but is super-easy with HTML placed in the label’s String property.

Unfortunately, while Java Swing (and therefore Matlab) honors only a [large] sub-set of HTML and CSS. The most important directives are parsed but some others are not, and this is often difficult to debug. Luckily, using HTML and CSS there are often multiple ways to achieve the same visual effect, so if one method fails we can usually find an alternative. Such was the case when a reader asked me why the following seemingly-simple HTML snippet failed to right-align his button label:

hButton.String = '<html><div style="text-align:right">text';

As I explained in my answer, it’s not Matlab that ignores the CSS align directive but rather the underlying Swing behavior, which snugly fits the text in the center of the button, and of course aligning text within a tight-fitting box has no effect. The workaround that I suggested simply forces Swing to use a non-tightly-fitting boundary box, within which we can indeed align the text:

pxPos = getpixelposition(hButton);
hButton.String = ['<html><div width="' num2str(pxPos(3)-20) 'px" align="right">text'];  % button margins use 20px

centered (default) button label   right-aligned button label

Centered (default) and right-aligned button labels

This solution is very easy to set up and maintain, and requires no special knowledge other than a bit of HTML/CSS, which most programmers know in this day and age.

Of course, the solution relies on the actual button size. So, if the button is created with normalized units and changes its size when its parent container is resized, we’d need to set a callback function on the parent (e.g., SizeChangedFcn of a uipanel) to automatically adjust the button’s string based on its updated size. A better solution that would be independent of the button’s pixel-size and would work even when the button is resized needs to use Java.

A related solution for table cells uses a different HTML-based trick: this time, we embed an HTML table cell within the Matlab control’s cell, employing the fact that HTML table cells can easily be aligned. We just need to ensure that the HTML cell is defined to be larger than the actual cell width, so that the alignment fits well. We do this by setting the HTML cell width to 9999 pixels (note that the tr and td HTML tags are necessary, but the table tag is optional):

uitable('Units','norm','Pos',[0,0,0.3,0.3], 'Data', ...
        {'Left', ...
         '<html><tr><td align=center width=9999>Center', ...
         '<html><tr><td align=right  width=9999>Right'});

Non-default alignment of uitable cells

Non-default alignment of uitable cells

As noted above, a better solution might be to set the underlying Java component’s alignment properties (or in the case of the uitable, its underlying JTable component’s cellrenderer’s alignment). But in the general case, simple HTML such as above could well be sufficient.

]]>
https://undocumentedmatlab.com/blog/aligning-uicontrol-contents/feed 6
Zero-testing performancehttps://undocumentedmatlab.com/blog/zero-testing-performance https://undocumentedmatlab.com/blog/zero-testing-performance#comments Wed, 31 Aug 2016 17:00:44 +0000 http://undocumentedmatlab.com/?p=6622
 
Related posts:
  1. uicontextmenu performance Matlab uicontextmenus are not automatically deleted with their associated objects, leading to leaks and slow-downs. ...
  2. rmfield performance The performance of the builtin rmfield function (as with many other builtin functions) can be improved by simple profiling. ...
  3. tic / toc – undocumented option Matlab's built-in tic/toc functions have an undocumented option enabling multiple nested clockings...
  4. Solving a MATLAB bug by subclassing Matlab's Image Processing Toolbox's impoint function contains an annoying bug that can be fixed using some undocumented properties....
 
]]>
I would like to introduce guest blogger Ken Johnson, a MATLAB Connections partner specializing in electromagnetic optics simulation. Today Ken will explore some performance subtleties of zero testing in Matlab.

I often have a need to efficiently test a large Matlab array for any nonzero elements, e.g.

>> a = zeros(1e4);
>> tic, b = any(a(:)~=0); toc
Elapsed time is 0.126118 seconds.

Simple enough. In this case, when a is all-zero, the internal search algorithm has no choice but to inspect every element of the array to determine whether it contains any nonzeros. In the more typical case where a contains many nonzeros you would expect the search to terminate almost immediately, as soon as it finds the first nonzero. But that’s not how it works:

>> a = round(rand(1e4));
>> tic, b = any(a(:)~=0); toc
Elapsed time is 0.063404 seconds.

There is significant runtime overhead in constructing the logical array “a(:)~=0”, although the “any(…)” operation apparently terminates at the first true value it finds.

The overhead can be eliminated by taking advantage of the fact that numeric values may be used as logicals in Matlab, with zero implicitly representing false and nonzero representing true. Repeating the above test without “~=0”, we get a huge runtime improvement:

>> a = round(rand(1e4));
>> tic, b = any(a(:)); toc
Elapsed time is 0.000026 seconds.

However, there is no runtime benefit when a is all-zero:

>> a = zeros(1e4);
>> tic, b = any(a(:)); toc
Elapsed time is 0.125120 seconds.

(I do not quite understand this. There should be some runtime benefit from bypassing the logical array construction.)

NaN values

There is also another catch: The above efficiency trick does not work when a contains NaN values (if you consider NaN to be nonzero), e.g.

>> any([0,nan])
ans =
     0

The any function ignores entries that are NaN, meaning it treats NaNs as zero-equivalent. This is inconsistent with the behavior of the inequality operator:

>> any([0,nan]~=0)
ans =
     1

To avoid this problem, an explicit isnan test is needed. Efficiency is not impaired when a contains many nonzeros, but there is a 2x efficiency loss when a is all-zero:

>> a = round(rand(1e4));
>> tic, b = any(a(:)) || any(isnan(a(:))); toc
Elapsed time is 0.000027 seconds.
 
>> a = zeros(1e4);
>> tic, b = any(a(:)) || any(isnan(a(:))); toc
Elapsed time is 0.256604 seconds.

For testing all-nonzero the NaN problem does not occur:

>> all([1 nan])
ans =
     1

In this context NaN is treated as nonzero and the all-nonzero test is straightforward:

>> a = round(rand(1e4));
>> tic, b = all(a(:)); toc
Elapsed time is 0.000029 seconds.

For testing any-zero and all-zero, use the complements of the above tests:

>> b = ~any(a(:)) || any(isnan(a(:)));  % all zero?
>> b = ~all(a(:));  % any zero?

Efficient find

The find operation can also be optimized by bypassing construction of a logical temporary array, e.g.

>> a = round(rand(1e4));
>> tic, b = find(a(:)~=0, 1); toc
Elapsed time is 0.065697 seconds.
 
>> tic, b = find(a(:), 1); toc
Elapsed time is 0.000029 seconds.

There is no problem with NaNs in this case; the find function treats NaN as nonzero, e.g.

>> find([0,nan,1], 1)
ans =
     2
]]>
https://undocumentedmatlab.com/blog/zero-testing-performance/feed 5
Customizing axes part 5 – origin crossover and labelshttps://undocumentedmatlab.com/blog/customizing-axes-part-5-origin-crossover-and-labels https://undocumentedmatlab.com/blog/customizing-axes-part-5-origin-crossover-and-labels#comments Wed, 27 Jul 2016 17:00:02 +0000 http://undocumentedmatlab.com/?p=6564
 
Related posts:
  1. Customizing axes rulers HG2 axes can be customized in numerous useful ways. This article explains how to customize the rulers. ...
  2. Customizing axes part 2 Matlab HG2 axes can be customized in many different ways. This article explains some of the undocumented aspects. ...
  3. Undocumented scatter plot jitter Matlab's scatter plot can automatically jitter data to enable better visualization of distribution density. ...
  4. Performance: accessing handle properties Handle object property access (get/set) performance can be significantly improved using dot-notation. ...
 
]]>
When HG2 graphics was finally released in R2014b, I posted a series of articles about various undocumented ways by which we can customize Matlab’s new graphic axes: rulers (axles), baseline, box-frame, grid, back-drop, and other aspects. Today I extend this series by showing how we can customize the axes rulers’ crossover location.

Non-default axes crossover location

Non-default axes crossover location


The documented/supported stuff

Until R2015b, we could only specify the axes’ YAxisLocation as 'left' (default) or 'right', and XAxisLocation as 'bottom' (default) or 'top'. For example:

x = -2*pi : .01 : 2*pi;
plot(x, sin(x));
hAxis = gca;
hAxis.YAxisLocation = 'left';    % 'left' (default) or 'right'
hAxis.XAxisLocation = 'bottom';  % 'bottom' (default) or 'top'

Default axis locations: axes crossover is non-fixed

Default axis locations: axes crossover is non-fixed

The crossover location is non-fixed in the sense that if we zoom or pan the plot, the axes crossover will remain at the bottom-left corner, which changes its coordinates depending on the X and Y axes limits.

Since R2016a, we can also specify 'origin' for either of these properties, such that the X and/or Y axes pass through the chart origin (0,0) location. For example, move the YAxisLocation to the origin:

hAxis.YAxisLocation = 'origin';

Y-axis location at origin: axes crossover at 0 (fixed), -1 (non-fixed)

Y-axis location at origin: axes crossover at 0 (fixed), -1 (non-fixed)

And similarly also for XAxisLocation:

hAxis.XAxisLocation = 'origin';

X and Y-axis location at origin: axes crossover fixed at (0,0)

X and Y-axis location at origin: axes crossover fixed at (0,0)

The axes crossover location is now fixed at the origin (0,0), so as we move or pan the plot, the crossover location changes its position in the chart area, without changing its coordinates. This functionality has existed in other graphic packages (outside Matlab) for a long time and until now required quite a bit of coding to emulate in Matlab, so I’m glad that we now have it in Matlab by simply updating a single property value. MathWorks did a very nice job here of dynamically updating the axles, ticks and labels as we pan (drag) the plot towards the edges – try it out!

The undocumented juicy stuff

So far for the documented stuff. The undocumented aspect is that we are not limited to using the (0,0) origin point as the fixed axes crossover location. We can use any x,y crossover location, using the FirstCrossoverValue property of the axes’ hidden XRuler and YRuler properties. In fact, we could do this since R2014b, when the new HG2 graphics engine was released, not just starting in R2016a!

% Set a fixed crossover location of (pi/2,-0.4)
hAxis.YRuler.FirstCrossoverValue = pi/2;
hAxis.XRuler.FirstCrossoverValue = -0.4;

Custom fixed axes crossover location at (π/2,-0.4)

Custom fixed axes crossover location at (π/2,-0.4)

For some reason (bug?), setting XAxisLocation/YAxisLocation to ‘origin’ has no visible effect in 3D plots, nor is there any corresponding ZAxisLocation property. Luckily, we can set the axes crossover location(s) in 3D plots using FirstCrossoverValue just as easily as for 2D plots. The rulers also have a SecondCrossoverValue property (default = -inf) that controls the Z-axis crossover, as Yaroslav pointed out in a comment below. For example:

N = 49;
x = linspace(-10,10,N);
M = peaks(N);
mesh(x,x,M);

Default crossover locations at (-10,±10,-10)

Default crossover locations at (-10,±10,-10)

hAxis.XRuler.FirstCrossoverValue  = 0; % X crossover with Y axis
hAxis.YRuler.FirstCrossoverValue  = 0; % Y crossover with X axis
hAxis.ZRuler.FirstCrossoverValue  = 0; % Z crossover with X axis
hAxis.ZRuler.SecondCrossoverValue = 0; % Z crossover with Y axis

Custom fixed axes crossover location at (0,0,-10)

Custom fixed axes crossover location at (0,0,-10)

hAxis.XRuler.SecondCrossoverValue = 0; % X crossover with Z axis
hAxis.YRuler.SecondCrossoverValue = 0; % Y crossover with Z axis

Custom fixed axes crossover location at (0,0,0)

Custom fixed axes crossover location at (0,0,0)

Labels

Users will encounter the following unexpected behavior (bug?) when using either the documented *AxisLocation or the undocumented FirstCrossoverValue properties: when setting an x-label (using the xlabel function, or the internal axes properties), the label moves from the center of the axes (as happens when XAxisLocation=’top’ or ‘bottom’) to the right side of the axes, where the secondary label (e.g., exponent) usually appears, whereas the secondary label is moved to the left side of the axis:

Unexpected label positions

Unexpected label positions

In such cases, we would expect the labels locations to be reversed, with the main label on the left and the secondary label in its customary location on the right. The exact same situation occurs with the Y labels, where the main label unexpectedly appears at the top and the secondary at the bottom. Hopefully MathWorks will fix this in the next release (it is probably too late to make it into R2016b, but hopefully R2017a). Until then, we can simply switch the strings of the main and secondary label to make them appear at the expected locations:

% Switch the Y-axes labels:
ylabel(hAxis, '\times10^{3}');  % display secondary ylabel (x10^3) at top
set(hAxis.YRuler.SecondaryLabel, 'Visible','on', 'String','main y-label');  % main label at bottom
 
% Switch the X-axes labels:
xlabel(hAxis, '2^{nd} label');  % display secondary xlabel at right
set(hAxis.XRuler.SecondaryLabel, 'Visible','on', 'String','xlabel');  % main label at left

As can be seen from the screenshot, there’s an additional nuisance: the main label appears a bit larger than the axes font size (the secondary label uses the correct font size). This is because by default Matlab uses a 110% font-size for the main axes label, ostensibly to make them stand out. We can modify this default factor using the rulers’ hidden LabelFontSizeMultiplier property (default=1.1). For example:

hAxis.YRuler.LabelFontSizeMultiplier = 1;   % use 100% font-size (same as tick labels)
hAxis.XRuler.LabelFontSizeMultiplier = 0.8; % use 80% (smaller than standard) font-size

Note: I described the ruler objects in my first article of the axes series. Feel free to read it for more ideas on customizing the axes rulers.

]]>
https://undocumentedmatlab.com/blog/customizing-axes-part-5-origin-crossover-and-labels/feed 5
Handling red Java console errorshttps://undocumentedmatlab.com/blog/handling-red-java-console-errors https://undocumentedmatlab.com/blog/handling-red-java-console-errors#comments Wed, 29 Jun 2016 17:00:51 +0000 http://undocumentedmatlab.com/?p=6468
 
Related posts:
  1. Setting system tray icons System-tray icons can be programmatically set and controlled from within Matlab, using new functionality available since R2007b....
  2. Setting system tray popup messages System-tray icons and messages can be programmatically set and controlled from within Matlab, using new functionality available since R2007b....
  3. JGit-Matlab integration JGit source-control integration package can easily be integrated in Matlab. ...
  4. Non-textual editor actions The UIINSPECT utility can be used to expand EditorMacro capabilities to non-text-insertion actions. This is how:...
 
]]>
Anyone who has worked with non-trivial Matlab GUIs knows that from time to time we see various red Java stack-trace errors appear in the Matlab console (Command Window). These errors do not appear often when using documented Matlab controls, but they do from time to time. The errors appear significantly more frequently when working with undocumented Java-based hacks that I often show on this blog, and especially when working with complex controls such as uitable or uitree. Such controls have a very large code-base under the hood, and the Matlab code and data sometimes clashes with the asynchronous Java methods that run on a separate thread. Such clashes and race conditions often lead to red Java stack-trace errors that are spewed onto the Matlab console. For example:

Exception in thread "AWT-EventQueue-0" java.lang.NullPointerException
	at com.jidesoft.plaf.basic.BasicCellSpanTableUI.paint(Unknown Source)
	at javax.swing.plaf.ComponentUI.update(Unknown Source)
	at javax.swing.JComponent.paintComponent(Unknown Source)
	at com.jidesoft.grid.CellStyleTable.paintComponent(Unknown Source)
	at javax.swing.JComponent.paint(Unknown Source)
	at javax.swing.JComponent.paintToOffscreen(Unknown Source)
	...

Exception in thread "AWT-EventQueue-0" java.lang.ArrayIndexOutOfBoundsException: 1 >= 0
	at java.util.Vector.elementAt(Unknown Source)
	at javax.swing.table.DefaultTableColumnModel.getColumn(Unknown Source)
	at com.jidesoft.grid.ContextSensitiveTable.getCellRenderer(Unknown Source)
	at com.jidesoft.grid.CellSpanTable.getCellRenderer(Unknown Source)
	at com.jidesoft.grid.TreeTable.getActualCellRenderer(Unknown Source)
	at com.jidesoft.grid.GroupTable.getCellRenderer(Unknown Source)
	at com.jidesoft.grid.JideTable.b(Unknown Source)
	at com.jidesoft.grid.CellSpanTable.calculateRowHeight(Unknown Source)
	...

In almost all such Java error messages, the error is asynchronous to the Matlab code and does not interrupt it. No error exception is thrown (or can be trapped), and the Matlab code proceeds without being aware that anything is wrong. In fact, in the vast majority of such cases, nothing is visibly wrong – the program somehow overcomes the reported problem and there are no visible negative effects on the GUI. In other words, these error messages are harmless and can almost always be ignored. Still, if we could only stop those annoying endless red stack-trace messages in the Matlab console!

Note that today’s post only discusses untrappable asynchronous Java error messages, not synchronous errors that can be trapped in Matlab via try-catch. These synchronous errors are often due to programmatic errors (e.g., bad method input args or an empty reference handle) and can easily be handled programmatically. On the other hand, the asynchronous errors are non-trappable, so they are much more difficult to isolate and fix.

In many of the cases, the error occurs when the control’s underlying data model is changed by the Matlab code, and some of the controls’s Java methods are not synced with the new model by the time they run. This can be due to internal bugs in the Matlab or Java control’s implementation, or to simple race conditions that occur between the Matlab thread and the Java Event Dispatch Thread (EDT). As noted here, such race conditions can often be solved by introducing a simple delay into the Matlab code:

javaControl.doSomething();
pause(0.05); drawnow;javaControl.doSomethingElse();

In addition, asking Matlab to run the Java component’s methods on the EDT can also help solve race conditions:

javaControl = javaObjectEDT(javaControl);

Unfortunately, sometimes both of these are not enough. In such cases, one of the following ideas might help:

  • Add fprintf(' \b') to your Matlab code: this seemingly innocent hack of displaying a space & immediately erasing it with backspace, appears to force the Java engine to flush its event queue and synchronize things, thereby avoiding the annoying Java console errors. I know it sounds like adding a sprinkle of useless black magic to the code, but it does really work in some cases!
    javaControl.doSomething();
    pause(0.05); drawnow;  % this never hurt anyone!
    fprintf(' \b');javaControl.doSomethingElse();
  • It is also possible to directly access the console text area and remove all the text after a certain point. Note that I strongly discourage messing around with the console text in this manner, since it might cause problems with Matlab’s internals. Still, if you are adventurous enough to try, then here’s an example:
    jCmdWinDoc = com.mathworks.mde.cmdwin.CmdWinDocument.getInstance;
    currentPos = cmdWinDoc.getLength;
     
    javaControl.doSomething();
    pause(0.05); drawnow;  % this never hurt anyone!
    javaControl.doSomethingElse();
     
    pause(0.1);  % let the java error time to display itself in the console
    jCmdWinDoc.remove(currentPos, cmdWinDoc.getLength-currentPos);
  • When all else fails, consider simply clearing the Matlab console using the Matlab clc command a short while after updating the Java control. This will erase the red Java errors, along with everything else in the console, so naturally it cannot be freely used if you use the console to display useful information to the user.

It should be emphasized: not all of these suggested remedies work in all cases; in some cases some of them work, and in other cases others might work. There does not seem to be a general panacea to this issue. The main purpose of the article was to list the possible solutions in a single place, so that users could try them out and select those that work for each specific case.

Do you know of any other (perhaps better) way of avoiding or hiding such asynchronous Java console errors? If so, then please post a comment below.

]]>
https://undocumentedmatlab.com/blog/handling-red-java-console-errors/feed 6
rmfield performancehttps://undocumentedmatlab.com/blog/rmfield-performance https://undocumentedmatlab.com/blog/rmfield-performance#comments Wed, 25 May 2016 07:00:48 +0000 http://undocumentedmatlab.com/?p=6427
 
Related posts:
  1. tic / toc – undocumented option Matlab's built-in tic/toc functions have an undocumented option enabling multiple nested clockings...
  2. Solving a MATLAB bug by subclassing Matlab's Image Processing Toolbox's impoint function contains an annoying bug that can be fixed using some undocumented properties....
  3. Plot LimInclude properties The plot objects' XLimInclude, YLimInclude, ZLimInclude, ALimInclude and CLimInclude properties are an important feature, that has both functional and performance implications....
  4. Class object creation performance Performance aspects of Matlab class object creation are discussed, with specific suggestions. ...
 
]]>
Once again I would like to introduce guest blogger Hanan Kavitz of Applied Materials. Several months ago Hanan discussed some quirks with compiled Matlab DLLs. Today Hanan will discuss how they overcame a performance bottleneck with Matlab’s builtin rmfield function, exemplifying the general idea that we can sometimes improve performance by profiling the core functionality that causes a performance hotspot and optimizing it, even when it is part of a builtin Matlab function. For additional ideas of improving Matlab peformance, search this blog for “Performance” articles, and/or get the book “Accelerating MATLAB Performance“.

Accelerating MATLAB Performance
I’ve been using Matlab for many years now and from time to time I need to profile low-throughput code. When I profile this code sometimes I realize that a computational ‘bottleneck’ is due to a builtin Matlab function (part of the core language). I can often find ways to accelerate such builtin functions and get significant speedup in my code.

I recently found Matlab’s builtin rmfield function being too slow for my needs. It works great when one needs to remove a few fields from a small structure, but in our case we needed to remove thousands of fields from a structure containing about 5000 fields – and this is executed in a function that is called many times inside an external loop. The program was significantly sluggish.

It started when a co-worker asked me to look at a code that looked just slightly more intelligent than this:

for i = 1:5000
    myStruct = rmfield(myStruct,fieldNames{i});
end

Running this code within a tic/toc pair yielded the following results:

>> tic; myFunc(); t1 = toc
t1 =
      25.7713

In my opinion 25.77 secs for such a simple functionality seems like an eternity…

The obvious thing was to change the code to the documented faster (vectorized) version:

>> tic; myStruct = rmfield(myStruct,fieldNames); t2 = toc
t2 =
      0.6097

This is obviously much better but since rmfield is called many times in my application, I needed something even better. So I profiled rmfield and was not happy with the result.

The original code of rmfield (%matlabroot%/toolbox/matlab/datatypes/rmfield.m) looks something like this (I deleted some non-essential code for brevity):

function t = rmfield(s,field)
 
% get fieldnames of struct
f = fieldnames(s);
 
% Determine which fieldnames to delete.
idxremove = [];
for i=1:length(field)
   j = find(strcmp(field{i},f) == true);   idxremove = [idxremove;j];
end
 
% set indices of fields to keep
idxkeep = 1:length(f);
idxkeep(idxremove) = [];
 
% remove the specified fieldnames from the list of fieldnames.
f(idxremove,:) = [];
 
% convert struct to cell array
c = struct2cell(s);
 
% find size of cell array
sizeofarray = size(c);
newsizeofarray = sizeofarray;
 
% adjust size for fields to be removed
newsizeofarray(1) = sizeofarray(1) - length(idxremove);
 
% rebuild struct
t = cell2struct(reshape(c(idxkeep,:),newsizeofarray),f);

When I profiled the code, the highlighted row was the bottleneck I was looking for.

First, I noticed the string comparison equals to true part – while '==true' is not the cause of the bottleneck, it does leave an impression of bad coding style :-( Perhaps this code was created as some apprentice project, which might also explain its suboptimal performance.

The real performance problem here is that for each field that we wish to remove, rmfield compares it to all existing fields to find its location in a cell array of field names. This is algorithmically inefficient and makes the code hard to understand (just try – it took me hard, long minutes).

So, I created a variant of rmfield.m called fast_rmfield.m, as follows (again, omitting some non-essential code):

function t = fast_rmfield(s,field)
 
% get fieldnames of struct
f = fieldnames(s);
[f,ia] = setdiff(f,field,'R2012a');
 
% convert struct to cell array
c = squeeze(struct2cell(s));
 
% rebuild struct
t = cell2struct(c(ia,:),f)';

This code is much shorter, easier to explain and maintain, but also (and most importantly) much faster:

>> tic; myStruct = fast_rmfield(myStruct,fieldNames); t3 = toc
t3 =
      0.0302
 
>> t2/t3
ans =
      20.1893

This resulted in a speedup of ~850x compared to the original version (of 25.77 secs), and ~20x compared to the vectorized version. A nice improvement in my humble opinion…

The point in all this is that we can and should rewrite Matlab builtin functions when they are too slow for our needs, whether it is found to be an algorithmic flaw (as in this case), extraneous sanity checks (as in the case of ismember or datenum), bad default parameters (as in the case of fopen/fwrite or scatter), or merely slow implementation (as in the case of save, cellfun, or the conv family of functions).

A good pattern is to save such code pieces in file names that hint to the original code. In our case, I used fast_rmfield to suggest that it is a faster alternative to rmfield.

Do you know of any other example of a slow implementation in a built-in Matlab function that can be optimized? If so, please leave a comment below.

]]>
https://undocumentedmatlab.com/blog/rmfield-performance/feed 5
Viewing saved profiling resultshttps://undocumentedmatlab.com/blog/viewing-saved-profiling-results https://undocumentedmatlab.com/blog/viewing-saved-profiling-results#respond Wed, 18 May 2016 18:00:37 +0000 http://undocumentedmatlab.com/?p=6421
 
Related posts:
  1. Plot performance Undocumented inner plot mechanisms can significantly improve plotting performance ...
  2. Undocumented Profiler options part 2 Several undocumented features of the Matlab Profiler can make it much more useful - part 2 of series. ...
  3. Undocumented Profiler options part 3 An undocumented feature of the Matlab Profiler can report call history timeline - part 3 of series. ...
  4. Undocumented Profiler options part 4 Several undocumented features of the Matlab Profiler can make it much more useful - part 4 of series. ...
 
]]>
Many Matlab users know and utilize Matlab’s built-in Profiler tool to identify performance bottlenecks and code-coverage issues. Unfortunately, not many are aware of the Profiler’s programmatic interface. In past articles as well as my performance book I explained how we can use this programmatic interface to save profiling results and analyze it offline. In fact, I took this idea further and even created a utility (profile_history) that displays the function call timeline in a standalone Matlab GUI, something that is a sorely missed feature in the built-in profiler:

Function call timeline profiling (click for full-size image)
Function call timeline profiling (click for full-size image)

Today I will discuss a related undocumented feature of the Profiler: loading and viewing pre-saved profiling results.

Programmatic access to profiling results

Matlab’s syntax for returning the detailed profiling results in a data struct is clearly documented in the profile function’s doc page. Although the documentation does not explain the resulting struct and sub-struct fields, they have meaningful names and we can relatively easily infer what each of them means (I added a few annotation comments for clarity):

>> profile('on','-history')
>> surf(peaks); drawnow
>> profile('off')
>> profData = profile('info')
profData = 
      FunctionTable: [26x1 struct]
    FunctionHistory: [2x56 double]
     ClockPrecision: 4.10517962829241e-07
         ClockSpeed: 2501000000
               Name: 'MATLAB'
           Overhead: 0
 
>> profData.FunctionTable(1)
ans = 
          CompleteName: 'C:\Program Files\Matlab\R2016a\toolbox\matlab\specgraph\peaks.m>peaks'
          FunctionName: 'peaks'
              FileName: 'C:\Program Files\Matlab\R2016a\toolbox\matlab\specgraph\peaks.m'
                  Type: 'M-function'
              Children: [1x1 struct]
               Parents: [0x1 struct]
         ExecutedLines: [9x3 double]
           IsRecursive: 0
    TotalRecursiveTime: 0
           PartialData: 0
              NumCalls: 1
             TotalTime: 0.0191679078068094
 
>> profData.FunctionTable(1).Children
ans = 
        Index: 2   % index in profData.FunctionTable array
     NumCalls: 1
    TotalTime: 0.00136415141013509
 
>> profData.FunctionTable(1).ExecutedLines   % line number, number of calls, duration in secs
ans =
         43      1      0.000160102031282782
         44      1      2.29890096200918e-05
         45      1      0.00647592190637408
         56      1      0.0017093970724654
         57      1      0.00145036019621044
         58      1      0.000304193859437286
         60      1      4.39254290955326e-05
         62      1      3.44835144301377e-05
         63      1      0.000138755093778411
 
>> profData.FunctionHistory(:,1:5)
ans =
     0     0     1     1     0   % 0=enter, 1=exit
     1     2     2     1     6   % index in profData.FunctionHistory array

As we can see, this is pretty intuitive so far.

Loading and viewing saved profiling results

If we wish to save these results results in a file and later load and display them in the Profiler’s visualization browser, then we need to venture deeper into undocumented territory. It seems that while retrieving the profiling results (via profile(‘info’)) is fully documented, doing the natural complementary action (namely, loading this data into the viewer) is not. For the life of me I cannot understand the logic behind this decision, but that’s the way it is.

Luckily, the semi-documented built-in function profview does exactly what we need: profview accepts 2 input args (function name and the profData struct) and displays the resulting profiling info. The first input arg (function name) accepts either a string (e.g., 'peaks' or 'view>isAxesHandle'), or the numeric value 0 which signifies the home (top-level) page:

profView(0, profData);  % display profiling home (top-level) page
profview('peaks', profData);  % display a specific profiling page

I use the 0 input value much more frequently than the string inputs, because I often don’t know which functions exactly were profiled, and starting at the home page enables me to easily drill-down the profiling results interactively.

Loading saved profiling results from a different computer

Things get slightly complicated if we try to load saved profiling results from a different computer. If the other computer has exactly the same folder structure as our computer, and all our Matlab functions reside in exactly the same disk folders/path, then everything will work out of the box. The problem is that in general the other computer will have the functions in different folders. When we then try to load the profData on our computer, it will not find the associated Matlab functinos in order to display the line-by-line profiling results. We will only see the profiling data at the function level, not line level. This significantly reduces the usefulness of the profiling data. The Profiler page will display the following error message:

This file was modified during or after profiling. Function listing disabled.

We can solve this problem in either of two ways:

  1. Modify our profData to use the correct folder path on the local computer, rather than the other computer’s path (which is invalid on the local computer). For example:
    % Save the profData on computer #1:
    profData = profile('info');
    save('profData.mat', 'profData');
     
    % Load the profData on computer #2:
    fileData = load('profData.mat');
    profData = fileData.profData;
    path1 = 'N:\Users\Juan\programs\myProgram';
    path2 = 'C:\Yair\consulting\clients\Intel\code';
    for idx = 1 : numel(profData.FunctionTable)
       funcData = profData.FunctionTable(idx);
       funcData.FileName     = strrep(funcData.FileName,     path1, path2); 
       funcData.CompleteName = strrep(funcData.CompleteName, path1, path2);
       profData.FunctionTable(idx) = funcData;
    end
    % note: this loop can be vectorized if you wish
  2. As an alternative, we can modify Matlab’s profview.m function (%matlabroot%/toolbox/matlab/codetools/profview.m) to search for the function’s source code in the current Matlab path, if the specified direct path is not found (note that changing profview.m may require administrator priviledges). For example, the following is the code from R2016a’s profview.m file, line #506:
        % g894021 - Make sure the MATLAB code file still exists
        if ~exist(fullName, 'file')
            [~,fname,fext] = fileparts(fullName);  % Yair        fname = which([fname fext]);           % Yair        if isempty(fname)                      % Yair            mFileFlag = 0;
            else                                   % Yair            fullName = fname;                  % Yair        end                                    % Yair    end

These two workarounds complement each other: the first workaround does not require changing any installed Matlab code, and so is platform- and release-independent, but would require rerunning the code snippet for each and every profiling data file that we receive from external computers. On the other hand, the second workaround is a one-time operation that should work for multiple saved profiling results, although we would need to redo it whenever we install Matlab.

Additional profview customizations

Modifying the profview.m function can be used for different improvements as well.

For example, several years ago I explained how this function can be modified to display 1 ms timing resolutions, rather than the default 10 mS.

Another customization that I often do after I install Matlab is to change the default setting of truncating function lines longer than 40 characters – I typically modify this to 60 or 80 (depending on the computer monitor’s size…). All we need to do is to update the truncateDisplayName sub-function within profview.m as follows (taken from R2016a again, line #1762):

function shortFileName = truncateDisplayName(longFileName,maxNameLen)
%TRUNCATEDISPLAYNAME  Truncate the name if it gets too long
maxNameLen = max(60,maxNameLen);  % YairshortFileName = escapeHtml(longFileName);
if length(longFileName) > maxNameLen,
    shortFileName = char(com.mathworks.util.FileUtils.truncatePathname( ...
        shortFileName, maxNameLen));
end

You can see additional undocumented profiling features in the “Related posts” section below, as well as in Chapter 2 of my book “Accelerating MATLAB Performance“.

Do you have any other customization to the profiling results? If so, please share it in a comment.

]]>
https://undocumentedmatlab.com/blog/viewing-saved-profiling-results/feed 0
Smart listbox & editbox scrollbarshttps://undocumentedmatlab.com/blog/smart-listbox-editbox-scrollbars https://undocumentedmatlab.com/blog/smart-listbox-editbox-scrollbars#comments Wed, 20 Apr 2016 17:47:46 +0000 http://undocumentedmatlab.com/?p=6379
 
Related posts:
  1. FindJObj GUI – display container hierarchy The FindJObj utility can be used to present a GUI that displays a Matlab container's internal Java components, properties and callbacks....
  2. Editbox data input validation Undocumented features of Matlab editbox uicontrols enable immediate user-input data validation...
  3. Customizing combobox popups Matlab combobox (dropdown) popups can be customized in a variety of ways. ...
  4. GUI integrated HTML panel Simple HTML can be presented in a Java component integrated in Matlab GUI, without requiring the heavy browser control....
 
]]>
A good friend recently asked me for examples where using Java in Matlab programs provides a significant benefit that would offset the risk of using undocumented/unsupported functionality, which may possibly stop working in some future Matlab release. Today I will discuss a very easy Java-based hack that in my opinion improves the appearance of Matlab GUIs with minimal risk of a catastrophic failure in a future release.

The problem with Matlab listbox and multi-line editbox controls in the current (non web-based) GUI, is that they use a scrollbar whose behavior policy is set to VERTICAL_SCROLLBAR_ALWAYS. This causes the vertical scrollbar to appear even when the listbox does not really require it. In many cases, when the listbox is too narrow, this also causes the automatic appearance of a horizontal scrollbar. The end result is a listbox that displays 2 useless scrollbars, that possibly hide some listbox contents, and are a sore to the eyes:

Standard (left) and smart (right) listbox scrollbars

Standard (left) and smart (right) listbox scrollbars

   
default scrollbars (VERTICAL_SCROLLBAR_ALWAYS)

default scrollbars (VERTICAL_SCROLLBAR_ALWAYS)

non-default scrollbars (VERTICAL_SCROLLBAR_AS_NEEDED)     non-default scrollbars (VERTICAL_SCROLLBAR_AS_NEEDED)

non-default scrollbars (VERTICAL_SCROLLBAR_AS_NEEDED)

By default, Matlab implements a vertical scrollbar policy of VERTICAL_SCROLLBAR_ALWAYS for sufficiently tall uicontrols (>20-25 pixels, which practically means always) and VERTICAL_SCROLLBAR_NEVER for shorter uicontrols (this may possibly be platform-dependent).

A similar problem happens with the horizontal scrollbar: Matlab implements a horizontal scrollbar policy of HORIZONTAL_SCROLLBAR_NEVER for all editboxes and also for narrow listboxes (<35 pixels), and HORIZONTAL_SCROLLBAR_AS_NEEDED for wide listboxes.

In many cases we may wish to modify the settings, as in the example shown above. The solution to this is very easy, as I explained back in 2010.

All we need to do is to retrieve the control’s underlying Java reference (a Java JScrollPane object) and change the policy value to VERTICAL_SCROLLBAR_AS_NEEDED:

% Create a multi-line (Max>1) editbox uicontrol
hEditbox = uicontrol('style','edit', 'max',5, ...);
 
try  % graceful-degradation for future compatibility
   % Get the Java scroll-pane container reference
   jScrollPane = findjobj(hEditbox);
 
   % Modify the scroll-pane's scrollbar policies
   % (note the equivalent alternative methods used below)
   set(jScrollPane,'VerticalScrollBarPolicy',javax.swing.ScrollPaneConstants.VERTICAL_SCROLLBAR_AS_NEEDED);   %VERTICAL_SCROLLBAR_AS_NEEDED=20
   jScrollPane.setHorizontalScrollBarPolicy(javax.swing.ScrollPaneConstants.HORIZONTAL_SCROLLBAR_AS_NEEDED);  %HORIZONTAL_SCROLLBAR_AS_NEEDED=30
catch
   % Never mind...
end

Note that updating the uicontrol handle Position property has the side-effect of automatically reverting the scrollbar policies to their default values (HORIZONTAL_SCROLLBAR_NEVER and VERTICAL_SCROLLBAR_ALWAYS/NEVER). This also happens whenever the uicontrol is resized interactively (by resizing its container figure window, for example). It is therefore advisable to set jScrollPane’s ComponentResizedCallback property to “unrevert” the policies:

cbFunc = @(h,e) set(h,'VerticalScrollBarPolicy',20, 'HorizontalScrollBarPolicy',30);
hjScrollPane = handle(jScrollPane,'CallbackProperties');
set(hjScrollPane,'ComponentResizedCallback',cbFunc);

smart_scrollbars utility

I created a new utility called smart_scrollbars that implements all of this, which you can download from the Matlab File Exchange. The usage in Matlab code is very simple:

% Fix scrollbars for a specific listbox
hListbox = uicontrol('style','list', ...);
smart_scrollbars(hListbox)
 
% Fix scrollbars for a specific editbox
hEditbox = uicontrol('style','edit', 'max',5, ...);
smart_scrollbars(hEditbox)
 
% Fix all listbox/editbox scrollbars in a panel or figure
smart_scrollbars              % fixes all scrollbars in current figure (gcf)
smart_scrollbars(hFig)        % fixes all scrollbars in a specific figure
smart_scrollbars(hContainer)  % fixes all scrollbars in a container (panel/tab/...)

Performance considerations

Finding the underlying JScrollPane reference of Matlab listboxes/editboxes can take some time. While the latest version of findjobj significantly improved the performance of this, it can still take quite a while in complex GUIs. For this reason, it is highly advisable to limit the search to a Java container of the control that includes as few internal components as possible.

In R2014b or newer, this is easily achieved by wrapping the listbox/editbox control in a tightly-fitting invisible uipanel. The reason is that in R2014b, uipanels have finally become full-fledged Java components (which they weren’t until then), but more to the point they now contain a property with a direct reference to the underlying JPanel. By using this panel reference we limit findjobj‘s search only to the contained scrollpane, and this is much faster:

% Slower code:
hListbox = uicontrol('style','list', 'parent',hParent, 'pos',...);
smart_scrollbars(hListbox)
 
% Much faster (using a tightly-fitting transparent uipanel wrapper):
hPanel = uipanel('BorderType','none', 'parent',hParent, 'pos',...);  % same position/units/parent as above
hListbox = uicontrol('style','list', 'parent',hPanel, 'units','norm', 'pos',[0,0,1,1], ...);
smart_scrollbars(hListbox)

The smart_scrollbars utility detects cases where there is a potential for such speedups and reports it in a console warning message:

>> smart_scrollbars(hListbox)
Warning: smart_scrollbars can be much faster if the list/edit control is wrapped in a tightly-fitting uipanel (details)

If you wish, you can suppress this warning using code such as the following:

oldWarn = warning('off', 'YMA:smart_scrollbars:uipanel');
smart_scrollbars(hListbox)
warning(oldWarn);  % restore warnings

Musings on future compatibility

Going back to my friend’s question at the top of today’s post, the risk of future compatibility was highlighted in the recent release of Matlab R2016a, which introduced web-based uifigures and controls, for which the vast majority of Java hacks that I presented in this blog since 2009 (including today’s hack) will not work. While the full transition from Java-based to web-based GUIs is not expected anytime soon, this recent addition highlighted the risk inherent in using unsupported functionality.

Users can take a case-by-case decision whether any improved functionality or appearance using Java hacks is worth the extra risk: On one hand, such hacks have been quite stable and worked remarkably well for the past decade, and will probably continue working into 2020 or so (or longer if you keep using a not up-to-the-moment Matlab release, or if you create compiled applications). On the other hand, once they stop working sometime in R2020a (or whenever), major code rewrites may possibly be required, depending on the amount of dependency of your code on these hacks.

There is an obvious tradeoff between improved GUIs now and for the coming years, versus increased maintainability cost a few years in the future. Each specific GUI will have its own sweet spot on the wide spectrum between using no such hacks at all, through non-critical hacks that provide graceful functionality degradation if they ever fail, to major Java-based functionality that would require complete rework. It is certainly NOT an all-or-nothing decision. Users who take the conservative approach of using no unsupported feature at all, lose the opportunity to have professional grade Matlab GUIs today and in the upcoming years. Decisions, decisions, …

In any case, we can reduce the risk of using such hacks today by carefully wrapping all their code in try-catch blocks. This way, even if the code fails in some future Matlab release, we’d still be left with a working implementation based on fully-supported functionality. This is the reason why I’ve used such a block in the code snippet above, as well as in my smart_scrollbars utility. What this means is that you can safely use smart_scrollbars in your code today and if the worst happens and it stops working in a few years, then it will simply do nothing without causing any error. In other word, future compatibility in the form of graceful degradation. I strongly advise using such defensive coding techniques whenever you use unsupported features.

]]>
https://undocumentedmatlab.com/blog/smart-listbox-editbox-scrollbars/feed 4
Better MEX error messageshttps://undocumentedmatlab.com/blog/better-mex-error-messages https://undocumentedmatlab.com/blog/better-mex-error-messages#comments Wed, 24 Feb 2016 18:00:25 +0000 http://undocumentedmatlab.com/?p=6287
 
Related posts:
  1. Matlab mex in-place editing Editing Matlab arrays in-place can be an important technique for optimizing calculations. This article shows how to do it using Mex. ...
  2. Matlab’s internal memory representation Matlab's internal memory structure is explored and discussed. ...
  3. Undocumented Matlab MEX API Matlab's MEX API contains numerous undocumented functions, that can be extremely useful. ...
  4. Explicit multi-threading in Matlab part 3 Matlab performance can be improved by employing POSIX threads in C/C++ code. ...
 
]]>
I would like to introduce guest blogger Pavel Holoborodko, the developer of the Advanpix Multiprecision Computing Toolbox for MATLAB. Pavel has already posted here in the past as a guest blogger about undocumented Matlab MEX functions. Today he will discuss another little-known aspect of advanced MEX programming with Matlab.

MEX API provides two functions for proper handling of erroneous situations: the legacy mexErrMsgTxt and the newer mexErrMsgIdAndTxt. Both show error message, interrupt execution of a MEX module and return control to Matlab immediately.

Under the hood, these functions are implemented through C++ exceptions. The reason for this design choice is unclear. Throwing C++ exceptions across module boundary (e.g. from dynamic library to host process) is unsafe and generally considered as bad practice in software design. Exceptions are part of C++ run-time library and different versions of it might have incompatible implementations.

This restricts MEX to use only the same version of C++ run-time and GCC which were used to build that particular version of Matlab itself. This is one of the reasons why TMW distributes its own version of GCC/libstdc++ along with every release of Matlab and pushes developers to use it for MEX compilation.

Such unfortunate design decision have two unpleasant consequences:

  1. developers must use some old version of GCC (forget all the fancy stuff from C++11, C++14, etc.)
  2. compiled MEX modules most likely will not work on older/newer versions of MATLAB (re-compilation is required).

The good news is that both issues are solvable and I will write more about this in the future (it is actually possible to create single binary MEX module, which will work on any GNU Linux flavor regardless of the versions of libc/libstc++/gcc installed in the system or used in Matlab).

Here I propose just first step towards freedom – avoid direct usage of mexErrMsg** functions. Use a simple wrapper instead:

void mxShowCriticalErrorMessage(const char *msg)
{
    mxArray *arg;
    arg = mxCreateString(msg);
    mexCallMATLAB(0,0,1,&arg,"error");
}

The mxShowCriticalErrorMessage function calls Matlab’s built-in error function via the interpreter, with the error message as input parameter.

In addition to being safe, this approach potentially gives us better control over what additional information is shown together with error messages. Instead of a string, we can use an errorStruct as input argument to Matlab’s error function, with its fields tuned to our requirements (not shown here as I want to keep the example simple).

Even without tuning, output of mxShowCriticalErrorMessage is much more informative and user-friendly:

  • Error message from Matlab’s built-in functionality:

    >> A = magic(3);
    >> A(0)
    Subscript indices must either be real positive integers or logicals.

    Nice one-line message without any distracting information.
     

  • Error message from MEX using mexErrMsgTxt/mexErrMsgIdAndTxt:

    >> A = mp(magic(3));   % convert matrix to arbitrary precision type, provided by our toolbox
    >> A(0)                % subsref is called from toolbox, it is implemented in mpimpl.mex
    Error using mpimpl
    Subscript indices must either be real positive integers or logicals.
    Error in mp/subsref (line 860)
            [varargout{1:nargout}] = mpimpl(170, varargin{:});

    Intimidating four lines with actual error message lost in the middle. All the additional information is meaningless for end-user and actually misleading.

    The worst thing is that such error message is very different from what user get used to (see above one-liner), which leads to confusion if MEX plugin is properly working at all.
     

  • Error message from MEX using mxShowCriticalErrorMessage:

    >> A = magic(3);
    >> A(0)
    Error using mp/subsref (line 860)
    Subscript indices must either be real positive integers or logicals.

    Now the message is clear and short, with error description in the last line where the user focus is.
     

]]>
https://undocumentedmatlab.com/blog/better-mex-error-messages/feed 2
Simple GUI Tabs for Advanced Matlab Trading Apphttps://undocumentedmatlab.com/blog/simple-gui-tabs-for-advanced-matlab-trading-app https://undocumentedmatlab.com/blog/simple-gui-tabs-for-advanced-matlab-trading-app#comments Wed, 17 Feb 2016 18:00:40 +0000 http://undocumentedmatlab.com/?p=6275
 
Related posts:
  1. Panel-level uicontrols Matlab's uipanel contains a hidden handle to the title label, which can be modified into a checkbox or radio-button control...
  2. Multi-line tooltips Multi-line tooltips are very easy to set up, once you know your way around a few undocumented hiccups....
  3. Multi-line uitable column headers Matlab uitables can present long column headers in multiple lines, for improved readability. ...
  4. HG2 update HG2 appears to be nearing release. It is now a stable mature system. ...
 
]]>
I’d like to introduce guest blogger Alex Boykov, one of the developers of the Walk-Forward Analysis Toolbox for Matlab (WFAToolbox), which enables accelerated trading strategies development using Matlab. Today, Alex will explain how they used tabs in a way that can be replicated by any other Matlab GUI, not necessarily having the latest Matlab release.

In this post, we want to tell you about how we solved the problem of tab creation for WFAToolbox. We required the following criteria:

  • The tabs need to be attractive and look like tabs, not like buttons with panels
  • The tabs need to have been drawn using the editor GUIDE so that the contents of the tab panel can be easily edited
  • The tabs can be easily added and removed without significant code additions. They must be simple to use in different projects and tasks

The sophisticated user of Matlab might think that this is a trivial objective, seeing as there are numerous solutions for this problem in the Matlab Exchange and since Matlab R2014b, it supports creating native tabs with the help of uitab and uitabgroup functions. Also, with the addition of App Designer, it might appear that this issue will be solved with the new interface for GUI creation; tabs can be created right in the editor. However, in this post, we will attempt to explain why none of the methods above fit the three criteria stated and we will present our own solution for the tabs.

Regardless of the fact that we only took on the problem in 2013, when we first started creating our WFAToolbox, at the moment of writing this article (January 2016), this problem is still a relevant issue for many Matlab users. After the release of R2016a, it is doubtful the problem will be entirely solved. This is why we created our own example of a code which we have released on the Matlab File Exchange (see below).


Tab-enabled WFAToolbox (Matlab app for algorithmic trading)

Tab-enabled WFAToolbox (Matlab app for algorithmic trading)

1. The tabs have to look like tabs

When we created WFAToolbox, our goal was to create an application which would allow everyone interested to create a strategy for trading on the financial markets to be able to do so, along with having the opportunity to use the full potential of Matlab and its progressive tools, including genetic algorithms, parallel computing, econometrics, neural networks, and much, much more (basically, any data analysis that can be done in Matlab). At the same time, we do not want our users to spend time on developing an advanced software environment for testing, analysis, and strategy execution, but rather to do it from an easy-to use GUI. Thus, in WFAToolbox, you can create, test, and, finally, launch your own trading strategy or test a hypothesis within minutes, even with little prior knowledge of Matlab programming.

Of course, in order to fit these features into a single application, guarantee that it would be easy to understand even by beginners, and that it would be simple to operate, it was necessary to pay special attention to the graphic interface. In our opinion, perhaps the most intelligent solution for placing the many controls and functions necessary for sophisticated applications is by creating tabs. Because we knew that we were not the only ones who thought this way, we started looking for examples of codes that were previously created in the Matlab Exchange. We were very surprised when we found only a few solutions, most of which did not even match our first criteria of tab attractiveness! Unfortunately, a majority of them were old and quite unattractive (they looked more like buttons with panels). Even the new App Designer has tabs that in our eyes look more like buttons than tabs.

Having tried a lot of these utilities in our test versions, we came to the conclusion that Tab Panel Constructor v.2.8 would be the best option for us. It fits all three criteria above. In 2013, we used it quite successfully in our first versions of WFAToolbox. Everything looked great, but, unfortunately, it later turned out that the problem was far from being solved.


Tab-enabled WFAToolbox (Matlab app for algorithmic trading)

Tab-enabled WFAToolbox (Matlab app for algorithmic trading)

2. The tabs need to be created through GUIDE

Unfortunately, with time, it turned out that with the newer version of Matlab it didn’t work smoothly and the code we wanted to use as our solution practically fell apart in front of us. After adding a couple of elements in GUI, partial formatting was lost and we had to redo everything. The process of adding the tags created a lot of bugs which needed to be solved immediately.

In 2014, we already had more than 500 clients using our application. We started hearing, more and more often, that it would be great if the colors and locations of the tabs could be changed. It turned out that, depending on the operating system and Matlab version, the tab format changes. So, we made the decision to change our tabs.

By that time, a new version of Matlab was released, R2014b. It allowed us to build tabs with the help of the uitabgroup and uitab functions. The results looked exactly how we wanted: attractive, pleasant, and appeared like real tabs: UI With Tab Panel in Matlab R2014b. However, we were discouraged that they could not be created in GUIDE!

During that time, we were developing a module for WFAToolbox which would allow users to download data from Google Finance: 10,000+ free daily and intraday quotes from 20+ exchanges. Tabs were the easiest to use when switching between downloading free data from Google Finance and downloading custom user data from the Matlab Workspace. But entering so many elements through code and not through an editor? What will happen when we add 100,000+ free historical data from Yahoo Finance for futures, bonds, currency, stocks and others? We didn’t want to create all of this without the GUIDE editor! This is why we came to the conclusion that it is necessary for us to create a code of tabs, starting from scratch, so that they would correspond with all three of our criteria.


Tab-enabled WFAToolbox (Matlab app for algorithmic trading)

Tab-enabled WFAToolbox (Matlab app for algorithmic trading)

3. The tabs should be easy to add and edit

We chose the Simple Tab Panel, which has existed in the Matlab File Exchange since 2007, as a base for our new code because we considered it to be the most elegant and attractive example of GUIDE tabs. This solution fit our first two criteria, but we really wanted it to be universal and easy to use. We also wanted to have a simplified process of tab addition and deletion so that instead of having to copy and rewrite a large amount of code and other details, we could just add a single line of code. We wanted to save on labor costs, because we often add new features to WFAToolbox and this includes having to constantly add new elements to existing tabs, as well as adding new tabs.

So, we rewrote the code and created our own universal example so that everyone could use it to their advantage. We uploaded the code to the Matlab File Exchange, where it can be freely downloaded: Simple Optimized GUI Tab.

Next, we will describe how to use this code for tab addition and how to use the process for the implementation of tasks.

So, in order to add a new tab, you need to:

  1. Open GUIDE and apply uipanel and uitext in a way that will make uipanel easier to work with in the future, and place uitext in a place where the tab switch will be located.
  2. Rename the Tag of the uitext to [‘tab’,N,’text’], where N is the tab index. In our example, we are creating the 3rd tab, so our tag would be ‘tab3text’. Using this same principle, [‘tab’,N,’Panel’] needs to be renamed to tag of uipanel in the ‘tab3Panel’.
  3. Add the name of the new tab to the TabNames variable. In our example, we use ‘Tab3’ (but you can use any name).
TabNames = {'Tab 1','Tab 2','Tab3'};

How the code was created

The primary principle of how our code works is that we create the uipanel and uitext objects in GUIDE, then we take the uitext coordinates and replace the objects to the axes and text objects. We assign a callback function to them which works when the object is clicked on. The function makes the uipanels visible/invisible and changes the colors of tab.

Let’s look at the function code SimpleOptimizedTabs2.m, which is part of the Simple Optimized GUI Tab submission.

1. Tab settings

% Settings
TabFontSize = 10;
TabNames = {'Tab 1','Tab 2'};
FigWidth = 0.265;

If we change the parameters under Settings, we can control the appearance of our GUI and tabs. So, the parameter of TabFontSize changes the font size on the tab switch, and, with the help of TabNames we can rename or add tab names, and with FigWidth, we can determine the normalized width of the GUI.

2. Changing the figure width

% Figure resize
set(handles.SimpleOptimizedTab,'Units','normalized')
pos = get(handles. SimpleOptimizedTab, 'Position');
set(handles. SimpleOptimizedTab, 'Position', [pos(1) pos(2) FigWidth pos(4)])

The GUI width changes in the code because it isn’t comfortable to manually stretch and narrow the figure. It is more beneficial to see the contents of all tabs and work with them without having to change the width every time you make a small change. If you want to make your uipanels bigger than in the example, then do this with the GUIDE editor. However, don’t forget to change the FigWidth parameter.

Please note that, due to the peculiarities of the editor, you cannot narrow a figure by height without shifting tab locations. You can only do this if you are changing the width, so we only recommend adding tabs by increasing the width of the figure and not the length.

3. Creating tabs

Do the following for each tab: obtain the uitext coordinates, which we entered into the GUI panel, and position the axes and text using these coordinates (using the necessary settings of external apparel). Using the ButtonDownFcn parameter, we can link the callback function, called ClickOnTab, in order to switch tabs when clicking on the text or axes.

% Tabs Execution
handles = TabsFun(handles,TabFontSize,TabNames);
 
% --- TabsFun creates axes and text objects for tabs
function handles = TabsFun(handles,TabFontSize,TabNames)
 
% Set the colors indicating a selected/unselected tab
handles.selectedTabColor=get(handles.tab1Panel,'BackgroundColor');
handles.unselectedTabColor=handles.selectedTabColor-0.1;
 
% Create Tabs
TabsNumber = length(TabNames);
handles.TabsNumber = TabsNumber;
TabColor = handles.selectedTabColor;
for i = 1:TabsNumber
    n = num2str(i);
 
    % Get text objects position
    set(handles.(['tab',n,'text']),'Units','normalized')
    pos=get(handles.(['tab',n,'text']),'Position');
 
    % Create axes with callback function
    handles.(['a',n]) = axes('Units','normalized',...
                    'Box','on',...
                    'XTick',[],...
                    'YTick',[],...
                    'Color',TabColor,...
                    'Position',[pos(1) pos(2) pos(3) pos(4)+0.01],...
                    'Tag',n,...
                    'ButtonDownFcn',[mfilename,'(''ClickOnTab'',gcbo,[],guidata(gcbo))']);
 
    % Create text with callback function
    handles.(['t',n]) = text('String',TabNames{i},...
                    'Units','normalized',...
                    'Position',[pos(3),pos(2)/2+pos(4)],...
                    'HorizontalAlignment','left',...
                    'VerticalAlignment','middle',...
                    'Margin',0.001,...
                    'FontSize',TabFontSize,...
                    'Backgroundcolor',TabColor,...
                    'Tag',n,...
                    'ButtonDownFcn',[mfilename,'(''ClickOnTab'',gcbo,[],guidata(gcbo))']);
 
    TabColor = handles.unselectedTabColor;
end
 
% Manage panels (place them in the correct position and manage visibilities)
set(handles.tab1Panel,'Units','normalized')
pan1pos=get(handles.tab1Panel,'Position');
set(handles.tab1text,'Visible','off')
for i = 2:TabsNumber
    n = num2str(i);
    set(handles.(['tab',n,'Panel']),'Units','normalized')
    set(handles.(['tab',n,'Panel']),'Position',pan1pos)
    set(handles.(['tab',n,'Panel']),'Visible','off')
    set(handles.(['tab',n,'text']),'Visible','off')
end

Actually, if you have long tab names and you want to change the switch size, then it you may possibly need to correct the Position parameter for the text object by adding the correcting coefficients to it. Unfortunately, this is also a feature of GUIDE. If someone can solve this problem so that the text would always be shown in the middle of the switch tab regardless of the width, we would be happy to read any suggestions in the comments to this post.

4. The callback function ClickOnTab

The callback function ClickOnTab is used every time when clicking on the tab switch and the result of the switches are visible/invisible in the uipanels and in changes to the colors of the switches.

% --- Callback function for clicking on tab
function ClickOnTab(hObject,~,handles)
m = str2double(get(hObject,'Tag'));
 
for i = 1:handles.TabsNumber;
    n = num2str(i);
    if i == m
        set(handles.(['a',n]),'Color',handles.selectedTabColor)
        set(handles.(['t',n]),'BackgroundColor',handles.selectedTabColor)
        set(handles.(['tab',n,'Panel']),'Visible','on')
    else
        set(handles.(['a',n]),'Color',handles.unselectedTabColor)
        set(handles.(['t',n]),'BackgroundColor',handles.unselectedTabColor)
        set(handles.(['tab',n,'Panel']),'Visible','off')
    end
end

More information about our Walk-Forward Analysis Toolbox for Algorithmic Trading (WFAToolbox) can be found at wfatoolbox.com.

]]>
https://undocumentedmatlab.com/blog/simple-gui-tabs-for-advanced-matlab-trading-app/feed 15
Quirks with compiled Matlab DLLshttps://undocumentedmatlab.com/blog/quirks-with-compiled-matlab-dlls https://undocumentedmatlab.com/blog/quirks-with-compiled-matlab-dlls#respond Wed, 10 Feb 2016 19:00:00 +0000 http://undocumentedmatlab.com/?p=6262
 
Related posts:
  1. Speeding up compiled apps startup The MCR_CACHE_ROOT environment variable can reportedly help to speed-up deployed Matlab executables....
  2. uicontextmenu performance Matlab uicontextmenus are not automatically deleted with their associated objects, leading to leaks and slow-downs. ...
  3. Transparency in uicontrols Matlab uicontrols' CData property can be customized to provide background transparency....
  4. Matlab compiler bug and workaround Both the Matlab compiler and the publish function have errors when parsing block-comments in Matlab m-code. ...
 
]]>
I would like to introduce guest blogger Hanan Kavitz of Applied Materials. Today Hanan will discuss several quirks that they have encountered with compiled Matlab DLLs.

I work for Applied Materials Israel (PDC) in the algorithm development department. My group provides Matlab software solutions across all of our products. I am a big fan of Yair’s blog and was happy to receive an invitation to be a guest blogger.

In PDC we are using Matlab compiler and Java Builder to deploy our algorithmic products at the customer sites. The software we produce includes binaries of three sorts: exe, C++ dlls, and jar files, all compiled using Matlab deployment tools.

C++ dlls are not as common and are a relatively new kind of binary in production so there’s not as much experience using them in PDC and from time to time we discover weird behaviors that might be interesting to the readers. Here are three of the latest quirks that we encountered:

Leftover tmp files

Recently I faced a problem that googling revealed that I might be the first to encounter it (outside MathWorks development team) as there was no mention of this anywhere. It all started when I got a mail from a fellow developer with content of the type: “Hi Hanan, what are these dump files for ???” and a screenshot of tempdir showing multiple tmp files that all had a naming convention of ‘mathworks_tmp_XXX_YYY’ (xxx and yyy being some numbers, presumably process ids):

multiple leftover tmp files in tmpdir (click for details)

multiple leftover tmp files in tmpdir (click for details)

Each file was about 43MB in size, and combined they consumed the entire free space in the hard-disk, preventing applications from lunching.

At first I looked in our code for the code that will produce files with this naming convention and once I was sure there is no such code I was comfortable to assume that they are MathWorks internal-use files (indeed – who other than MathWorkers would call their files mathworks_tmp_…).

Compiling and running a small ‘hello world’ C++ dll showed that every time that a Matlab-compiled C++ dll executed and ran, a temp file is created and then deleted in the %tmp% dir (tempdir). If the process that runs this dll is ‘killed’ during the run, the temp file is not deleted and remains in the %tmp% folder, accumulating over time until the hard disk becomes completely full.

On another machine I found a whole bunch of different temp files with different extensions. Opening them in editor didn’t reveal their purpose. They all had two things in common:

  1. All had a naming convention mathworks_tmp_XXX__YYY, whatever the extension of the file may be.
  2. They are all useless when the Matlab application is done running.

Once this was clear, the solution was very simple:

  • When lunching the process that runs Matlab compiled dll, change the %tmp% directory to a new one.
  • Delete this directory once done running or on the next process start.

mclInitializeApplication

As a separate discussion, our compiled algorithm is embarrassingly parallel in nature so our C++ team is running it in parallel processes across several machines and many cores.

As with all Matlab-compiled C++ DLLs, it needs to be initialized with a call to mclInitializeApplication per running process. This worked well when we had a relatively small number of processes running concurrently but lately we encountered a problem that out of dozens of calls to this function from time to time a call is stuck (not failed – exactly that: stuck) and the process needs to be killed.

We don’t know why this happens and the solution we are currently using is pretty “shaky” at best – we retry the call to this function several times until it eventually works (and it does work after several tries).

I have a suspicion that something is not entirely parallel with this function and there exists some hidden internal shared resource among different processes but no way to know exactly as this is an internal MathWorks function.

Implicit multithreading

This is relatively old issue we encountered at the beginning when we just started using Matlab-compiled DLLs, so this might not surprise some readers. While Matlab’s builtin (automatic) multithreading is great when used in the MATLAB IDE (non-compiled), it creates a problem when running compiled DLLs in parallel across multiple processes: this multithreading ‘starves’ for cores because they are busy at running other processes.

The solution is simple and well documented – when calling mclInitializeApplication before launching the DLL, pass it the singleCompThread flag to prevent implicit multithreading. We found this single threaded DLL to run faster because it reduced CPU ‘starvation’ drastically in a multi process environment.

Addendum (by Yair again):

I am delighted to announce that a few days ago I received an apparently-automated email from MathWorks telling me that 4 of the issues that I have reported early last year were fixed in R2015b. IMHO, this marks a very important step of closing the loop with the original issue reporter, something that I have suggested publicly back in December 2014 (and also privately over the years). I believe that such automated feedbacks increase user motivation to report misbehaving or missing features, which will ultimately make Matlab better for the benefit of us all. I also believe that this proves once again my claim that beneath the corporate jargon, MathWorks is indeed a company run by engineers like us, who cares about its users and the interaction with them more than typical in the corporate world. I can only praise this email and hope that it is now part of the ongoing development release process, rather than being an isolated case. If I may suggest a followup improvement, it would be nice to receive such emails before the new release is out (i.e., during the beta phase) so that the original reporter could have a chance to test it on the beta before it actually goes live. But in any case, thanks MathWorks for listening and making yet another improvement :-)

Happy New Year of the Monkey everybody!

]]>
https://undocumentedmatlab.com/blog/quirks-with-compiled-matlab-dlls/feed 0
Convolution performancehttps://undocumentedmatlab.com/blog/convolution-performance https://undocumentedmatlab.com/blog/convolution-performance#comments Wed, 03 Feb 2016 19:00:35 +0000 http://undocumentedmatlab.com/?p=6256
 
Related posts:
  1. tic / toc – undocumented option Matlab's built-in tic/toc functions have an undocumented option enabling multiple nested clockings...
  2. Plot LimInclude properties The plot objects' XLimInclude, YLimInclude, ZLimInclude, ALimInclude and CLimInclude properties are an important feature, that has both functional and performance implications....
  3. Matrix processing performance Matrix operations performance is affected by internal subscriptions in a counter-intuitive way....
  4. Preallocation performance Preallocation is a standard Matlab speedup technique. Still, it has several undocumented aspects. ...
 
]]>
MathWorks’ latest MATLAB Digest (January 2016) featured my book “Accelerating MATLAB Performance“. I am deeply honored and appreciative of MathWorks for this.

Accelerating MATLAB Performance bookI would like to dedicate today’s post to a not-well-known performance trick from my book, that could significantly improve the speed when computing the convolution of two data arrays. Matlab’s internal implementation of convolution (conv, conv2 and convn) appears to rely on a sliding window approach, using implicit (internal) multithreading for speed.

However, this can often be sped up significantly if we use the Convolution Theorem, which states in essence that conv(a,b) = ifft(fft(a,N) .* fft(b,N)), an idea proposed by Bruno Luong. In the following usage example we need to remember to zero-pad the data to get comparable results:

% Prepare the input vectors (1M elements each)
x = rand(1e6,1);
y = rand(1e6,1);
 
% Compute the convolution using the builtin conv()
tic, z1 = conv(x,y); toc
 => Elapsed time is 360.521187 seconds.
 
% Now compute the convolution using fft/ifft: 780x faster!
n = length(x) + length(y) - 1;  % we need to zero-pad
tic, z2 = ifft(fft(x,n) .* fft(y,n)); toc
 => Elapsed time is 0.463169 seconds.
 
% Compare the relative accuracy (the results are nearly identical)
disp(max(abs(z1-z2)./abs(z1)))
 => 2.75200348450538e-10

This latest result shows that the results are nearly identical, up to a tiny difference, which is certainly acceptable in most cases when considering the enormous performance speedup (780x in this specific case). Bruno’s implementation (convnfft) is made even more efficient by using MEX in-place data multiplications, power-of-2 FFTs, and use of GPU/Jacket where available.

It should be noted that the builtin Matlab functions can still be faster for relatively small data arrays, or if your machine has a large number of CPU cores and free memory that Matlab’s builtin conv* functions can utilize, and of course also depending on the Matlab release. So, your mileage might well vary. But given the significant speedup potential, I contend that you should give it a try and see how well it performs on your specific system and data.

If you have read my book, please be kind enough to post your feedback about it on Amazon (link), for the benefit of others. Thanks in advance!

]]>
https://undocumentedmatlab.com/blog/convolution-performance/feed 6
Graphic sizing in Matlab R2015bhttps://undocumentedmatlab.com/blog/graphic-sizing-in-matlab-r2015b https://undocumentedmatlab.com/blog/graphic-sizing-in-matlab-r2015b#comments Wed, 20 Jan 2016 18:00:31 +0000 http://undocumentedmatlab.com/?p=6244
 
Related posts:
  1. HG2 update HG2 appears to be nearing release. It is now a stable mature system. ...
  2. Modifying default toolbar/menubar actions The default Matlab figure toolbar and menu actions can easily be modified using simple pure-Matlab code. This article explains how....
  3. FIG files format FIG files are actually MAT files in disguise. This article explains how this can be useful in Matlab applications....
  4. A couple of internal Matlab bugs and workarounds A couple of undocumented Matlab bugs have simple workarounds. ...
 
]]>
I would like to introduce Daniel Dolan of Sandia National Laboratories. Dan works on a variety of data analysis projects in Matlab, and is an active lurker on MATLAB Central. Dan has a habit of finding interesting bugs for the Mac version of Matlab. Today he will discuss graphic sizing in Matlab and important changes that occurred in release R2015b.

Matlab-generated graphics are often not displayed at their requested size. This problem has been known for some time and has a well-known solution: setting the root object’s ScreenPixelsPerInch property to the display’s actual DPI (dots per inch) value. Release R2015b no longer supports this solution, creating problems for publication graphics and general readability.

Physical sizing in R2015a vs. R2015b (click for full-size)

Physical sizing in R2015a vs. R2015b (click for full-size)


Physical sizing

Matlab supports graphic sizing in various physical units: inches, centimeters, and points. For example:

figure; axes('Box','on', 'Units','inches','Position',[0.3 0.3 4 4]);

requests to display an axes having square sizes measuring exactly 4″ (101.6 mm) each. It is evident, however, that the displayed axes is smaller than 4″. The mismatch between requested and physical size depends on the display and operating system — go ahead, try it on your system. The problem is particularly severe on Mac laptops, presumably even worse for those with Retina displays.

The problem is that Matlab cannot determine pixel size, which varies from one display to the other. Generating a figure spanning a particular number of pixels (e.g., 1024 x 768) is easy, but absolute physical units requires a conversion factor called ScreenPixelsPerInch, which is a root property (see related post on setting/getting default graphics property values):

DPI = 110;                             % dots per inch for my 27" Apple Cinema Display
set(0,    'ScreenPixelsPerInch',DPI);  % all releases prior to R2015b
set(groot,'ScreenPixelsPerInch',DPI);  % R2014b through R2015a

DPI values tend to be higher for laptops, usually in the 120-130 range. Retina displays are supposed to be >300 DPI, but I have not been able to test that myself.

There are several ways to determine the correct DPI setting for a particular display. It may be available in the hardware specifications, and it can be calculated from the diagonal size and the number of pixels. Unfortunately these methods are not always reliable. If you really care about physical sizing, the best approach is to actually calibrate your display. There are tools for doing this at Matlab Central, but it’s not hard to do manually:

  • Create a figure.
  • Manually resize the figure to match a convenient width. I often use a piece of US letter paper as 8.5″ guide on the display.
  • Determine the width of the figure in pixels:
    set(gcf,'Units','pixels');
    pos = get(gcf,'Position');
    width = 8.5; % inches
    DPI = pos(3) / width;

I usually apply the DPI settings in my startup file so that Matlab begins with a calibrated display.

What changed in 2015b?

ScreenPixelsPerInch is a read-only property in R2015b, so display calibration no longer works. The following sequence of commands:

figure('Units','inches', 'PaperPositionMode','auto', 'Position',[0 0 4 4]);
set(gcf, 'MenuBar','none', 'ToolBar','none', 'DockControls','off', 'NumberTitle','off');
axes('FontUnits','points', 'FontSize',10);
image

now renders differently in R2015b than does for a calibrated display in R2015a. Differences between the two outputs are shown in the screenshot at the top of this post. The grid behind the figures was rendered at 8.5″ x 8.5″ inches on my display; if your browser’s zoom level isn’t 100%, it may appear larger or smaller.

A side effect of improper graphic sizing is that text is difficult to read — the uncalibrated axes labels are clearly smaller than 10 points. These examples were rendered on ~110 DPI display. Matlab assumes that Macs use 72 DPI (96 DPI on Windows), so graphics appear at 65% of the request size.

The loss of ScreenPixelsPerInch as an adjustable setting strongly affects anyone using Matlab for publication graphics. Scientific and engineering journals are extremly strict about figure widths. With a calibrated screen, figure appear exactly as they will when printed to a file (usually EPS or PDF). Figures are often made as small as possible to and densely packed to save journal space, and accurate sized display helps the author determine legibility. Displaying accurately sized graphics is very difficult in R2015b, which is unfortunate given the many enhancements in this release.

Developers who create graphical interfaces for other users should also care about this change. A common complaint I get is that text and control labels is too small to easily read. Screen calibration deals with this problem, but this option is no longer available.

Where do we go from here?

I reported the above issues to the Mathworks several months ago. It does not appear as a formal bug, but technical support is aware of the problem. The change is part of the “DPI aware” nature of release R2015b. So far I have found no evidence this release is any more aware of pixel size than previous releases, but my experience is limited to non-Retina Macs. I welcome input from users on other operating systems, particularly those with high-resolution displays.

To be fair, correct physical sizing is not an easy across the many platforms that Matlab runs on. Display resolution is particularly tricky when it changes during a Matlab session, such as when computer is connector to projector/television or a laptop is connected to a docking station.

Thankfully, printed graphic sizes are rendered correctly when a figure’s PaperPositionMode property is 'auto'. Many users can (and will) ignore the display problem if they aren’t dealing with strict size requirements and text legibility isn’t too bad. Some users may be willing to periodically print publication figures to externally verify sizing, but this breaks the interactive nature of Matlab figures.

A potential work around is the creating of a new figure class that oversizes figures (as needed) to account for a particular display. I started working on such a class, but the problem is more complicated than one might think:

  • Child objects (axes, uicontrols, etc.) also must be resized if they are based on physical units.
  • Resized objects must be temporarily restored to their original size for printing, and new objects must be tracked whenever they are added.
  • Figure resolution may need to be changed when moving to different computer systems.

These capabilities are quite possible to implement, but this is a complicated solution to problem that was once easy to fix.

Retina displays don’t suffer as badly as one might think from the DPI mismatch. Even though the display specification may be greater than 200 DPI, OS X and/or Matlab must perform some intermediate size transformations. The effective DPI in R2015a is 110-120 for 13-15″ MacBook Pro laptops (at the default resolution). Objected sized with physical units still appear smaller than they should (~72/110), but not as small as I expected (<72/200).

Effect pixel size can also be changed by switching between different monitor scalings. This isn’t entirely surprising, but it can lead to some interesting results because Matlab only reads these settings at startup. Changing the display scaling during a session can cause square figures to appear rectangular. Also, the effective DPI changes for setting: I could reach values of ~60-110 DPI on an Apple Cinema Display.

So where does this leave us? Display calibration was always a finicky matter, but at least in principle one could make graphics appear exactly the same size on two different displays. Now it seems that sizing is completely variable between operation systems, displays, and display settings. For publication graphics, there will almost always be a disconnect between figure size on the screen and the printed output; some iteration may be needed to ensure everything looks right in the finished output. For graphical interfaces, font sizes may need to generated in normalized units and then converted to pixels (to avoid resizing).

Physical accuracy may not be important for non-publication figures, but the issue of text legibility remains. Some text objects–such as axes and tick labels–can easily be resized because the parent axes automatically adjusts itself as needed. Free floating text objects and uincontrols are much more difficult to deal with. Controls are often sized around the extent of their text label, so changing font sizes may require changes to the control position; adjacent controls may overlap after resizing for text clarity. Normalized units partially solve this problem, but their effect on uicontrols is not always desirable: do you really want push buttons to get larger/smaller when the figure is resized?

Can you think of a better workaround to this problem? If so, then please post a comment below. I will be very happy to hear your ideas, as I’m sure others who have high resolution displays would as well.

(cross-reference: CSSM newsgroup post)

Addendum Dec 31, 2016: Dan Dolan just posted a partial workaround on the MathWorks File Exchange. Also see the related recent article on working with non-standard DPI values.

]]>
https://undocumentedmatlab.com/blog/graphic-sizing-in-matlab-r2015b/feed 8
Solving a Matlab MCOS bughttps://undocumentedmatlab.com/blog/solving-a-matlab-mcos-bug https://undocumentedmatlab.com/blog/solving-a-matlab-mcos-bug#comments Wed, 02 Dec 2015 15:55:21 +0000 http://undocumentedmatlab.com/?p=6100
 
Related posts:
  1. Performance: accessing handle properties Handle object property access (get/set) performance can be significantly improved using dot-notation. ...
  2. Spy Easter egg The built-in Matlab function spy has an undocumented feature (Easter egg) when it is called with no input arguments....
  3. Undocumented scatter plot jitter Matlab's scatter plot can automatically jitter data to enable better visualization of distribution density. ...
  4. HG2 update HG2 appears to be nearing release. It is now a stable mature system. ...
 
]]>
A few days ago, one of my consulting clients asked me to help him with a very strange problem: he had a Matlab class having a constant property that holds a reference to some handle class object. The problem was that when he tried to modify the property’s inner values he got Matlab run-time errors because the inner value apparently remained unmodified!

Here is a distilled version of my client’s classes:

classdef (Abstract) MainClass
    properties (Constant)
        inner = InnerClass
    end
    methods (Static)
        function setInnerValue(newValue)
            MainClass.inner.value1 = newValue;
        end
    end
end
 
classdef InnerClass < handle
    properties
        value1
        value2
    end
end

And the strange run-time behavior:

>> MainClass.inner.value1 = 5
MainClass = 
    inner: [1x1 struct]
 
>> MainClass.inner.value2  % causes a strange run-time error!
Reference to non-existent field 'value2'.
 
>> MainClass.inner.value1  % strange - value1 appears unmodified!
ans =
     []
 
>> MainClass.inner  % strange - value1 appears ok here, but where is value2 ?!
ans = 
    value1: 5
 
>> MainClass.setInnerValue(7)  % another strange run-time error!
Reference to non-existent field 'setInnerValue'.
 
>> clear classes  % let's try it afresh...
>> MainClass.setInnerValue(7)  % looks ok, no error...
>> MainClass.inner   % strange - now we have both value1 & value2, but value1 is not updated!
ans = 
  InnerClass with properties:
 
    value1: []
    value2: []
 
>> MainClass.inner.value1 = 9   % one last attempt, that also fails!
MainClass = 
    inner: [1x1 struct]
 
>> MainClass.inner
ans = 
  InnerClass with properties:
 
    value1: []
    value2: []
 
>> MainClass.inner.value1
ans =
     []

Understanding the [buggy?] behavior

What the heck is going on here? did Matlab’s MCOS flip its lid somehow? Well, apparently not. It turns out that all these strange behaviors can be attributed to a single Matlab oddity (I call it a “bug”) in its class object system (MCOS) implementation. Understanding this oddity/bug then leads to a very simply workaround.

The underlying problem is that Matlab does not understand MainClass.inner.value1 to mean “the value1 property of the inner property of the MainClass class”. Matlab does understand MainClass.inner to mean “the inner property of the MainClass class”, but adding another dereferencing level (.value1) in the same Matlab expression is too complex for MCOS, and it does not properly understand it.

This being the case, Matlab’s internal interpreter reverts to the simplistic understanding that MainClass.inner.value1 means “the value1 field of the struct inner, which is itself a field of the outer struct named MainClass“. In fact, this creates a new struct variable named MainClass in our current workspace. Still, due to Matlab’s internal precedence rules, the MainClass class overshadows the new variable MainClass, and so when we try to read (as opposed to update) MainClass.inner we are actually referencing the inner reference handle of the MainClass class, rather than the corresponding field in the new struct. The reference handle’s value1 property remains unmodified because whenever we try to set MainClass.inner.value1 to any value, we’re just updating the struct variable in the local workspace.

Unfortunately, no run-time warning is issued to alert us of this. However, if we load MainClass.m in the Matlab editor, we get a somewhat-cryptic MLint warning that hints at this:

MLint warning about class object appearing as a struct

Confusing? indeed!

(However, refer to the reader comments below, which provide insight into the reasoning of what’s going on here, and contend that this is actually the expected behavior. I will let the readers decide for themselves whether they agree or not.)

The workaround

Now that we understand the source of all the problems above, the solution is easy: help Matlab’s internal interpreter understand that we want to reference a property of the inner object reference. We do this by simply splitting the 3-element construct MainClass.inner.value1 into two parts, first storing innerObj = MainClass.inner in a temporary local variable (innerObj), then using it to access the internal innerObj.value1 property:

innerObj = MainClass.inner;
innerObj.value1 = 7;

Hopefully, in some future Matlab release, MCOS will be smarter than today and automatically handle multi-element MCOS constructs, as well as it currently does for 2-element ones (or multi-element Java/.Net/COM constructs).

Have you ever encountered any other Matlab bug that appears perplexing at first but has a very simple workaround as above? If so, then please share your findings in a comment below.

]]>
https://undocumentedmatlab.com/blog/solving-a-matlab-mcos-bug/feed 10