Don’t fix it, if it ain’t broke…

In Applied Materials Israel (PDC) we use Matlab code for both algorithm development and deployment (production). As part of the dev-ops build system, which builds our product software versions, we build Matlab artifacts (binaries) from the Matlab source code.
A typical software version has several hundreds Matlab artifacts that are automatically rebuilt on a daily basis, and we have many such versions – totaling many thousands of compilations each day.
This process takes a long time, so we were looking for a way to make it more efficient.
The idea that we chose to implement sounds simple – take a single binary module in any software version (Ex. foo.exe – Matlab-compiled exe) and check it: if the source code for this module has not changed since the last compilation then simply don’t compile it, just copy it from previous software version repository. Since most of our code doesn’t change daily (some of it hasn’t changed in years), we can skip the compilation time of most binaries and just copy them from some repository of previously compiled binaries.

In a broader look, avoiding lengthy compilations cycles by not compiling unchanged code is a common programming practice, implemented by all modern compilers. For example, the ‘make’ utility uses a ‘makefile’ to check the time stamps of all dependencies of every object file in order to decide which object requires recompilation. In reality, this is not always the best solution as time stamps may be incorrect, but it works well in the vast majority of cases.
Coming back to Matlab, now comes the hard part – how could our build system know that nothing has changed in module X and that something has changed in module Y? How does it even know which source files it needs to ensure didn’t change?
The credit for the idea goes to my manager, Lior Cohen, as follows: You can actually check the dependency of a given binary after compilation. The basis of the solution is that a Matlab executable is in fact a compressed (zip) file. The idea is then to:

1. Compile the binary once
2. Unzip the binary and “see” all your dependencies (source files are encrypted and resources are not, but we only need the list of file names – not their content).
3. Now build a list of all your dependency files and compute the CRC value of each from the source control. Save it for the next time you are required to compile this module.
4. In the next compilation cycle, find this dependency list, review it, dependency source file at a time and make sure CRC of the dependency hasn’t changed since last time.
5. If no dependency CRC has changed, then copy the binary from the repository of previous software version, without compiling.
6. Otherwise, recompile the binary and rebuild the CRC list of all dependencies again, in preparation for the next compilation cycle.

That’s it! That simple? Well… not really – the reality is a bit more complex since there are many other dependencies that need to be checked. Some of them are:

1. Did the requested Matlab version of the binary change since the last compilation?
2. Did the compilation instructions themselves (we have a sort of ‘makefile’) change?

Basically, I implemented a policy that if anything changed, or if the dependency check itself failed, then we don’t take any chances and just compile this binary. Keeping in mind that this dependencies check and file copying is much faster than a Matlab compilation, we save a lot of actual compilation time using this method.
Bottom line: Given a software version containing hundreds of compilation instructions to execute and assuming not much has changed in the version (which is often the case), we skip over 90% of compilations altogether and only rebuild what really changed. The result is a version build that takes about half an hour, instead of many hours. Moreover, since the compilation process is working significantly less, we get fewer failures, fewer stuck or crashed mcc processes, and [not less importantly] less maintenance required by me.
Note that in our implementation we rely on the undocumented fact that Matlab binaries are in fact compressed zip archives. If and when a future Matlab release will change the implementation such that the binaries will no longer be zip archives, another way will need to be devised in order to ensure the consistency of the target executable with its dependent source files.

Don’t kill it, if it ain’t bad…

I want to share a very weird issue I investigated over a year ago when using Matlab compiled exe. It started with a user showed me a Matlab compiled exe that didn’t run – I’m not talking about a regular Matlab exception: the process was crashing with an MS Windows popup window popping, stating something very obscure.
It was a very weird behavior that I couldn’t explain – the compiler seemed to work well but the compiled executable process kept crashing. Compiling completely different code showed the same behavior.
This issue has to do with the system compiler configuration that is being used. As you might know, when installing the Matlab compiler, before the first compilation is ever made, the user has to state the C compiler that the Matlab compiler should use in its compilation process. This is done by command ‘mbuild –setup’. This command asks the users to choose the C compiler and saves the configuration (batch file back then, xml in the newer versions of Matlab) in the user’s prefdir folder. At the time we were using Microsoft Visual C++ compiler 9.0 SP1.
The breakthrough in the investigation came when I ran mcc command with –verbose flag, which outputs much more compilation info than I would typically ever want… I discovered that although the target executable file had been created, a post compilation step failed to execute, while issuing a very cryptic error message:

mt.exe : general error c101008d: Failed to write the updated manifest to the resource of file “…”. Access is denied.

set POSTLINK_CMDS2=mt.exe -outputresource: %MBUILD_OUTPUT_FILE_NAME%;%MANIFEST_RESOURCE% -manifest "%MANIFEST_FILE_NAME%"

This line of code takes an XML manifest file and inserts it into the generated binary file (additional details).
If you open a valid R2010a (and probably other old versions as well) Matlab-generated exe in a text editor you can actually see a small XML code embedded in it, while in a non-functioning exe I could not see this XML code.
So why would this command fail?
It turned out, as funny as it sounds, to be an antivirus issue – our IT department updated its antivirus policies and this ‘post link’ command suddenly became an illegal operation. Once our IT eased the policy, this command worked well again and the compiled executables stopped crashing, to our great joy.

The post Matlab compilation quirks – take 2 appeared first on Undocumented Matlab.

I’ve been using Matlab for many years now and from time to time I need to profile low-throughput code. When I profile this code sometimes I realize that a computational ‘bottleneck’ is due to a builtin Matlab function (part of the core language). I can often find ways to accelerate such builtin functions and get significant speedup in my code.

I recently found Matlab’s builtin rmfield function being too slow for my needs. It works great when one needs to remove a few fields from a small structure, but in our case we needed to remove thousands of fields from a structure containing about 5000 fields – and this is executed in a function that is called many times inside an external loop. The program was significantly sluggish.

It started when a co-worker asked me to look at a code that looked just slightly more intelligent than this:

for i = 1:5000
    myStruct = rmfield(myStruct,fieldNames{i});
end

Running this code within a tic/toc pair yielded the following results:

>> tic; myFunc(); t1 = toc
t1 =
      25.7713

In my opinion 25.77 secs for such a simple functionality seems like an eternity…

The obvious thing was to change the code to the documented faster (vectorized) version:

>> tic; myStruct = rmfield(myStruct,fieldNames); t2 = toc
t2 =
      0.6097

This is obviously much better but since rmfield is called many times in my application, I needed something even better. So I profiled rmfield and was not happy with the result.

The original code of rmfield (%matlabroot%/toolbox/matlab/datatypes/rmfield.m) looks something like this (I deleted some non-essential code for brevity):

function t = rmfield(s,field)

  % get fieldnames of struct
  f = fieldnames(s);

  % Determine which fieldnames to delete.
  idxremove = [];
  for i=1:length(field)
    j = find(strcmp(field{i},f) == true);
    idxremove = [idxremove;j];
  end

  % set indices of fields to keep
  idxkeep = 1:length(f);
  idxkeep(idxremove) = [];

  % remove the specified fieldnames from the list of fieldnames.
  f(idxremove,:) = [];

  % convert struct to cell array
  c = struct2cell(s);

  % find size of cell array
  sizeofarray = size(c);
  newsizeofarray = sizeofarray;

  % adjust size for fields to be removed
  newsizeofarray(1) = sizeofarray(1) - length(idxremove);

  % rebuild struct
  t = cell2struct(reshape(c(idxkeep,:),newsizeofarray),f);

When I profiled the code, the highlighted row was the bottleneck I was looking for.

First, I noticed the string comparison equals to true part – while '==true' is not the cause of the bottleneck, it does leave an impression of bad coding style 🙁 Perhaps this code was created as some apprentice project, which might also explain its suboptimal performance.

The real performance problem here is that for each field that we wish to remove, rmfield compares it to all existing fields to find its location in a cell array of field names. This is algorithmically inefficient and makes the code hard to understand (just try – it took me hard, long minutes).

So, I created a variant of rmfield.m called fast_rmfield.m, as follows (again, omitting some non-essential code):

function t = fast_rmfield(s,field)
  % get fieldnames of struct
  f = fieldnames(s);
  [f,ia] = setdiff(f,field,'R2012a');

  % convert struct to cell array
  c = squeeze(struct2cell(s));

  % rebuild struct
  t = cell2struct(c(ia,:),f)';

This code is much shorter, easier to explain and maintain, but also (and most importantly) much faster:

>> tic; myStruct = fast_rmfield(myStruct,fieldNames); t3 = toc
t3 =
      0.0302
>> t2/t3
ans =
      20.1893

This resulted in a speedup of ~850x compared to the original version (of 25.77 secs), and ~20x compared to the vectorized version. A nice improvement in my humble opinion…

The point in all this is that we can and should rewrite Matlab builtin functions when they are too slow for our needs, whether it is found to be an algorithmic flaw (as in this case), extraneous sanity checks (as in the case of ismember or datenum), bad default parameters (as in the case of fopen/fwrite or scatter), or merely slow implementation (as in the case of save, cellfun, or the conv family of functions).

A good pattern is to save such code pieces in file names that hint to the original code. In our case, I used fast_rmfield to suggest that it is a faster alternative to rmfield.

Do you know of any other example of a slow implementation in a built-in Matlab function that can be optimized? If so, please leave a comment below.

The post rmfield performance appeared first on Undocumented Matlab.

Leftover tmp files

Recently I faced a problem that googling revealed that I might be the first to encounter it (outside MathWorks development team) as there was no mention of this anywhere. It all started when I got a mail from a fellow developer with content of the type: “Hi Hanan, what are these dump files for ???” and a screenshot of tempdir showing multiple tmp files that all had a naming convention of ‘mathworks_tmp_XXX_YYY’ (xxx and yyy being some numbers, presumably process ids):

1. All had a naming convention mathworks_tmp_XXX__YYY, whatever the extension of the file may be.
2. They are all useless when the Matlab application is done running.

Once this was clear, the solution was very simple:

• When lunching the process that runs Matlab compiled dll, change the %tmp% directory to a new one.
• Delete this directory once done running or on the next process start.

mclInitializeApplication

As a separate discussion, our compiled algorithm is embarrassingly parallel in nature so our C++ team is running it in parallel processes across several machines and many cores.
As with all Matlab-compiled C++ DLLs, it needs to be initialized with a call to mclInitializeApplication per running process. This worked well when we had a relatively small number of processes running concurrently but lately we encountered a problem that out of dozens of calls to this function from time to time a call is stuck (not failed – exactly that: stuck) and the process needs to be killed.
We don’t know why this happens and the solution we are currently using is pretty “shaky” at best – we retry the call to this function several times until it eventually works (and it does work after several tries).
I have a suspicion that something is not entirely parallel with this function and there exists some hidden internal shared resource among different processes but no way to know exactly as this is an internal MathWorks function.

Implicit multithreading

This is relatively old issue we encountered at the beginning when we just started using Matlab-compiled DLLs, so this might not surprise some readers. While Matlab’s builtin (automatic) multithreading is great when used in the MATLAB IDE (non-compiled), it creates a problem when running compiled DLLs in parallel across multiple processes: this multithreading ‘starves’ for cores because they are busy at running other processes.
The solution is simple and well documented – when calling mclInitializeApplication before launching the DLL, pass it the singleCompThread flag to prevent implicit multithreading. We found this single threaded DLL to run faster because it reduced CPU ‘starvation’ drastically in a multi process environment.

Addendum (by Yair again):

I am delighted to announce that a few days ago I received an apparently-automated email from MathWorks telling me that 4 of the issues that I have reported early last year were fixed in R2015b. IMHO, this marks a very important step of closing the loop with the original issue reporter, something that I have suggested publicly back in December 2014 (and also privately over the years). I believe that such automated feedbacks increase user motivation to report misbehaving or missing features, which will ultimately make Matlab better for the benefit of us all. I also believe that this proves once again my claim that beneath the corporate jargon, MathWorks is indeed a company run by engineers like us, who cares about its users and the interaction with them more than typical in the corporate world. I can only praise this email and hope that it is now part of the ongoing development release process, rather than being an isolated case. If I may suggest a followup improvement, it would be nice to receive such emails before the new release is out (i.e., during the beta phase) so that the original reporter could have a chance to test it on the beta before it actually goes live. But in any case, thanks MathWorks for listening and making yet another improvement 🙂
Happy New Year of the Monkey everybody!

The post Quirks with compiled Matlab DLLs appeared first on Undocumented Matlab.