- Undocumented Matlab - https://undocumentedmatlab.com -
Quirks with compiled Matlab DLLs
Posted By Yair Altman On February 10, 2016 | No Comments
I would like to introduce guest blogger Hanan Kavitz [1] of Applied Materials [2]. Today Hanan will discuss several quirks that they have encountered with compiled Matlab DLLs.
I work for Applied Materials Israel (PDC) in the algorithm development department. My group provides Matlab software solutions across all of our products. I am a big fan of Yair’s blog and was happy to receive an invitation to be a guest blogger.
In PDC we are using Matlab compiler and Java Builder to deploy our algorithmic products at the customer sites. The software we produce includes binaries of three sorts: exe, C++ dlls, and jar files, all compiled using Matlab deployment tools.
C++ dlls are not as common and are a relatively new kind of binary in production so there’s not as much experience using them in PDC and from time to time we discover weird behaviors that might be interesting to the readers. Here are three of the latest quirks that we encountered:
Recently I faced a problem that googling revealed that I might be the first to encounter it (outside MathWorks development team) as there was no mention of this anywhere. It all started when I got a mail from a fellow developer with content of the type: “Hi Hanan, what are these dump files for ???” and a screenshot of tempdir showing multiple tmp files that all had a naming convention of ‘mathworks_tmp_XXX_YYY’ (xxx and yyy being some numbers, presumably process ids):
Once this was clear, the solution was very simple:
As a separate discussion, our compiled algorithm is embarrassingly parallel [4] in nature so our C++ team is running it in parallel processes across several machines and many cores.
As with all Matlab-compiled C++ DLLs, it needs to be initialized with a call to mclInitializeApplication per running process. This worked well when we had a relatively small number of processes running concurrently but lately we encountered a problem that out of dozens of calls to this function from time to time a call is stuck (not failed – exactly that: stuck) and the process needs to be killed.
We don’t know why this happens and the solution we are currently using is pretty “shaky” at best – we retry the call to this function several times until it eventually works (and it does work after several tries).
I have a suspicion that something is not entirely parallel with this function and there exists some hidden internal shared resource among different processes but no way to know exactly as this is an internal MathWorks function.
This is relatively old issue we encountered at the beginning when we just started using Matlab-compiled DLLs, so this might not surprise some readers. While Matlab’s builtin (automatic) multithreading is great when used in the MATLAB IDE (non-compiled), it creates a problem when running compiled DLLs in parallel across multiple processes: this multithreading ‘starves’ for cores because they are busy at running other processes.
The solution is simple and well documented – when calling mclInitializeApplication before launching the DLL, pass it the singleCompThread flag to prevent implicit multithreading. We found this single threaded DLL to run faster because it reduced CPU ‘starvation’ drastically in a multi process environment.
I am delighted to announce that a few days ago I received an apparently-automated email from MathWorks telling me that 4 of the issues that I have reported early last year were fixed in R2015b. IMHO, this marks a very important step of closing the loop with the original issue reporter, something that I have suggested publicly [5] back in December 2014 (and also privately over the years). I believe that such automated feedbacks increase user motivation to report misbehaving or missing features, which will ultimately make Matlab better for the benefit of us all. I also believe that this proves once again my claim that beneath the corporate jargon, MathWorks is indeed a company run by engineers like us, who cares about its users and the interaction with them more than typical in the corporate world. I can only praise this email and hope that it is now part of the ongoing development release process, rather than being an isolated case. If I may suggest a followup improvement, it would be nice to receive such emails before the new release is out (i.e., during the beta phase) so that the original reporter could have a chance to test it on the beta before it actually goes live. But in any case, thanks MathWorks for listening and making yet another improvement 🙂
Happy New Year of the Monkey everybody!
Categories: Guest bloggers, Low risk of breaking in future versions, Undocumented feature
Article printed from Undocumented Matlab: https://undocumentedmatlab.com
URL to article: https://undocumentedmatlab.com/articles/quirks-with-compiled-matlab-dlls
URLs in this post:
[1] Hanan Kavitz: http://www.mathworks.com/matlabcentral/profile/authors/2392999-hanan-kavitz
[2] Applied Materials: http://www.appliedmaterials.com
[3] Image: http://undocumentedmatlab.com/images/compiler_tmp_files.jpg
[4] embarrassingly parallel: https://en.wikipedia.org/wiki/Embarrassingly_parallel
[5] suggested publicly: http://undocumentedmatlab.com/blog/couple-of-matlab-bugs-and-workarounds#comment-339302
[6] Matlab compilation quirks – take 2 : https://undocumentedmatlab.com/articles/matlab-compilation-quirks-take-2
[7] Speeding up compiled apps startup : https://undocumentedmatlab.com/articles/speeding-up-compiled-apps-startup
[8] Quirks with parfor vs. for : https://undocumentedmatlab.com/articles/quirks-with-parfor-vs-for
[9] Docking figures in compiled applications : https://undocumentedmatlab.com/articles/docking-figures-in-compiled-applications
[10] Explicit multi-threading in Matlab part 4 : https://undocumentedmatlab.com/articles/explicit-multi-threading-in-matlab-part4
[11] Explicit multi-threading in Matlab part 2 : https://undocumentedmatlab.com/articles/explicit-multi-threading-in-matlab-part2
Click here to print.
Copyright © Yair Altman - Undocumented Matlab. All rights reserved.