Undocumented Matlab
  • SERVICES
    • Consulting
    • Development
    • Training
    • Gallery
    • Testimonials
  • PRODUCTS
    • IQML: IQFeed-Matlab connector
    • IB-Matlab: InteractiveBrokers-Matlab connector
    • EODML: EODHistoricalData-Matlab connector
    • Webinars
  • BOOKS
    • Secrets of MATLAB-Java Programming
    • Accelerating MATLAB Performance
    • MATLAB Succinctly
  • ARTICLES
  • ABOUT
    • Policies
  • CONTACT
  • SERVICES
    • Consulting
    • Development
    • Training
    • Gallery
    • Testimonials
  • PRODUCTS
    • IQML: IQFeed-Matlab connector
    • IB-Matlab: InteractiveBrokers-Matlab connector
    • EODML: EODHistoricalData-Matlab connector
    • Webinars
  • BOOKS
    • Secrets of MATLAB-Java Programming
    • Accelerating MATLAB Performance
    • MATLAB Succinctly
  • ARTICLES
  • ABOUT
    • Policies
  • CONTACT

Using Infiniband with Matlab Parallel Computing Toolbox

March 26, 2015 2 Comments

I would like to welcome guest blogger Brock Palen, who is the Associate Director for Advanced Research Computing at the University of Michigan. Brock worked in High Performance computing since 2004 and is also one half of the Research Computing podcast; Research Computing and Engineering. You can find him blogging at failureasaservice.com. This is an updated repost of Brock’s article on UMich’s Flux HPC blog. Additional information on the Parallel Computing Toolbox can be found in my book Accelerating MATLAB Performance.
In High Performance Computing (HPC) there are a number of network types commonly used, among these are: Ethernet, the common network found on all computer equipment. Infiniband, a specialty high performance low latency interconnect common on commodity clusters. There are also several propriety types and a few other less common types but I will focus on Ethernet and Infiniband.
Ethernet and really its mate protocol, TCP, are the most common supported MPI networks. Almost all computer platforms support this network type and can be as simple as using your home network switch. It is ubiquitous and easy to support. Networks like Infiniband though require special drivers, uncommon hardware but the effort is normally worth it.
The MATLAB Parallel Computing Toolbox provides a collection of functions that allow MATLAB users to utilize multiple compute nodes to work on larger problems. Many may not realize that MathWorks chose to use the standard MPI routines to implement this toolbox. MathWorks also chose, for ease of use, to ship MATLAB with the MPICH2 MPI library, and the version they use only supports Ethernet for communication between nodes.
Unfortunately, Ethernet is about the slowest common network used in parallel applications. The question is how much can this impact performance.
Mmmmm Data:


The data was generated on 12 nodes of Xeon x5650 total 144 cores. The code was the stock MATLAB paralleldemo_backslash_bench(1.25) from MATLAB 2013b. You can find my M-code at Gist.
The data shows two trends: the first is that independently of the network type, many parallel algorithms do not scale unless the amount of data for each core to work on is sufficiently large. In this case, for Ethernet especially, the peak performance is never reached. What should be really noted though is that without Infiniband, at many problem sizes, over half of the performance of the nodes is lost. The second trend is that the network type really matters.

How to have MATLAB use Infiniband?

MathWorks does not ship an MPI library with the parallel computing toolbox that can use infiniband by default. This is reasonable, I would be curious how large the average PCT cluster is, and/or how big the jobs ran on the toolbox are. Lucky for us MathWorks allows a way for introducing your own MPI library. Let me be the first to proclaim:

Thank you MathWorks for adding mpiLibConf.m as a feature.
— Brock Palen

A simple example (note that MATLAB’s local scheduler also uses mpiLibConf, so we need to check for this case):

function [lib, extras] = mpiLibConf
%MATLAB MPI Library overloading for Infiniband and Ethernet Networks
%
%USAGE
%   place in ~/matlab/mpiLibConf.m
%   Update to point to your MPICH / Intel MPI etc location
% Check first if we're running the local scheduler - if we are, then get the default and exit
dfcn = getenv('MDCE_DECODE_FUNCTION');
if strcmp(dfcn, 'parallel.internal.decode.localMpiexecTask')
    % Get the local scheduler's default libs
    [lib, extras] = distcomp.mpiLibConfs('default');
else
    % We're not running the local scheduler or using the default MATLAB libmpich
    lib = '/home/software/rhel6/mvapich2/1.8/lib/libmpich.so';
    % mvapich has two extra libraries libmpl.so and libopa.so
    %  use # ldd /home/software/rhel6/mvapich2/1.8/lib/libmpich.so
    %   Any libraries from the mpich/mvapich install location need to be included in extras
    extras = {'/home/software/rhel6/mvapich2/1.8/lib/libmpl.so',
              '/home/software/rhel6/mvapich2/1.8/lib/libopa.so'};
end

function [lib, extras] = mpiLibConf %MATLAB MPI Library overloading for Infiniband and Ethernet Networks % %USAGE % place in ~/matlab/mpiLibConf.m % Update to point to your MPICH / Intel MPI etc location % Check first if we're running the local scheduler - if we are, then get the default and exit dfcn = getenv('MDCE_DECODE_FUNCTION'); if strcmp(dfcn, 'parallel.internal.decode.localMpiexecTask') % Get the local scheduler's default libs [lib, extras] = distcomp.mpiLibConfs('default'); else % We're not running the local scheduler or using the default MATLAB libmpich lib = '/home/software/rhel6/mvapich2/1.8/lib/libmpich.so'; % mvapich has two extra libraries libmpl.so and libopa.so % use # ldd /home/software/rhel6/mvapich2/1.8/lib/libmpich.so % Any libraries from the mpich/mvapich install location need to be included in extras extras = {'/home/software/rhel6/mvapich2/1.8/lib/libmpl.so', '/home/software/rhel6/mvapich2/1.8/lib/libopa.so'}; end

In the above test we used Intel MPI for the infiniband test and mpich for the ethernet test. The choice of MPI is important. The MPI standard enforces a shared API (Application programming interface), but not a shared ABI (Application binary interface). Thus the MPI library you substitute needs to match the one MATLAB is compiled against. Lucky for us they used mpich, so any mpich clone should work; MVAPICH, IntelMPI, etc.
If you are using the MATLAB Parallel Computing Toolbox on more than one node, and if your cluster has a network other than Ethernet/TCP (there are non-TCP Ethernet networks that perform very well) I highly encourage that the effort be put in to ensure you use that network.
For Flux users we have this setup, but you have to do some setup for yourself before you see the benefit. Please visit the ARC MATLAB documentation, or send us a question at hpc-support@umich.edu.

Related posts:

  1. Explicit multi-threading in Matlab part 2 – Matlab performance can be improved by employing .Net (C#, VB, F# or C++) threads. ...
  2. Fixing Matlab's actxserver – Matlab's COM (ActiveX) server behavior can be fixed in a couple of useful manners. ...
  3. Explicit multi-threading in Matlab part 4 – Matlab performance can be improved by employing timer objects and spawning external processes. ...
  4. Graphic sizing in Matlab R2015b – Matlab release R2015b's new "DPI-aware" nature broke some important functionality. Here's what can be done... ...
  5. Quirks with compiled Matlab DLLs – Several quirks with Matlab-compiled DLLs are discussed and workarounds suggested. ...
  6. Explicit multi-threading in Matlab part 3 – Matlab performance can be improved by employing POSIX threads in C/C++ code. ...
Brock Palen Performance Pure Matlab Undocumented feature
Print Print
« Previous
Next »
2 Responses
  1. Obliczone March 30, 2015 at 01:29 Reply

    Thank you for this article, especially for source code. Very useful.

  2. Chris Marshall April 24, 2015 at 07:15 Reply

    Thanks for the article. I’ve modified my mpiLibConf file to use a local mvpapich2 lib (2.1) that I’ve compiled using gcc. Trouble is matlab stops with a signal 11. Is there a way of getting this to work with gcc or do I need to use the Intel compiler?

    Thanks

Leave a Reply
HTML tags such as <b> or <i> are accepted.
Wrap code fragments inside <pre lang="matlab"> tags, like this:
<pre lang="matlab">
a = magic(3);
disp(sum(a))
</pre>
I reserve the right to edit/delete comments (read the site policies).
Not all comments will be answered. You can always email me (altmany at gmail) for private consulting.

Click here to cancel reply.

Useful links
  •  Email Yair Altman
  •  Subscribe to new posts (email)
  •  Subscribe to new posts (feed)
  •  Subscribe to new posts (reader)
  •  Subscribe to comments (feed)
 
Accelerating MATLAB Performance book
Recent Posts

Speeding-up builtin Matlab functions – part 3

Improving graphics interactivity

Interesting Matlab puzzle – analysis

Interesting Matlab puzzle

Undocumented plot marker types

Matlab toolstrip – part 9 (popup figures)

Matlab toolstrip – part 8 (galleries)

Matlab toolstrip – part 7 (selection controls)

Matlab toolstrip – part 6 (complex controls)

Matlab toolstrip – part 5 (icons)

Matlab toolstrip – part 4 (control customization)

Reverting axes controls in figure toolbar

Matlab toolstrip – part 3 (basic customization)

Matlab toolstrip – part 2 (ToolGroup App)

Matlab toolstrip – part 1

Categories
  • Desktop (45)
  • Figure window (59)
  • Guest bloggers (65)
  • GUI (165)
  • Handle graphics (84)
  • Hidden property (42)
  • Icons (15)
  • Java (174)
  • Listeners (22)
  • Memory (16)
  • Mex (13)
  • Presumed future risk (394)
    • High risk of breaking in future versions (100)
    • Low risk of breaking in future versions (160)
    • Medium risk of breaking in future versions (136)
  • Public presentation (6)
  • Semi-documented feature (10)
  • Semi-documented function (35)
  • Stock Matlab function (140)
  • Toolbox (10)
  • UI controls (52)
  • Uncategorized (13)
  • Undocumented feature (217)
  • Undocumented function (37)
Tags
ActiveX (6) AppDesigner (9) Callbacks (31) Compiler (10) Desktop (38) Donn Shull (10) Editor (8) Figure (19) FindJObj (27) GUI (141) GUIDE (8) Handle graphics (78) HG2 (34) Hidden property (51) HTML (26) Icons (9) Internal component (39) Java (178) JavaFrame (20) JIDE (19) JMI (8) Listener (17) Malcolm Lidierth (8) MCOS (11) Memory (13) Menubar (9) Mex (14) Optical illusion (11) Performance (78) Profiler (9) Pure Matlab (187) schema (7) schema.class (8) schema.prop (18) Semi-documented feature (6) Semi-documented function (33) Toolbar (14) Toolstrip (13) uicontrol (37) uifigure (8) UIInspect (12) uitools (20) Undocumented feature (187) Undocumented function (37) Undocumented property (20)
Recent Comments
  • Josh (10 hours 19 minutes ago): Dear Yair, Small typo; you wrote >>Move lines up or down – CTRL + ALT + UP or DOWN allows you to move selected lines up or down but it’s actually ALT+SHIFT then UP/DOWN...
  • Yair Altman (7 days 4 hours ago): You can try to increase the Java heap memory size in the Matlab preferences (General => Java Heap Memory). Any changes to the settings will only take effect after restarting...
  • Thomas (7 days 6 hours ago): Hello, thanks for sharing! I currently receive the following memory error message when using savezip with a big object (Matlab R2020b, Windows 10). Error using...
  • Yair Altman (10 days 13 hours ago): Yerlan – either use menuItem1.setEnabled(0) or set(menuItem1,'Enabled',0)
  • Manzn (10 days 16 hours ago): Thank you man! you saved me, when there was no more light 😀
  • Yerlan (11 days 21 hours ago): Hello Mr. Altman, how can I disable menu items in the context menu? E.g. when I am trying to do this: menuItems = jmenu.getSubElements; menuItem1 = menuItems(1);...
  • Yair Altman (12 days 13 hours ago): Thanks for the note Eric – you forgot the crucial call to jTable.setLabelTable(labelTabl e) – I added it into your code snippet above.
  • Erik (14 days 0 hours ago): Thank you for this — I don’t know if this has been mentioned anywhere else before, but it could be useful to mention how to add custom labels to a jslider. I did the...
  • turhv (26 days 19 hours ago): very nice! work perfectly to me in MATLAB 2019a. thanks!!!
  • Jianfei (59 days 22 hours ago): I have tried the MathWorks CheckBoxList in Matlab 2015b. For whatever the reason, I can’t change the font properties. I can change the font color, but changing font properties...
  • Donato Coppola (65 days 18 hours ago): Hi Yair, I have a problem with the double format. When I run the treeTable function, the numbers in double format cells are displayed with comma as decimal separator. How can...
  • Kim (71 days 8 hours ago): Yair, the following didn’t work for me either: jh.setBorderPainted(false); Regards, Kim
  • Adil (73 days 10 hours ago): Thank you for the blog, it was useful for me. I have a file named App_project.mlapp.zip and when I extract it through Winzip it gives all the files exactly as you described above. I...
  • Mr Ashley Trowell (75 days 19 hours ago): Thank you so much for this analysis. Also, I find it somewhat horrifying. The take away seems to be to use && / || and NOT and / or. Thanks a bunch! ~Ashley
  • Matt (80 days 21 hours ago): Late reply I know, but you can call custom shortcuts with alt-#. Hold down Alt to see what number is assigned to the shortcuts you’ve created. Now if there was a way to create a...
Contact us
Undocumented Matlab © 2009 - Yair Altman
Scroll to top