Undocumented Matlab
  • SERVICES
    • Consulting
    • Development
    • Training
    • Gallery
    • Testimonials
  • PRODUCTS
    • IQML: IQFeed-Matlab connector
    • IB-Matlab: InteractiveBrokers-Matlab connector
    • EODML: EODHistoricalData-Matlab connector
    • Webinars
  • BOOKS
    • Secrets of MATLAB-Java Programming
    • Accelerating MATLAB Performance
    • MATLAB Succinctly
  • ARTICLES
  • ABOUT
    • Policies
  • CONTACT
  • SERVICES
    • Consulting
    • Development
    • Training
    • Gallery
    • Testimonials
  • PRODUCTS
    • IQML: IQFeed-Matlab connector
    • IB-Matlab: InteractiveBrokers-Matlab connector
    • EODML: EODHistoricalData-Matlab connector
    • Webinars
  • BOOKS
    • Secrets of MATLAB-Java Programming
    • Accelerating MATLAB Performance
    • MATLAB Succinctly
  • ARTICLES
  • ABOUT
    • Policies
  • CONTACT

Using Infiniband with Matlab Parallel Computing Toolbox

March 26, 2015 2 Comments

I would like to welcome guest blogger Brock Palen, who is the Associate Director for Advanced Research Computing at the University of Michigan. Brock worked in High Performance computing since 2004 and is also one half of the Research Computing podcast; Research Computing and Engineering. You can find him blogging at failureasaservice.com. This is an updated repost of Brock’s article on UMich’s Flux HPC blog. Additional information on the Parallel Computing Toolbox can be found in my book Accelerating MATLAB Performance.
In High Performance Computing (HPC) there are a number of network types commonly used, among these are: Ethernet, the common network found on all computer equipment. Infiniband, a specialty high performance low latency interconnect common on commodity clusters. There are also several propriety types and a few other less common types but I will focus on Ethernet and Infiniband.
Ethernet and really its mate protocol, TCP, are the most common supported MPI networks. Almost all computer platforms support this network type and can be as simple as using your home network switch. It is ubiquitous and easy to support. Networks like Infiniband though require special drivers, uncommon hardware but the effort is normally worth it.
The MATLAB Parallel Computing Toolbox provides a collection of functions that allow MATLAB users to utilize multiple compute nodes to work on larger problems. Many may not realize that MathWorks chose to use the standard MPI routines to implement this toolbox. MathWorks also chose, for ease of use, to ship MATLAB with the MPICH2 MPI library, and the version they use only supports Ethernet for communication between nodes.
Unfortunately, Ethernet is about the slowest common network used in parallel applications. The question is how much can this impact performance.
Mmmmm Data:


The data was generated on 12 nodes of Xeon x5650 total 144 cores. The code was the stock MATLAB paralleldemo_backslash_bench(1.25) from MATLAB 2013b. You can find my M-code at Gist.
The data shows two trends: the first is that independently of the network type, many parallel algorithms do not scale unless the amount of data for each core to work on is sufficiently large. In this case, for Ethernet especially, the peak performance is never reached. What should be really noted though is that without Infiniband, at many problem sizes, over half of the performance of the nodes is lost. The second trend is that the network type really matters.

How to have MATLAB use Infiniband?

MathWorks does not ship an MPI library with the parallel computing toolbox that can use infiniband by default. This is reasonable, I would be curious how large the average PCT cluster is, and/or how big the jobs ran on the toolbox are. Lucky for us MathWorks allows a way for introducing your own MPI library. Let me be the first to proclaim:

Thank you MathWorks for adding mpiLibConf.m as a feature.
— Brock Palen

A simple example (note that MATLAB’s local scheduler also uses mpiLibConf, so we need to check for this case):

function [lib, extras] = mpiLibConf
%MATLAB MPI Library overloading for Infiniband and Ethernet Networks
%
%USAGE
%   place in ~/matlab/mpiLibConf.m
%   Update to point to your MPICH / Intel MPI etc location
% Check first if we're running the local scheduler - if we are, then get the default and exit
dfcn = getenv('MDCE_DECODE_FUNCTION');
if strcmp(dfcn, 'parallel.internal.decode.localMpiexecTask')
    % Get the local scheduler's default libs
    [lib, extras] = distcomp.mpiLibConfs('default');
else
    % We're not running the local scheduler or using the default MATLAB libmpich
    lib = '/home/software/rhel6/mvapich2/1.8/lib/libmpich.so';
    % mvapich has two extra libraries libmpl.so and libopa.so
    %  use # ldd /home/software/rhel6/mvapich2/1.8/lib/libmpich.so
    %   Any libraries from the mpich/mvapich install location need to be included in extras
    extras = {'/home/software/rhel6/mvapich2/1.8/lib/libmpl.so',
              '/home/software/rhel6/mvapich2/1.8/lib/libopa.so'};
end

function [lib, extras] = mpiLibConf %MATLAB MPI Library overloading for Infiniband and Ethernet Networks % %USAGE % place in ~/matlab/mpiLibConf.m % Update to point to your MPICH / Intel MPI etc location % Check first if we're running the local scheduler - if we are, then get the default and exit dfcn = getenv('MDCE_DECODE_FUNCTION'); if strcmp(dfcn, 'parallel.internal.decode.localMpiexecTask') % Get the local scheduler's default libs [lib, extras] = distcomp.mpiLibConfs('default'); else % We're not running the local scheduler or using the default MATLAB libmpich lib = '/home/software/rhel6/mvapich2/1.8/lib/libmpich.so'; % mvapich has two extra libraries libmpl.so and libopa.so % use # ldd /home/software/rhel6/mvapich2/1.8/lib/libmpich.so % Any libraries from the mpich/mvapich install location need to be included in extras extras = {'/home/software/rhel6/mvapich2/1.8/lib/libmpl.so', '/home/software/rhel6/mvapich2/1.8/lib/libopa.so'}; end

In the above test we used Intel MPI for the infiniband test and mpich for the ethernet test. The choice of MPI is important. The MPI standard enforces a shared API (Application programming interface), but not a shared ABI (Application binary interface). Thus the MPI library you substitute needs to match the one MATLAB is compiled against. Lucky for us they used mpich, so any mpich clone should work; MVAPICH, IntelMPI, etc.
If you are using the MATLAB Parallel Computing Toolbox on more than one node, and if your cluster has a network other than Ethernet/TCP (there are non-TCP Ethernet networks that perform very well) I highly encourage that the effort be put in to ensure you use that network.
For Flux users we have this setup, but you have to do some setup for yourself before you see the benefit. Please visit the ARC MATLAB documentation, or send us a question at hpc-support@umich.edu.

Related posts:

  1. Explicit multi-threading in Matlab part 2 – Matlab performance can be improved by employing .Net (C#, VB, F# or C++) threads. ...
  2. Fixing Matlab's actxserver – Matlab's COM (ActiveX) server behavior can be fixed in a couple of useful manners. ...
  3. Explicit multi-threading in Matlab part 4 – Matlab performance can be improved by employing timer objects and spawning external processes. ...
  4. Graphic sizing in Matlab R2015b – Matlab release R2015b's new "DPI-aware" nature broke some important functionality. Here's what can be done... ...
  5. Quirks with compiled Matlab DLLs – Several quirks with Matlab-compiled DLLs are discussed and workarounds suggested. ...
  6. Explicit multi-threading in Matlab part 3 – Matlab performance can be improved by employing POSIX threads in C/C++ code. ...
Brock Palen Performance Pure Matlab Undocumented feature
Print Print
« Previous
Next »
2 Responses
  1. Obliczone March 30, 2015 at 01:29 Reply

    Thank you for this article, especially for source code. Very useful.

  2. Chris Marshall April 24, 2015 at 07:15 Reply

    Thanks for the article. I’ve modified my mpiLibConf file to use a local mvpapich2 lib (2.1) that I’ve compiled using gcc. Trouble is matlab stops with a signal 11. Is there a way of getting this to work with gcc or do I need to use the Intel compiler?

    Thanks

Leave a Reply
HTML tags such as <b> or <i> are accepted.
Wrap code fragments inside <pre lang="matlab"> tags, like this:
<pre lang="matlab">
a = magic(3);
disp(sum(a))
</pre>
I reserve the right to edit/delete comments (read the site policies).
Not all comments will be answered. You can always email me (altmany at gmail) for private consulting.

Click here to cancel reply.

Useful links
  •  Email Yair Altman
  •  Subscribe to new posts (feed)
  •  Subscribe to new posts (reader)
  •  Subscribe to comments (feed)
 
Accelerating MATLAB Performance book
Recent Posts

Speeding-up builtin Matlab functions – part 3

Improving graphics interactivity

Interesting Matlab puzzle – analysis

Interesting Matlab puzzle

Undocumented plot marker types

Matlab toolstrip – part 9 (popup figures)

Matlab toolstrip – part 8 (galleries)

Matlab toolstrip – part 7 (selection controls)

Matlab toolstrip – part 6 (complex controls)

Matlab toolstrip – part 5 (icons)

Matlab toolstrip – part 4 (control customization)

Reverting axes controls in figure toolbar

Matlab toolstrip – part 3 (basic customization)

Matlab toolstrip – part 2 (ToolGroup App)

Matlab toolstrip – part 1

Categories
  • Desktop (45)
  • Figure window (59)
  • Guest bloggers (65)
  • GUI (165)
  • Handle graphics (84)
  • Hidden property (42)
  • Icons (15)
  • Java (174)
  • Listeners (22)
  • Memory (16)
  • Mex (13)
  • Presumed future risk (394)
    • High risk of breaking in future versions (100)
    • Low risk of breaking in future versions (160)
    • Medium risk of breaking in future versions (136)
  • Public presentation (6)
  • Semi-documented feature (10)
  • Semi-documented function (35)
  • Stock Matlab function (140)
  • Toolbox (10)
  • UI controls (52)
  • Uncategorized (13)
  • Undocumented feature (217)
  • Undocumented function (37)
Tags
ActiveX (6) AppDesigner (9) Callbacks (31) Compiler (10) Desktop (38) Donn Shull (10) Editor (8) Figure (19) FindJObj (27) GUI (141) GUIDE (8) Handle graphics (78) HG2 (34) Hidden property (51) HTML (26) Icons (9) Internal component (39) Java (178) JavaFrame (20) JIDE (19) JMI (8) Listener (17) Malcolm Lidierth (8) MCOS (11) Memory (13) Menubar (9) Mex (14) Optical illusion (11) Performance (78) Profiler (9) Pure Matlab (187) schema (7) schema.class (8) schema.prop (18) Semi-documented feature (6) Semi-documented function (33) Toolbar (14) Toolstrip (13) uicontrol (37) uifigure (8) UIInspect (12) uitools (20) Undocumented feature (187) Undocumented function (37) Undocumented property (20)
Recent Comments
  • Nicholas (3 days 18 hours ago): Hi Yair, Thanks for the reply. I am on Windows 10. I also forgot to mention that this all works wonderfully out of the editor. It only fails once compiled. So, yes, I have tried a...
  • Nicholas (3 days 18 hours ago): Hi Yair, Thanks for the reply. I am on Windows 10. I also forgot to mention that this all works wonderfully out of the editor. It only fails once compiled. So, yes, I have tried a...
  • Yair Altman (4 days 2 hours ago): Nicholas – yes, I used it in a compiled Windows app using R2022b (no update). You didn’t specify the Matlab code location that threw the error so I can’t help...
  • Nicholas (4 days 22 hours ago): Hi Yair, Have you attempted your displayWebPage utility (or the LightweightHelpPanel in general) within a compiled application? It appears to fail in apps derived from both R2022b...
  • João Neves (8 days 3 hours ago): I am on matlab 2021a, this still works: url = struct(struct(struct(struct(hF ig).Controller).PlatformHost). CEF).URL; but the html document is empty. Is there still a way to do...
  • Yair Altman (11 days 1 hour ago): Perhaps the class() function could assist you. Or maybe just wrap different access methods in a try-catch so that if one method fails you could access the data using another...
  • Jeroen Boschma (11 days 4 hours ago): Never mind, the new UI components have an HTML panel available. Works for me…
  • Alexandre (11 days 5 hours ago): Hi, Is there a way to test if data dictionnatry entry are signal, simulink parameters, variables … I need to access their value, but the access method depends on the data...
  • Nicholas (11 days 19 hours ago): In case anyone is looking for more info on the toolbar: I ran into some problems creating a toolbar with the lightweight panel. Previously, the Browser Panel had an addToolbar...
  • Jeroen Boschma (15 days 2 hours ago): I do not seem to get the scrollbars (horizontal…) working in Matlab 2020b. Snippets of init-code (all based on Yair’s snippets on this site) handles.text_explorer...
  • Yair Altman (43 days 4 hours ago): m_map is a mapping tool, not even created by MathWorks and not part of the basic Matlab system. I have no idea why you think that the customizations to the builtin bar function...
  • chengji chen (43 days 11 hours ago): Hi, I have tried the method, but it didn’t work. I plot figure by m_map toolbox, the xticklabel will add to the yticklabel at the left-down corner, so I want to move down...
  • Yair Altman (51 days 4 hours ago): @Alexander – this is correct. Matlab stopped including sqlite4java in R2021b (it was still included in 21a). You can download the open-source sqlite4java project from...
  • Alexander Eder (56 days 23 hours ago): Unfortunately Matlab stopped shipping sqlite4java starting with R2021(b?)
  • K (63 days 10 hours ago): Is there a way to programmatically manage which figure gets placed where? Let’s say I have 5 figures docked, and I split it into 2 x 1, I want to place 3 specific figures on the...
Contact us
Captcha image for Custom Contact Forms plugin. You must type the numbers shown in the image
Undocumented Matlab © 2009 - Yair Altman
This website and Octahedron Ltd. are not affiliated with The MathWorks Inc.; MATLAB® is a registered trademark of The MathWorks Inc.
Scroll to top