We can add a Matlab toolstrip to 3 types of Matlab GUI windows:

1. To a Java-based Matlab figure (so-called “legacy” figures, created using GUIDE or the figure function)
2. To a container window of docked Java-based figures, typically called an “App” (marketing name) or “Tool Group” (internal technical name)
3. To a JavaScript/HTML-based Matlab figure (so called “web” figures, created using App Designer or the uifigure function)

Today I will show how to add a basic dynamic toolstrip to a ToolGroup (App, window type #2):

ToolGroup with clients and dynamic toolstrip

Figure containers (“Tool Groups”)

Most Matlab users are familiar with window types #1 and #3 (legacy and web-based figures), but type #2 may seem strange. In fact, it shouldn’t be: All the Matlab “Apps” and Desktop components use such a container of docked clients. For example, both the Matlab Editor and Desktop are containers of individual client windows (individual files in the Editor; Command Window, Workspace etc. in the desktop).

Similarly, when we dock figures, they dock as client windows into a container called “Figures” (this can be controlled programmatically: see my setFigDockGroup utility on the File Exchange). This is the basis for all Matlab “Apps”, as far as I am aware (some Apps may possibly use a different GUI container, after all there are ~100 Matlab Apps and I’m not familiar with all of them). Such Apps are basically stand-alone Tool Groups (client container windows) that contain one or more docked figures, a toolstrip, and a side-panel with controls (so-called “Data Browser”).

Note: MathWorks uses confusing terminology here, using the same term “App” for both MathWorks-created GUIs containers (that have toolstrips, Data Browser and docked figures) and also user-created utilities on the File Exchange (that do not have these). Unfortunately, MathWorks has chosen not [yet] to release to the general public its set of tools that enable creating true “Apps”, i.e. those that have a toolstrip, Data Browser and docked figures.

Today’s post will attempt to fill this gap, by showing how we can create user Apps that have a toolstrip and docked figures. I will ignore the Data Browser today, and will describe it in a future post. Since docking figures into a standalone user-created container is a solved problem (using my setFigDockGroup utility), this post will focus on adding a toolstrip to such a container.

A ToolGroup object (matlab.ui.internal.desktop.ToolGroup) is created either implicitly (by docking a figure into a group that has a new name), or explicitly (by invoking its constructor):

% Create a new non-visible empty App (Tool Group)
hToolGroup = matlab.ui.internal.desktop.ToolGroup('Toolstrip example on UndocumentedMatlab.com');

Some things only work properly after the app is displayed, so let’s display the ToolGroup (however, note that for improved performance it is better to do whatever customizations and GUI updates that you can before the app is made visible):

% Display the ToolGroup window
hToolGroup.open();

Basic empty ToolGroup (without toolstrip or clients)

An annoying quirk with ToolGroups is that they automatically close when their reference handle is deleted from Matlab memory. The specific behavior changes depending on the contents of the container and the Matlab release, but in general it’s safest to preserve the hToolGroup variable, to prevent the window from closing, when this variable goes out of scope, when the function (in which we create the ToolGroup) returns. There are many ways to persist this variable. Here’s one alternative, in which we persist it in itself (or rather, attached to its internal Java peer control):

% Store toolgroup reference handle so that app will stay in memory
jToolGroup = hToolGroup.Peer;
internal.setJavaCustomData(jToolGroup, hToolGroup);

internal.setJavaCustomData is an undocumented Matlab function that adds a new custom property to a Java reference handle. In our case, it adds a CustomData property to the jToolGroup handle and sets its value to the Matlab hToolGroup handle. The source code for internal.setJavaCustomData is available in %matlabroot%/toolbox/shared/controllib/general/+internal/setJavaCustomData.m and is very simple: it essentially uses the old schema-based schema.prop method for adding new properties to handles. Schema is an old deprecated mechanism that is mostly replaced by the newer MCOS (Matlab Class Object System), but for some specific cases such as this it’s still very useful (the standard addprop function can add new properties to Matlab GUI handles, but not to Java reference handles).

Finally, let’s discard the Data Browser side panel (I’ll discuss it in a separate future post):

% Discard the Data-browser left panel
hToolGroup.disableDataBrowser();

Adding a toolstrip to the ToolGroup

Now that we have the basic container ready, let’s add a toolstrip. To simplify matters in this introductory post (after all, I have still not described the internal packages and classes that make up a toolstrip), we’ll use some of the tabs used in the showcaseToolGroup example that I discussed in my previous post. You can see the relevant source code in %matlabroot%/toolbox/matlab/toolstrip/+matlab/+ui/+internal/+desktop/*.m, in case you want to jump ahead and customize your own toolstrip tabs, groups and buttons. In the code snippet below, we first create an empty TabGroup, then add toolstrip tabs into it, and finally add this TabGroup into our ToolGroup using its addTabGroup(hTabGroup) method:

% Create a new tab group
%hTabGroup = matlab.ui.internal.desktop.showcaseBuildTabGroup('swing');
hTabGroup = matlab.ui.internal.toolstrip.TabGroup();
hTab1 = matlab.ui.internal.desktop.showcaseBuildTab_Buttons('swing');
hTabGroup.add(hTab1);
%hTabGroup.add(matlab.ui.internal.desktop.showcaseBuildTab_Gallery());
hTabGroup.add(matlab.ui.internal.desktop.showcaseBuildTab_Layout('swing'));
 
% Select tab #1 (common)
hTabGroup.SelectedTab = hTab1;
 
% Add the tab group to the built-in toolstrip
hToolGroup.addTabGroup(hTabGroup);

We now have an “App” that has a toolstrip, but no clients (yet), and a hidden Data Browser side-panel:

Now let’s add some clients (docked figures):

Adding clients (docked figures) to the ToolGroup

There are two easy variants for adding docked figures in a ToolGroup: The easiest is to use the ToolGroup’s addFigure(hFigure) method:

% Create a figure and dock it into the tool-group
hFig1 = figure('Name','3D');
surf(peaks);
hToolGroup.addFigure(hFig1);

The second variant enables to dock a figure that has a specific set of toolstrip tabs attached to it. These tabs will only display in the toolstrip when that particular figure has focus. We do this by creating a new TabGroup (just as we have done above), and then add the figure and TabGroup to the container using the ToolGroup’s addClientTabGroup(hFigure,hTabGroup) method:

% Create the 2nd figure
hFig2 = figure('Name','2D');
plot(rand(5)); drawnow
 
% Add a few tabs that are only relevant to this specific figure
hTabGroup2 = matlab.ui.internal.toolstrip.TabGroup();
hTab2 = matlab.ui.internal.desktop.showcaseBuildTab_Selections();
hTabGroup2.add(hTab2);
hTabGroup2.add(matlab.ui.internal.desktop.showcaseBuildTab_EditValue());
 
% Add the figure and tabs to the ToolGroup
hToolGroup.addClientTabGroup(hFig2, hTabGroup2);

ToolGroup with clients and dynamic toolstrip

In this example, the “Selection” and “Values” toolstrip tabs only appear when the 2nd figure (“2D”) has focus. A similar behavior exists in the Matlab Desktop and Editor, where some tabs are only shown when certain clients have focus.

Removing the View tab

Note that the “View” toolstrip tab (which enables setting the appearance of the docked figures: layout, tab positions (top/bottom/left/right), ordering etc.) is automatically added to the toolstrip and always appears last. We can remove this View tab using the ToolGroup’s hideViewTab() method. The tab will not immediately be removed, only when the toolstrip is repainted, for example, when we switch focus between the docked figures:

hToolGroup.hideViewTab;  % toolstrip View tab is still visible at this point
figure(hFig1);  % change focus to hFig1 - toolstrip is repainted without View tab

Conclusion

It’s relatively easy to dock figures into a standalone “App” window that has a custom toolstrip, which can even be dynamically modified based on the figure which is currently in focus. Naturally, this has little benefit if we cannot customize the toolstrip components: labels, icons, control type, grouping and most importantly – callbacks. This topic deserves a dedicated post, which I plan to be the next in this miniseries. Stay tuned – hopefully the next post will not take me as long to publish as this post (I was quite busy recently)…

My plan is to start the miniseries with a discussion of the built-in showcase examples, followed by a post on the built-in classes that make up the toolstrip building-blocks. Finally, I’ll describe how toolstrips can be added to figures, not just in client/tool groups.

Matlab’s internal showcase examples

I start the discussion with a description of built-in examples for the toolstrip functionality, located in %matlabroot%/toolbox/matlab/toolstrip/+matlab/+ui/+internal/+desktop/. The most important of these are showcaseToolGroup.m and showcaseMPCDesigner.m, both of which use Java-based (Swing) containers and controls. Readers who wish to integrate toolstrips into their app immediately, without waiting for my followup posts in this series, are welcome to dig into the examples’ source-code and replicate it in their programs:

1. showcaseToolGroup

h = matlab.ui.internal.desktop.showcaseToolGroup

2. showcaseMPCDesigner

>> h = matlab.ui.internal.desktop.showcaseMPCDesigner
 
h =
  showcaseMPCDesigner with properties:
 
    ToolGroup: [1×1 matlab.ui.internal.desktop.ToolGroup]
       Dialog: [1×1 toolpack.component.TSTearOffPopup]
      Figure1: [1×1 Figure]
      Figure2: [1×1 Figure]

3. showcaseHTML and showcaseCEF

In addition to these showcase examples, the folder also contains a showcaseHTML.m and showcaseCEF.m files, that are supposed to showcase the toolstrip functionality in JavaScript-based containers (browser webpage and uifigure apps, respectively). Unfortunately, on my system running these classes displays blank, although the toolstrip is indeed created, as seen below (if you find out how to make these classes work, please let me know):

>> h = matlab.ui.internal.desktop.showcaseHTML
building toolstrip hierarchy...
rendering toolstrip...
 
h = 
  Toolstrip with properties:
 
               SelectedTab: [1×1 matlab.ui.internal.toolstrip.Tab]
              DisplayState: 'expanded'
    DisplayStateChangedFcn: @PropertyChangedCallback
                       Tag: 'toolstrip'
 
>> hs = struct(h)
Warning: Calling STRUCT on an object prevents the object from hiding its implementation details and should thus be avoided.
Use DISP or DISPLAY to see the visible public details of an object. See 'help struct' for more information.
(Type "warning off MATLAB:structOnObject" to suppress this warning.)
 
hs = 
  struct with fields:
 
                         SelectedTab: [1×1 matlab.ui.internal.toolstrip.Tab]
                        DisplayState: 'expanded'
              DisplayStateChangedFcn: @PropertyChangedCallback
                 DisplayStatePrivate: 'expanded'
                        QABIdPrivate: '2741bf89'
               QuickAccessBarPrivate: [1×1 matlab.ui.internal.toolstrip.impl.QuickAccessBar]
       DisplayStateChangedFcnPrivate: @PropertyChangedCallback
         SelectedTabChangedListeners: [1×1 event.listener]
                                 Tag: 'toolstrip'
                                Type: 'Toolstrip'
                          TagPrivate: 'toolstrip'
    WidgetPropertyMap_FromMCOSToPeer: [3×1 containers.Map]
    WidgetPropertyMap_FromPeerToMCOS: [3×1 containers.Map]
                              Parent: [0×0 matlab.ui.internal.toolstrip.base.Node]
                            Children: [1×1 matlab.ui.internal.toolstrip.TabGroup]
                             Parent_: []
                           Children_: [1×1 matlab.ui.internal.toolstrip.TabGroup]
                                Peer: [1×1 com.mathworks.peermodel.impl.PeerNodeImpl]
                   PropertySetSource: [1 java.util.HashMap]
                    PeerModelChannel: '/ToolstripShowcaseChannel'
                   PeerEventListener: [1×1 handle.listener]
                 PropertySetListener: [1×1 handle.listener]
 
>> hs.Peer
ans =
PeerNodeImpl{id='4a1e4b08', type='Toolstrip', properties={displayState=expanded, hostId=ToolStripShowcaseDIV, tag=toolstrip, QABId=2741bf89}, parent=878b0e2b, children=[
    PeerNodeImpl{id='5bb9632c', type='TabGroup', properties={QAGroupId=ea9b628c, tag=, selectedTab=f90db10c}, parent=4a1e4b08, children=[
        PeerNodeImpl{id='f90db10c', type='Tab', properties={mnemonic=, tag=tab_buttons, title=BUTTONS}, parent=5bb9632c, children=[
            PeerNodeImpl{id='1ccc9246', type='Section', properties={collapsePriority=0.0, mnemonic=, tag=sec_push, title=PUSH BUTTON}, parent=f90db10c, children=[
                PeerNodeImpl{id='8323f06e', type='Column', properties={horizontalAlignment=left, width=0.0, tag=}, parent=1ccc9246, children=[
                    PeerNodeImpl{id='af368d7b', type='PushButton', properties={textOverride=, descriptionOverride=, mnemonic=, actionId=230d471b, iconOverride=, tag=pushV, iconPathOverride=}, parent=8323f06e, children=[]}]}
                PeerNodeImpl{id='a557a712', type='Column', properties={horizontalAlignment=left, width=0.0, tag=}, parent=1ccc9246, children=[
                    PeerNodeImpl{id='f0d6a9fc', type='EmptyControl', properties={tag=}, parent=a557a712, children=[]}
                    PeerNodeImpl{id='74bc4cd2', type='PushButton', properties={textOverride=, descriptionOverride=, mnemonic=, actionId=12d6a26a, iconOverride=, tag=pushH, iconPathOverride=}, parent=a557a712, children=[]}
                    PeerNodeImpl{id='bcb5a9d0', type='EmptyControl', properties={tag=}, parent=a557a712, children=[]}]}]}
            PeerNodeImpl{id='0e515319', type='Section', properties={collapsePriority=0.0, mnemonic=, tag=sec_dropdown, title=DROP DOWN BUTTON}, parent=f90db10c, children=[
                PeerNodeImpl{id='80482225', type='Column', properties={horizontalAlignment=left, width=0.0, tag=}, parent=0e515319, children=[
                    PeerNodeImpl{id='469f469a', type='DropDownButton', properties={textOverride=, descriptionOverride=, mnemonic=, actionId=c6ca7335, iconOverride=, tag=dropdownV, iconPathOverride=}, parent=80482225, children=[]}]}
                ...

Note: showcaseCEF has been removed in 2018, but is available in older Matlab releases.

Levels of toolstrip encapsulation

Matlab currently has several levels of encapsulation for toolstrip components:

• Top-level m-file classes for showcasing the toolstrip functionality and creating toolstrips in Java-based containers and web-based apps – these are located in %matlabroot%/toolbox/matlab/toolstrip/+matlab/+ui/+internal/+desktop/
• Mid-level m-file classes that contain the toolstrip building blocks (tabs, sections, controls) – these are located in %matlabroot%/toolbox/matlab/toolstrip/+matlab/+ui/+internal/+toolstrip/
• Low-level Java classes that implement the underlying user-interface for Java-based UI – these are located in %matlabroot%/java/jar/toolstrip.jar. I discussed this briefly in a post few years ago.

The top- and mid-level m-file classes are provided with full source code that is quite well-documented internally (as m-file source-code comments). However, note that it is not officially documented or supported (i.e., semi-documented in this blog’s parlance).

The low-level Java classes on the other hand are compiled without available source code – we can inspect these classes (e.g., using uiinspect or checkClass), but we cannot see their original source-code. Luckily, the higher-level m-file classes provide us with plenty of hints and usage examples that we could use to tailor the appearance, functionality and integration of toolstrip components into our app.

Robyn Jackey’s Widgets Toolbox

Users who hesitate to mess around with the built-in toolstrip functionality may find interest in MathWorker Robyn Jackey’s Toolstrip look-alike, which is part of his open-source Widgets Toolbox on the Matlab File Exchange. Unlike other parts of Robyn’s toolbox, which use undocumented functionality, his Toolstrip class seems to use documented components (panels, uicontrols etc.), with just a small reliance on undocumented functionality (matlab.ui.* for example). This has a fair chance to continue working well into future releases, even if Matlab’s built-in toolstrip functionality changes:

Strong caution

Over the years, Matlab’s internal toolstrip interface has changed somewhat, but not dramatically. This could change at any time, since the toolstrip uses deeply undocumented functionality. What I will demonstrate over the next few posts might stop working in R2019a, or in R2025b – nobody really knows, perhaps not even MathWorks at this stage. Something that we do know for a fact is that Matlab is slowly transitioning away from Java-based user interfaces to web-based (HTML/JavaScript/CSS) interfaces, and this could have a drastic effect on the toolstrip functionality/API. It seems reasonable to assume that even if MathWorks would one day open up the toolstrip functionality, this would only be for the new web-based uifigure apps (not legacy Java-based figures), and might well have a different API than the one that I’ll discuss in this miniseries. Still, users could use the unofficial/undocumented information that I present here in their own Java figures today and quite possibly also in near-term upcoming releases.

Despite the many unknowns regarding future supportability/roadmap of the built-in toolstrip API, I believe that my readers are smart enough to decide for themselves whether they want to take the associated risks to improve their Matlab programs today, or wait until a documented API will possibly be provided sometime in the future. The choice is yours, as it always is when using undocumented tips from my blog.

With this warning stated, let’s start having fun with Matlab’s built-in toolstrip!

Note: I will be traveling to the USA in June, and to Spain in August. If you would like me to visit your location for onsite consulting or training, then please let me know.

Happy Easter/Passover!

Overview

Editor-plugin's cool logo

The default Keyboard shortcuts listed below for the features can be customized. Most variables can be customized as well (I will point out which variables are not [yet] customizable).

Most GUIs have a search field. Within this search field you can move the list or the tree up and down using arrow keys, or hit <escape> to return to editor. These search fields allow you to enter regular expressions to limit results shown in list or tree. Also, most GUIs are dockable.

The Editor-Plugin utility is open-source. It is available on GitHub and also mirrored on the Matlab File Exchange. A detailed setup guide is provided on the utility’s wiki section in GitHub.

If you discover any problem or have any suggestion for improvement, please visit the utility’s Issues section on GitHub, where all open/closed issues can be tracked and discussed.

A brief overview of some of the features is presented below. For a detailed explanation of these and other features (which are not listed below), please review the Features section of the utility’s wiki (you guessed it: on GitHub…).

Editing

• Delete / duplicate lines – CTRL + SHIFT + Y or D allows you to delete or duplicate current line.
  
• Move lines up or down – CTRL + ALT + UP or DOWN allows you to move selected lines up or down.
  
• Live (auto-replace) templates – Live Templates are editor commands you can design to insert predefined code. Here’s an example for the command %this% (delivered within the package). When you type a command into the editor the string will get replaced by the predefined text. This predefined text may include variables depending on what you want to achieve. %this% was designed to insert the fully qualified name of the current class you’re in (or function, or script).
  
• Clipboard Stack – CTRL + SHIFT + V opens the Clipboard Stack, where the last 10 copied/cut text are stored and can be directly inserted into the current editor position. The Clipboard Stack only stores text copied from editor.

Navigation

• Auto switch current folder and detail viewer – Switching editor tabs will update the Desktop’s current folder and detail viewer. This behavior can be directly changed in the detail viewer bar.
  Use the new icons in the Desktop’s Current Folder panel: to toggle detail viewer and to toggle following the current editor file.
  
• File Structure – I personally use this one the most: CTRL + F12 will show a GUI that let you search methods and properties including inherited ones.
  
• Navigation History – Every editor file that is opened is stored in the navigation history. Up to 50 editor file paths are stored. If you have a 5-Button Mouse, you can navigate through previous location (backward and forward).
• Recently Closed – CTRL + SHIFT + T will show a GUI that allows you to reopen closed editors this or last session.
• Bookmarks – CTRL + SHIFT + F2 will show a GUI that allows you to name, search and delete bookmarks. Matlab has this nice feature to delete bookmarks after closing the editor. This feature will store all bookmarks. If an editor or Matlab is closed and opened later. All Bookmarks will be restored.
  This feature has some issues though. The most obvious first: if the source file has been changed outside of Matlab, the bookmark does not get updated, and may point to the wrong line, or gets deleted. Also on some Systems the default shortcut does not work. But there are workarounds.

Other useful features

• Dockable Windows – As mentioned before, most GUIs are dockable.
  
• Preferences – Preferences panel integrated in Matlab’s main Preferences window.
  
• Execute current line – SHIFT + F9 allows you to execute the current line. This is equivalent to selecting the entire line, then clicking F9, and is similar in concept to the editor’s built-in ability to execute the current cell-block.
  
• VarDiff – Select two variables in the workspace and compare them using Matlab’s internal comparison feature (adding a custom hook to the Workspace context-menu was discussed by Yair back in 2010).
  
• KeyPressListener – Allows you to create custom keyboard shortcuts for your own functions. A similar functionality was discussed by Yair back in 2009.

As noted above, a detailed explanation of these and other features is provided in the Features section of the utility’s wiki. If you discover any problem or have any suggestion for improvement, please visit the utility’s Issues section on GitHub, where issues can be tracked and discussed.

There are several ways this problem can be solved, ranging from creating a custom table display function, to modifying the table’s internal disp method (%matlabroot%/toolbox/matlab/datatypes/@tabular/disp.m), to using this method’s second optional argument (disp(myTable,false)). Note that simply subclassing the table class to overload disp() will not work because the table class is Sealed, but we could instead subclass table‘s superclass (tabular) just like table does.

Inside the disp.m method mentioned above, the headers markup is controlled (around line 45, depending on your Matlab release) by matlab.internal.display.isHot. Unfortunately, there is no corresponding setHot() method, nor corresponding m- or p-code that can be inspected. But the term “Hot” rang a bell, and then I remembered my old post about the HotLinks feature, which is apparently reflected by matlab.internal.display.isHot.

feature('HotLinks',false);  % temporarily disable bold headers and hyperlinks (matlab.internal.display.isHot=false)
disp(myTable)
myTable        % this calls disp() implicitly
feature('HotLinks',true);   % restore the standard behavior (markup displayed, matlab.internal.display.isHot=true)

Searching for “isHot” or “HotLinks” under the Matlab installation folder, we find that this feature is used in hundreds of places (the exact number depends on your installed toolboxes). The general use appears to be to disable/enable output of hyperlinks to the Matlab console, such as when you display a Matlab class, when its class name is hyperlinked and so is the “Show all properties” message at the bottom. But in certain cases, such as for the table output above, the feature is also used to determine other types of markup (bold headers in this case).

>> feature('HotLinks',0)  % temporarily disable bold headers and hyperlinks
>> groot
ans = 
  Graphics Root with properties:
 
          CurrentFigure: [0×0 GraphicsPlaceholder]
    ScreenPixelsPerInch: 96
             ScreenSize: [1 1 1366 768]
       MonitorPositions: [2×4 double]
                  Units: 'pixels'
 
  Use GET to show all properties

>> feature('HotLinks',1)  % restore the standard behavior (markup displayed)
>> groot
ans = 
  Graphics Root with properties:
 
          CurrentFigure: [0×0 GraphicsPlaceholder]
    ScreenPixelsPerInch: 96
             ScreenSize: [1 1 1366 768]
       MonitorPositions: [2×4 double]
                  Units: 'pixels'
 
  Show all properties

There’s nothing really earth shuttering in all this, but the HotLinks feature could be useful when outputting console output into a diary file. Of course, if diary would have automatically stripped away markup tags we would not need to resort to such hackery. Then again, this is not the only problem with diary, which is long-overdue an overhaul.

Using TabComplete for user-defined functions

Unfortunately, this mechanism apparently broke in R2016a and was replaced with a new mechanism, as explained below.

The new mechanism relies on a file called functionSignatures.json which exists in every single folder that contains Matlab files that have functions whose input parameters ought to be tab-completable.

The new mechanism offers far greater versatility and flexability in defining the input types and inter-relationsships compared to the old TC.*-based mechanism. Another important benefit is that we can now add custom user-defined functionSignatures.json files to our user folders, next to our m-files, without having to modify any Matlab system file.

Note that you may need to restart Matlab the first time that you create a functionSignatures.json file. But once it’s created, you can modify it within a Matlab session and the changes take effect immediately.

Note: Credit for first posting about this new mechanism goes to Nicholas Mati. I’ve known about this new mechanism for over a year, but I never found the time to write about it until now, so Nicholas gets credit for breaking the scoop. The discussion below uses and expands Nicholas’ original post.

Syntax

The functionSignatures.json file has the general form:

{
	"FunctionName1":
	{
		"key1":"val1",
		"key2":"val2",
		"keyn":"valn"
	},
	"FunctionName2":
	{
		"key1":"val1",
		"key2":"val2",
		"keyn":"valn"
	}
}

A number of keys are supported including “platform“, “setsAns“, “inputs“, and “outputs“, although inputs and outputs are by far the most common (and presumably only inputs are relevant for tab-completion). These keys take an array of (or a single) object value(s). The objects typically take one of the following forms:

{"name":"variable_name", "kind":"kind_option", "type":"string_or_array_of_type"}
{"mutuallyExclusiveGroup":
	[
		...
	]
}
{"name":"varargin", "kind":"optional", "multiplicity":"append"}

The value for “kind” can be “required”, “optional”, “positional”, “flag”, “namevalue” or “platform” (and perhaps a few other lesser-used kinds):

• “required” means that the specified input is mandatory
• “optional” means that it can be added or omitted
• “positional” means that it’s an optional input but if it is specified then it must appear at the specified position relative to the previous (earlier) inputs
• “flag” means that it’s an optional input flag, from a predefined list of one or more single-token strings. For example, in regexp(s1,s2,'once') the last input arg ('once') is such a flag.
• “namevalue” means that it follows Matlab’s standard practice of using P-V pairs (parameter name followed by its value). For example, func('propName',propValue)
• “platform” indicates that this input is only available on the specified platform(s)

These “kind”s are all explained below.

The value for “type” can be a string such as “char” or “numeric” or “filepath”, or a more complicated JSON array (see below).

In addition to “name”, “kind” and “type”, we can also define a “default” value (e.g. "default":"false") and a “display” string. While these are [currently] not used by Desktop tab-completion, they might be used by other components such as the JIT compiler or the Editor, if not today then perhaps in a future release.

Note that while pure JSON format does not accept comments, Matlab’s functionSignatures.json does accept C++-style comments, as discovered by Heiko in a comment below. To add a comment, simply add // comment text at the end of any line, or /* comment text */ anywhere within a line.

Usage examples

Multiple examples of functionSignatures.json files can be found in subfolders of %matlabroot%/toolbox/matlab. For example, here’s the tab-completion definition for the visdiff function, which displays a visual comparison between two files, and resides in %matlabroot%/toolbox/shared/comparisons/functionSignatures.json:

{
"visdiff":
{
    "inputs":
    [
        {"name":"filename1", "kind":"required",   "type":"filepath"},
        {"name":"filename2", "kind":"required",   "type":"filepath"},
        {"name":"type",      "kind":"positional", "type":"choices={'text', 'binary'}"}
    ]
}
}

As can be seen in this example, the first and second inputs are expected to be a filename, whereas the third input is one of the two predefined strings ‘text’ or ‘binary’. This third input has “kind”:”positional”, meaning that it is optional, but if it is provided then it must be in the 3rd position and cannot appear sooner. Moreover, if the user specifies any input argument to the “right” of a positional input, then the positional argument becomes required, not optional.

Whereas a “positional” parameter has a specific position in the args list (#3 in the case of visdiff above), an “optional” parameter may appear anywhere in the list of inputs.

Here’s a more complex example, for the built-in regexprep function (in %matlabroot%/toolbox/matlab/strfun/functionSignatures.json). This example shows how to limit the input to certain data types and how to specify optional input flags with pre-defined choices:

"regexprep":
{
	"inputs":
	[
		{"name":"str",               "kind":"required",  "type":[["char"], ["cell"], ["string"]]},
		{"name":"expression",        "kind":"required",  "type":[["char"], ["cell"], ["string"]]},
		{"name":"replace",           "kind":"required",  "type":[["char"], ["cell"], ["string"]]},
		{"name":"optMatch",          "kind":"flag",      "display":"", "type":[["char", "choices={'all','once'}"], ["numeric", "scalar"]],   "default":"'all'"},
		{"name":"optWarnings",       "kind":"flag",      "display":"", "type":["char", "choices={'nowarnings','warnings'}"],                 "default":"'nowarnings'"},
		{"name":"optCase",           "kind":"flag",      "display":"", "type":["char", "choices={'matchcase','ignorecase','preservecase'}"], "default":"'matchcase'"},
		{"name":"optEmptyMatch",     "kind":"flag",      "display":"", "type":["char", "choices={'noemptymatch','emptymatch'}"],             "default":"'noemptymatch'"},
		{"name":"optDotAll",         "kind":"flag",      "display":"", "type":["char", "choices={'dotall','dotexceptnewline'}"],             "default":"'dotall'"},
		{"name":"optStringAnchors",  "kind":"flag",      "display":"", "type":["char", "choices={'stringanchors','lineanchors'}"],           "default":"'stringanchors'"},
		{"name":"optSpacing",        "kind":"flag",      "display":"", "type":["char", "choices={'literalspacing','freespacing'}"],          "default":"'literalspacing'"}
	],
	"outputs":
	[
		{"name":"newStr", "type":[["char"], ["cell"], ["string"]]}
	]
},

Here’s an even more complex example, this time for the codegen function (in %matlabroot%/toolbox/coder/matlabcoder/functionSignatures.json, part of the Matlab Coder toolbox). This example shows how to limit the filenames to certain extensions and how to specify name-value input pairs:

"codegen":
{
	"inputs":
	[
		{"name":"compile_only",  "kind":"flag",       "type":"choices={'-c'}"},
		{"name":"config_flag",   "kind":"flag",       "type":"choices={'-config:mex','-config:lib','-config:dll','-config:exe','-config:hdl'}"},
		{"name":"debug",         "kind":"flag",       "type":"choices={'-g'}"},
		{"name":"report",        "kind":"flag",       "type":"choices={'-report'}"},
		{"name":"launchreport",  "kind":"flag",       "type":"choices={'-launchreport'}"},
		{"name":"file",          "kind":"flag",       "type":"filepath=*.m,*.mlx,*.c,*.cpp,*.h,*.o,*.obj,*.a,*.so,*.lib,*.tmf", "multiplicity":"append"},
		{"name":"-d",            "kind":"namevalue",  "type":"folderpath"},
		{"name":"-I",            "kind":"namevalue",  "type":"folderpath"},
		{"name":"-globals",      "kind":"namevalue"},
		{"name":"-o",            "kind":"namevalue",  "type":[["char"], ["filepath"]]},
		{"name":"-O",            "kind":"namevalue",  "type":"choices={'enable:inline','disable:inline','enable:blas','disable:blas','enable:openmp','disable:openmp'}"},
		{"name":"-args",         "kind":"namevalue",  "type":[["identifier=variable"], ["char"]]},
		{"name":"-config",       "kind":"namevalue",  "type":[["identifier=variable"], ["char"]]},
		{"name":"verbose",       "kind":"flag",       "type":"choices={'-v'}"},
		{"name":"singleC",       "kind":"flag",       "type":"choices={'-singleC'}"},
		{"name":"-test",         "kind":"namevalue",  "type":"identifier=function"}
	]
},

Argument types

As noted above, we use "type":... to specify the expected data type of each parameter. This can be a simple string such as “char”, “cellstr”, “numeric”, “table”, “categorical”, “filepath”, “folderpath”, “matlabpath”, class name, or a more complicated JSON array. For example:

• "type":["numeric","scalar"]
• "type":["numeric","numel=3",">=4"]
• "type":[["char"], ["cellstr"], ["numeric"], ["logical","vector"]]
• "type":[["char", "choices={'-ascii'}"]]
• "type":[["filepath"], ["matlabpath=*.m,*.mlx"], ["char"]]
• "type":"identifier=variable,function,localfunction,package,classdef"
• "type":"matlab.graphics.axis.Axes"
• "type":"choices={'yes','no','maybe'}"

We can even specify on-the-fly Matlab computation that returns a cell-array of values, for example a list of available fonts via "type":"choices=listfonts". A more complex example is the definition of the rmfield function, where the possible input choices for the second input arg (highlighted) depend on the struct that is provided in the first input arg (by running the fieldnames function on it):

"rmfield":
{
	"inputs":
	[
		{"name":"s",     "kind":"required", "type":"struct"},
		{"name":"field", "kind":"required", "type":"choices=fieldnames(s)"}	],
	"outputs":
	[
		{"name":"s", "type":"struct"}
	]
},

Alternative inputs

Multiple alternative inputs can be specified in the functionSignatures.json file. The easiest way to do so is to simply create multiple different definitions for the same function, one beneath the other. Matlab’s tab-completion parser is smart enough to combine those definitions and proceed with the most appropriate one based on the user-entered inputs.

For example, in the same Coder file above we find 6 alternative definitions. If (for example) we start typing coder('-ecoder', and click <Tab>, Matlab would automatically auto-complete the second input to “false”, and then another <Tab> click would set the third input to the required ‘-new’ parameter (see highlighted lines below):

...
"coder":
{
	"inputs":
	[
		{"name":"projectname", "kind":"required", "type":"filepath=*.prj"}
	]
},
"coder":
{
	"inputs":
	[
		{"name":"-open", "kind":"namevalue", "type":"filepath=*.prj"}
	]
},
"coder":
{
	"inputs":
	[
		{"name":"-build", "kind":"namevalue", "type":"filepath=*.prj"}
	]
},
"coder":
{
	"inputs":
	[
		{"name":"-new", "kind":"namevalue", "type":[["filepath=*.prj"], ["char"]]}
	]
},
"coder":
{
	"inputs":
	[
		{"name":"ecoderFlag",  "kind":"required", "type":"choices={'-ecoder'}"},		{"name":"ecoderValue", "kind":"required", "type":[["logical"], ["choices={'false'}"]]},		{"name":"newFlag",     "kind":"required", "type":"choices={'-new'}"},		{"name":"newvalue",    "kind":"required", "type":[["filepath=*.prj"], ["char"]]}	]
},
"coder":
{
	"inputs":
	[
		{"name":"tocodeFlag",  "kind":"required", "type":"choices={'-tocode'}"},
		{"name":"tocodevalue", "kind":"required", "type":"filepath=*.prj"},
		{"mutuallyExclusiveGroup":
			[
				[],
				[
					{"name":"scriptFlag", "kind":"required", "type":"choices={'-script'}"},
					{"name":"scriptname", "kind":"required", "type":[["filepath=*.m"], ["char"]]}
				]
			]
		}
	]
}

This example also shows, in the last definition for the coder function, another mechanism for specifying alternative inputs, using “mutuallyExclusiveGroup” (aka “MEGs”). A MEG is defined using an array of options, enclosed in square brackets ([]). Each of the MEG options is exclusive to each of the others, meaning that we can only work with one of them and not the others. This is equivalent to duplicating the definition as we saw above, and saves us some copy-paste (in some cases a lot of copy-pastes, especially with multiple and/or nested MEGs). However, MEGs have a major drawback of reduced readability. I believe that in most cases we only have a single MEG and few input args, and in such cases it makes more sense to use repeated function defs rather than a MEG. The Matlab signature files contain numerous usage examples for either of these two mechanisms.

Platform dependencies

If a specific function (or a specific signature variant) depends on the running platform, this can be specified via the “platform” directive. For example, the winopen function only works on Windows, but not on Linux/Mac. Its corresponding signature definition is:

"winopen":
{
	"platform":"win32,win64",	"inputs":
	[
		{"name":"filename", "kind":"required", "type":"filepath"},
		{"name":"varargin", "kind":"optional", "multiplicity":"append"}
	]
}

Platform dependence could also be specified at the parameter level, not just the entire function level. For example, in the xlsread function (defined in %matlabroot%/toolbox/matlab/iofun/functionSignatures.json), we see that the usage variant xlsread(filename,-1) is only available on Windows (note that the numeric value is defined as "<=-1", not necessarily -1), and so is the “functionHandle” input (which is called processFcn in the documentation – for some reason that escapes me the names of many input args do not match in the documentation and functionSignature):

"xlsread":
{
	"inputs":
	[
		{"name":"filename", "kind":"required", "type":"filepath=*.xls,*.xlsx,*.xlsb,*.csv"},
		{"mutuallyExclusiveGroup":
			[
				{"name":"openExcel", "kind":"required", "display":"", "type":["numeric", "<=-1"], "platform":"win64,win32"},				{"name":"xlRange",   "kind":"required", "type":["char", "@(x) isempty(x) || ~isempty(strfind(x, ':'))"], "default":"''"},
				[
					{"name":"sheet",          "kind":"positional", "type":[["char", "choices=matlab.internal.language.introspective.tabcompletion.xlsread_vsheet(filename)"], ["numeric", ">=1"]], "default":"1"},
					{"name":"xlRange",        "kind":"positional", "type":"char", "default":"''"},
					{"name":"basic",          "kind":"positional", "display":"", "type":["char", "choices={'basic',''}"]},
					{"name":"functionHandle", "kind":"positional", "type":"function_handle", "platform":"win64,win32"}				]
			]
		}
	],
        ...

Parsing errors

The new mechanism is not very user-friendly when you get something wrong. In the best case, it issues a cryptic error message (see below), and in the worst case it simply ignores the changes and the user has no idea why the new custom tab-completion is not working as intended.

The most likely causes of such problems are:

• The most common problem is that you placed the functionSignatures.json file in a different folder than the Matlab function. For example, if the myFunction() function is defined in myFunction.m, then the tab-completion of this function MUST be located in a functionSignatures.json file that resides in the same folder, not anywhere else on the Matlab path. In other words, the Matlab path is NOT relevant for tab-completion.
• Your functionSignatures.json file does not follow the [extremely strict] syntax rules above, to the letter. For example, forgetting the top or final curly braces, forgetting a comma or adding an extra one, or not closing all brackets/braces properly.
• You mistyped one or more of the input parameters, types or options.

In case of a parsing error, you’d see a red error message on the Matlab console the next time that you try to use tab-completion:

Error parsing JSON data; Boost reports "(189): expected ',' or ']'".

Unfortunately the error message only tells us the problematic line location within the functionSignatures.json file, but not the file’s location, so if we haven’t recently edited this file we’d need to find it in the relevant folder. For example:

edit(fullfile(fileparts(which('myFunction')), 'functionSignatures.json')

Moreover, when a JSON syntax error (such as the one above) occurs, the entire file is not parsed, not just the definition that caused the error.

Another limitation of tab-completion is that it does not work while the main Matlab thread is working (e.g., during a uiwait or waitfor). This may be somewhat misleading since most editor/debugging actions do work.

Arguably, this new tab-completion mechanism could be made more programmer-friendly. Perhaps this will improve in a future Matlab release.

Addendum: MathWorks has now made this functionality supported and documented. Read about it here.

For a related mechanism, see my article on tab-completion for class properties and methods from 2014, which is apparently still relevant and functional.

>> get(0,'ScreenSize')   % with 100% DPI (unscaled standard)
ans =
        1       1      1366       768
 
>> get(0,'ScreenSize')   % with 125% DPI (scaled)
ans =
        1       1      1092.8     614.4

The same phenomenon also affects other related properties, for example MonitorPositions.

Raimund Schlüßler, a reader on this blog, was kind enough to point me to this problem and its workaround, which I thought worthy to share here: To get the physical screen-size, use the following builtin Java command:

>> jScreenSize = java.awt.Toolkit.getDefaultToolkit.getScreenSize
jScreenSize =
java.awt.Dimension[width=1366,height=768]
 
>> width = jScreenSize.getWidth
width =
        1366
 
>> height = jScreenSize.getHeight
height =
        768

Also see the related recent article on an issue with the DPI-aware feature starting with R2015b.

Upcoming travels – London/Belfast, Zürich & Geneva

I will shortly be traveling to consult some clients in Belfast (via London), Zürich and Geneva. If you are in the area and wish to meet me to discuss how I could bring value to your work, then please email me (altmany at gmail):

• Belfast: Nov 28 – Dec 1 (flying via London)
• Zürich: Dec 11-12
• Geneva: Dec 13-15

Exception in thread "AWT-EventQueue-0" java.lang.NullPointerException
	at com.jidesoft.plaf.basic.BasicCellSpanTableUI.paint(Unknown Source)
	at javax.swing.plaf.ComponentUI.update(Unknown Source)
	at javax.swing.JComponent.paintComponent(Unknown Source)
	at com.jidesoft.grid.CellStyleTable.paintComponent(Unknown Source)
	at javax.swing.JComponent.paint(Unknown Source)
	at javax.swing.JComponent.paintToOffscreen(Unknown Source)
	...

Exception in thread "AWT-EventQueue-0" java.lang.ArrayIndexOutOfBoundsException: 1 >= 0
	at java.util.Vector.elementAt(Unknown Source)
	at javax.swing.table.DefaultTableColumnModel.getColumn(Unknown Source)
	at com.jidesoft.grid.ContextSensitiveTable.getCellRenderer(Unknown Source)
	at com.jidesoft.grid.CellSpanTable.getCellRenderer(Unknown Source)
	at com.jidesoft.grid.TreeTable.getActualCellRenderer(Unknown Source)
	at com.jidesoft.grid.GroupTable.getCellRenderer(Unknown Source)
	at com.jidesoft.grid.JideTable.b(Unknown Source)
	at com.jidesoft.grid.CellSpanTable.calculateRowHeight(Unknown Source)
	...

In almost all such Java error messages, the error is asynchronous to the Matlab code and does not interrupt it. No error exception is thrown (or can be trapped), and the Matlab code proceeds without being aware that anything is wrong. In fact, in the vast majority of such cases, nothing is visibly wrong – the program somehow overcomes the reported problem and there are no visible negative effects on the GUI. In other words, these error messages are harmless and can almost always be ignored. Still, if we could only stop those annoying endless red stack-trace messages in the Matlab console!

Note that today’s post only discusses untrappable asynchronous Java error messages, not synchronous errors that can be trapped in Matlab via try-catch. These synchronous errors are often due to programmatic errors (e.g., bad method input args or an empty reference handle) and can easily be handled programmatically. On the other hand, the asynchronous errors are non-trappable, so they are much more difficult to isolate and fix.

In many of the cases, the error occurs when the control’s underlying data model is changed by the Matlab code, and some of the controls’s Java methods are not synced with the new model by the time they run. This can be due to internal bugs in the Matlab or Java control’s implementation, or to simple race conditions that occur between the Matlab thread and the Java Event Dispatch Thread (EDT). As noted here, such race conditions can often be solved by introducing a simple delay into the Matlab code:

javaControl.doSomething();
pause(0.05); drawnow;javaControl.doSomethingElse();

In addition, asking Matlab to run the Java component’s methods on the EDT can also help solve race conditions:

javaControl = javaObjectEDT(javaControl);

Unfortunately, sometimes both of these are not enough. In such cases, one of the following ideas might help:

• Add fprintf(' \b') to your Matlab code: this seemingly innocent hack of displaying a space & immediately erasing it with backspace, appears to force the Java engine to flush its event queue and synchronize things, thereby avoiding the annoying Java console errors. I know it sounds like adding a sprinkle of useless black magic to the code, but it does really work in some cases!

  javaControl.doSomething();
  pause(0.05); drawnow;  % this never hurt anyone!
  fprintf(' \b');javaControl.doSomethingElse();

• It is also possible to directly access the console text area and remove all the text after a certain point. Note that I strongly discourage messing around with the console text in this manner, since it might cause problems with Matlab’s internals. Still, if you are adventurous enough to try, then here’s an example:

  jCmdWinDoc = com.mathworks.mde.cmdwin.CmdWinDocument.getInstance;
  currentPos = cmdWinDoc.getLength;
   
  javaControl.doSomething();
  pause(0.05); drawnow;  % this never hurt anyone!
  javaControl.doSomethingElse();
   
  pause(0.1);  % let the java error time to display itself in the console
  jCmdWinDoc.remove(currentPos, cmdWinDoc.getLength-currentPos);

• When all else fails, consider simply clearing the Matlab console using the Matlab clc command a short while after updating the Java control. This will erase the red Java errors, along with everything else in the console, so naturally it cannot be freely used if you use the console to display useful information to the user.

It should be emphasized: not all of these suggested remedies work in all cases; in some cases some of them work, and in other cases others might work. There does not seem to be a general panacea to this issue. The main purpose of the article was to list the possible solutions in a single place, so that users could try them out and select those that work for each specific case.

Do you know of any other (perhaps better) way of avoiding or hiding such asynchronous Java console errors? If so, then please post a comment below.

Function call timeline profiling (click for full-size image)

Today I will discuss a related undocumented feature of the Profiler: loading and viewing pre-saved profiling results.

Programmatic access to profiling results

Matlab’s syntax for returning the detailed profiling results in a data struct is clearly documented in the profile function’s doc page. Although the documentation does not explain the resulting struct and sub-struct fields, they have meaningful names and we can relatively easily infer what each of them means (I added a few annotation comments for clarity):

>> profile('on','-history')
>> surf(peaks); drawnow
>> profile('off')
>> profData = profile('info')
profData = 
      FunctionTable: [26x1 struct]
    FunctionHistory: [2x56 double]
     ClockPrecision: 4.10517962829241e-07
         ClockSpeed: 2501000000
               Name: 'MATLAB'
           Overhead: 0
 
>> profData.FunctionTable(1)
ans = 
          CompleteName: 'C:\Program Files\Matlab\R2016a\toolbox\matlab\specgraph\peaks.m>peaks'
          FunctionName: 'peaks'
              FileName: 'C:\Program Files\Matlab\R2016a\toolbox\matlab\specgraph\peaks.m'
                  Type: 'M-function'
              Children: [1x1 struct]
               Parents: [0x1 struct]
         ExecutedLines: [9x3 double]
           IsRecursive: 0
    TotalRecursiveTime: 0
           PartialData: 0
              NumCalls: 1
             TotalTime: 0.0191679078068094
 
>> profData.FunctionTable(1).Children
ans = 
        Index: 2   % index in profData.FunctionTable array
     NumCalls: 1
    TotalTime: 0.00136415141013509
 
>> profData.FunctionTable(1).ExecutedLines   % line number, number of calls, duration in secs
ans =
         43      1      0.000160102031282782
         44      1      2.29890096200918e-05
         45      1      0.00647592190637408
         56      1      0.0017093970724654
         57      1      0.00145036019621044
         58      1      0.000304193859437286
         60      1      4.39254290955326e-05
         62      1      3.44835144301377e-05
         63      1      0.000138755093778411
 
>> profData.FunctionHistory(:,1:5)
ans =
     0     0     1     1     0   % 0=enter, 1=exit
     1     2     2     1     6   % index in profData.FunctionHistory array

As we can see, this is pretty intuitive so far.

Loading and viewing saved profiling results

If we wish to save these results results in a file and later load and display them in the Profiler’s visualization browser, then we need to venture deeper into undocumented territory. It seems that while retrieving the profiling results (via profile(‘info’)) is fully documented, doing the natural complementary action (namely, loading this data into the viewer) is not. For the life of me I cannot understand the logic behind this decision, but that’s the way it is.

Luckily, the semi-documented built-in function profview does exactly what we need: profview accepts 2 input args (function name and the profData struct) and displays the resulting profiling info. The first input arg (function name) accepts either a string (e.g., 'peaks' or 'view>isAxesHandle'), or the numeric value 0 which signifies the home (top-level) page:

profView(0, profData);  % display profiling home (top-level) page
profview('peaks', profData);  % display a specific profiling page

I use the 0 input value much more frequently than the string inputs, because I often don’t know which functions exactly were profiled, and starting at the home page enables me to easily drill-down the profiling results interactively.

Loading saved profiling results from a different computer

Things get slightly complicated if we try to load saved profiling results from a different computer. If the other computer has exactly the same folder structure as our computer, and all our Matlab functions reside in exactly the same disk folders/path, then everything will work out of the box. The problem is that in general the other computer will have the functions in different folders. When we then try to load the profData on our computer, it will not find the associated Matlab functinos in order to display the line-by-line profiling results. We will only see the profiling data at the function level, not line level. This significantly reduces the usefulness of the profiling data. The Profiler page will display the following error message:

We can solve this problem in either of two ways:

1. Modify our profData to use the correct folder path on the local computer, rather than the other computer’s path (which is invalid on the local computer). For example:

   % Save the profData on computer #1:
   profData = profile('info');
   save('profData.mat', 'profData');
    
   % Load the profData on computer #2:
   fileData = load('profData.mat');
   profData = fileData.profData;
   path1 = 'N:\Users\Juan\programs\myProgram';
   path2 = 'C:\Yair\consulting\clients\Intel\code';
   for idx = 1 : numel(profData.FunctionTable)
      funcData = profData.FunctionTable(idx);
      funcData.FileName     = strrep(funcData.FileName,     path1, path2); 
      funcData.CompleteName = strrep(funcData.CompleteName, path1, path2);
      profData.FunctionTable(idx) = funcData;
   end
   % note: this loop can be vectorized if you wish

2. As an alternative, we can modify Matlab’s profview.m function (%matlabroot%/toolbox/matlab/codetools/profview.m) to search for the function’s source code in the current Matlab path, if the specified direct path is not found (note that changing profview.m may require administrator priviledges). For example, the following is the code from R2016a’s profview.m file, line #506:

       % g894021 - Make sure the MATLAB code file still exists
       if ~exist(fullName, 'file')
           [~,fname,fext] = fileparts(fullName);  % Yair        fname = which([fname fext]);           % Yair        if isempty(fname)                      % Yair            mFileFlag = 0;
           else                                   % Yair            fullName = fname;                  % Yair        end                                    % Yair    end

These two workarounds complement each other: the first workaround does not require changing any installed Matlab code, and so is platform- and release-independent, but would require rerunning the code snippet for each and every profiling data file that we receive from external computers. On the other hand, the second workaround is a one-time operation that should work for multiple saved profiling results, although we would need to redo it whenever we install Matlab.

Additional profview customizations

Modifying the profview.m function can be used for different improvements as well.

For example, several years ago I explained how this function can be modified to display 1 ms timing resolutions, rather than the default 10 mS.

Another customization that I often do after I install Matlab is to change the default setting of truncating function lines longer than 40 characters – I typically modify this to 60 or 80 (depending on the computer monitor’s size…). All we need to do is to update the truncateDisplayName sub-function within profview.m as follows (taken from R2016a again, line #1762):

function shortFileName = truncateDisplayName(longFileName,maxNameLen)
%TRUNCATEDISPLAYNAME  Truncate the name if it gets too long
maxNameLen = max(60,maxNameLen);  % YairshortFileName = escapeHtml(longFileName);
if length(longFileName) > maxNameLen,
    shortFileName = char(com.mathworks.util.FileUtils.truncatePathname( ...
        shortFileName, maxNameLen));
end

You can see additional undocumented profiling features in the “Related posts” section below, as well as in Chapter 2 of my book “Accelerating MATLAB Performance“.

Do you have any other customization to the profiling results? If so, please share it in a comment.

Custom shortcut controls

Today’s article will focus on the new toolstrip interface of Matlab release R2012b and later; adaptation of the code to R2012a and earlier is relatively easy (in fact, simpler than the toolstrip-based code below).

Displaying the Shortcuts panel

Before we begin to modify shortcuts in the Toolstrip’s shortcuts menu, we need to ensure that the Shortcuts panel is visible and active (in current focus), otherwise our customizations will be ignored or cause an error. There is probably a more direct way of doing this, but a simple way that I found was to edit the current Desktop’s layout to include a directive to display the Shortcuts tab, and then load that layout:

jDesktop = com.mathworks.mde.desk.MLDesktop.getInstance;
hMainFrame = com.mathworks.mde.desk.MLDesktop.getInstance.getMainFrame;
jToolstrip = hMainFrame.getToolstrip;
isOk = jToolstrip.setCurrentTab('shortcuts');
if ~isOk  % i.e., Shortcuts tab is NOT displayed
    % Save the current Desktop layout
    jDesktop.saveLayout('Yair');  pause(0.15);
 
    % Update the layout file to display the Shortcuts tab
    filename = fullfile(prefdir, 'YairMATLABLayout.xml');
    fid = fopen(filename, 'rt');
    txt = fread(fid, '*char')';
    fclose(fid);
    txt = regexprep(txt,'(ShowShortcutsTab=)"[^"]*"','');
    txt = regexprep(txt,'(<Layout [^>]*)>','$1 ShowShortcutsTab="yes">');
    fid = fopen(filename, 'wt');
    fwrite(fid,txt);
    fclose(fid);
 
    % Load the modified layout
    jDesktop.restoreLayout('Yair');  pause(0.15);
 
    % The shortcuts tab should now be visible, so transfer focus to that tab
    jToolstrip.setCurrentTab('shortcuts');
end

Custom controls

As I explained in last week’s post, we can use scUtils.addShortcutToBottom to add a simple push-button shortcut to the relevant category panel within the Shortcuts toolstrip tab. To add custom controls, we can simply add the controls to the relevant shortcut category panel container (a com.mathworks.toolstrip.components.TSPanel object). The standard shortcuts are typically placed in the Shortcuts tab’s second TSPanel (“general”), and other categories have TSPanels of their own.

Now here’s the tricky part about TSPanels: we cannot directly add components to the sectino panel (that would be too easy…): the section panels are composed of an array of internal TSPanels, and we need to add the new controls to those internal panels. However, these panels only contain 3 empty slots. If we try to add more than 3 components, the 4th+ component(s) will simply not be displayed. In such cases, we need to create a new TSPanel to display the extra components.

Here then is some sample code to add a combo-box (drop-down) control:

% First, get the last internal TSPanel within the Shortcuts tab's "general" section panel
% Note: jToolstrip was defined in the previous section above
jShortcutsTab = jToolstrip.getModel.get('shortcuts').getComponent;
jSectionPanel = jShortcutsTab.getSectionComponent(1).getSection.getComponent;  % the TSPanel object "general"
jContainer = jSectionPanel.getComponent(jSectionPanel.getComponentCount-1);
 
% If the last internal TSPanel is full, then prepare a new internal TSPanel next to it
if jContainer.getComponentCount >= 3
    % Create a new empty TSPanel and add it to the right of the last internal TSPanel
    jContainer = com.mathworks.toolstrip.components.TSPanel;
    jContainer.setPreferredSize(java.awt.Dimension(100,72));
    jSectionPanel.add(jContainer);
    jSectionPanel.repaint();
    jSectionPanel.revalidate();
end
 
% Create the new control with a custom tooltip and callback function
optionStrings = {'Project A', 'Project B', 'Project C'};
jCombo = com.mathworks.toolstrip.components.TSComboBox(optionStrings);
jCombo = handle(javaObjectEDT(jCombo), 'callbackproperties'));
set(jCombo, 'ActionPerformedCallback', @myCallbackFunction);
jCombo.setToolTipText('Select the requested project');
 
% Now add the new control to the internal TSPanel
jContainer.add(jCombo);
jContainer.repaint();
jContainer.revalidate();

Custom shortcut controls

Matlab’s internal com.mathworks.toolstrip.components package contains many embeddable controls, including the following (I emphasized those that I think are most useful within the context of the Shortcuts panel): TSButton, TSCheckBox, TSComboBox, TSDropDownButton (a custom combo-box component), TSFormattedTextField, TSLabel, TSList, TSRadioButton, TSScrollPane, TSSlider, TSSpinner, TSSplitButton, TSTextArea, TSTextField, and TSToggleButton. These controls are in most cases simple wrappers of the corresponding Java Swing controls. For example, TSSpinner extends the standard Swing JSpinner control. In some cases, the controls are more complex: for example, the TSSplitButton is similar to Matlab’s uisplittool control.

Toolstrip controls

In fact, these controls can be used even outside the toolstrip, embedded directly in our Matlab figure GUI, using the javacomponent function. For example:

dataModel = javax.swing.SpinnerNumberModel(125, 15, 225, 0.5);  % defaultValue, minValue, maxValue, stepSize
jSpinner = com.mathworks.toolstrip.components.TSSpinner(dataModel);
jSpinner = handle(javaObjectEDT(jSpinner), 'CallbackProperties');
[hjSpinner, hContainer] = javacomponent(jSpinner, [10,10,60,20], gcf);

You can find additional interesting components within the %matlabroot%/java/jar/toolstrip.jar file, which can be opened in any zip file utility or Java IDE. In fact, whatever controls that you see Matlab uses in its Desktop toolstrip (including galleries etc.) can be replicated in custom tabs, sections and panels of our own design.

Matlab Desktop’s interactive GUI only enables creating simple push-button shortcuts having string callbacks (that are eval‘ed in run-time). Using the undocumented programmatic interface that I just showed, we can include more sophisticated controls, as well as customize those controls in ways that are impossible via the programmatic GUI: add tooltips, set non-string (function-handle) callbacks, enable/disable controls, modify icons in run-time etc.

For example (intentionally showing two separate ways of setting the component properties):

% Toggle-button
jTB = handle(javaObjectEDT(com.mathworks.toolstrip.components.TSToggleButton('Toggle button')), 'CallbackProperties')
jTB.setSelected(true)
jTB.setToolTipText('toggle me!')
jTB.ActionPerformedCallback = @(h,e)doSomething();
jContainer.add(jTB);
 
% Check-box
jCB = handle(javaObjectEDT(com.mathworks.toolstrip.components.TSCheckBox('selected !')), 'CallbackProperties');
set(jCB, 'Selected', true, 'ToolTipText','Please select me!', 'ActionPerformedCallback',{@myCallbackFunction,extraData});
jContainer.add(jCB);

(resulting in the screenshot at the top of this post)

Important note: none of these customizations is saved to file. Therefore, they need to be redone programmatically for each separate Matlab session. You can easily do that by calling the relevant code in your startup.m file.

If you wish me to assist with any customization of the Desktop shortcuts, or any other Matlab aspect, then contact me for a short consultancy.

Happy New Year everybody!

Such programmatic access is often useful. For example, a large company for which I consult is using centralized updates to users’ shortcuts, in order to manage and expose new features for all Matlab users from a central location. It is easy to send updates and manage a few users, but when your organization has dozens of Matlab users, centralized management becomes a necessity. It’s a pity that companies need to resort to external consultants and undocumented hacks to achieve this, but I’m not complaining since it keeps me occupied…

Shortcuts in Matlab R2012a and earlier

Shortcuts in Matlab R2012b and newer

Today’s post will describe “regular” shortcuts – those that are simple clickable buttons. Next week I will show how we can extend this to incorporate other types of shortcut controls, as well as some advanced customizations.

The shortcuts.xml file

It turns out that the shortcults toolbar (on R2012a and earlier) or toolstrip group (on R2012b onward) is a reflection of the contents of the [prefdir ‘\shortcuts.xml’] file (depending on your version, the file might be named somewhat differently, i.e. shortcuts_2.xml). This file can be edited in any text editor, Matlab’s editor included. So a very easy way to programmatically affect the shortcuts is to update this file. Here is a sample of this file:

<?xml version="1.0" encoding="utf-8"?>
<FAVORITESROOT version="2">
   <title>My Shortcuts</title>
   <FAVORITECATEGORY>
      <name>Help Browser Favorites</name>
      <FAVORITE>
         <label>Help Using the Desktop</label>
         <icon>Help icon</icon>
         <callback>helpview([docroot '/mapfiles/matlab_env.map'], 'matlabenvironment_desktop');</callback>
         <editable>true</editable>
      </FAVORITE>
   </FAVORITECATEGORY>
   <FAVORITE>
      <label>CSSM</label>
      <icon>Help icon</icon>
      <callback>disp('No callback specified for this shortcut')</callback>
      <editable>true</editable>
   </FAVORITE>
   <FAVORITE>
      <label>UndocML</label>
      <icon>MATLAB icon</icon>
      <callback>web('undocumentedMatlab.com')</callback>
      <editable>true</editable>
   </FAVORITE>
   <FAVORITE>
      <label>My favorite program</label>
      <icon>C:\Yair\program\icon.gif</icon>
      <callback>cd('C:\Yair\program'); myProgram(123);</callback>
      <editable>true</editable>
   </FAVORITE>
   ...
</FAVORITESROOT>

The file is only loaded once during Matlab startup, so any changes made to it will only take effect after Matlab restarts.

Updating the shortcuts in the current Matlab session

We can update the shortcuts directly, in the current Matlab session, using the builtin com.mathworks.mlwidgets.shortcuts.­ShortcutUtils class. This class has existed largely unchanged for numerous releases (at least as far back as R2008b).

For example, to add a new shortcut to the toolbar:

name = 'My New Shortcut';
cbstr = 'disp(''My New Shortcut'')';  % will be eval'ed when clicked
iconfile = 'c:\path\to\icon.gif';  % default icon if it is not found
isEditable = 'true';
scUtils = com.mathworks.mlwidgets.shortcuts.ShortcutUtils;
category = scUtils.getDefaultToolbarCategoryName;
scUtils.addShortcutToBottom(name,cbstr,iconfile,category,isEditable);

The shortcut’s icon can either be set to a specific icon filepath (e.g., ‘C:\Yair\program\icon.jpg’), or to one of the predefined names: ‘Help icon’, ‘Standard icon’, ‘MATLAB icon’ or ‘Simulink icon’. The editable parameter does not seem to have a visible effect that I could see.

The category name can either be set to the default name using scUtils.getDefaultToolbarCategoryName (‘Shortcuts’ on English-based Matlab R2012b onward), or it can be set to any other name (e.g., ‘My programs’). To add a shortcut to the Help Browser (also known as a “Favorite”), simply set the category to scUtils.getDefaultHelpCategoryName (=’Help Browser Favorites’ on English-based Matlab installations); to add the shortcut to the ‘Start’ button, set the category to ‘Shortcuts’. When you use a non-default category name on R2012a and earlier, you will only see the shortcuts via Matlab’s “Start” button (as seen in the screenshot below); on R2012b onward you will see it as a new category group within the Shortcuts toolstrip (as seen in the screenshot above). For example:

scUtils = com.mathworks.mlwidgets.shortcuts.ShortcutUtils;
scUtils.addShortcutToBottom('clear', 'clear; clc', 'Standard icon', 'Special commands', 'true');

Custom category in Matlab R2010a

To remove a shortcut, use the removeShortcut(category,shortcutName) method (note: this method does not complain if the specified shortcut does not exist):

scUtils.removeShortcut('Shortcuts', 'My New Shortcut');

The addShortcutToBottom() method does not override existing shortcuts. Therefore, to ensure that we don’t add duplicate shortcuts, we must first remove the possibly-existing shortcut using removeShortcut() before adding it. Since removeShortcut() does not complain if the specific shortcut is not found, we can safely use it without having to loop over all the existing shortcuts. Alternately, we could loop over all existing category shortcuts checking their label, and adding a new shortcut only if it is not already found, as follows:

scUtils = com.mathworks.mlwidgets.shortcuts.ShortcutUtils;
category = scUtils.getDefaultToolbarCategoryName; 
scVector = scUtils.getShortcutsByCategory(category);
scArray = scVector.toArray;  % Java array
foundFlag = 0;
for scIdx = 1:length(scArray)
   scName = char(scArray(scIdx));
   if strcmp(scName, 'My New Shortcut')
      foundFlag = 1; break;
      % alternatively: scUtils.removeShortcut(category, scName);
   end
end
if ~foundFlag
   scUtils.addShortcutToBottom(scName, callbackString, iconString, category, 'true');
end

As noted above, we can add categories by simply specifying a new category name in the call to scUtils.addShortcutToBottom(). We can also add and remove categories directly, as follows (beware: when removing a category, it is removed together with all its contents):

scUtils.addNewCategory('category name');
scUtils.removeShortcut('category name', []);  % entire category will be deleted

Shortcut tools on the Matlab File Exchange

Following my advice on StackOverflow back in 2010, Richie Cotton wrapped the code snippets above in a user-friendly utility (set of independent Matlab functions) that can now be found on the Matlab File Exchange and on his blog. Richie tested his toolbox on Matlab releases as old as R2008b, but the functionality may also work on even older releases.

Shortcuts panel embedded in Matlab GUI

Shortcuts are normally visible in the toolbar and the Matlab start menu (R2012a and earlier) or the Matlab Desktop’s toolstrip (R2012b onward). However, using com.mathworks.mlwidgets.shortcuts.ShortcutTreePanel, the schortcuts can also be displayed in any user GUI, complete with right-click context-menu:

jShortcuts = com.mathworks.mlwidgets.shortcuts.ShortcutTreePanel;
[jhShortcuts,hPanel] = javacomponent(jShortcuts, [10,10,300,200], gcf);

Shortcuts panel in Matlab figure GUI

Stay tuned…

Next week I will expand the discussion of Matlab shortcuts with the following improvements:

1. Displaying non-standard controls as shortcuts: checkboxes, drop-downs (combo-boxes) and toggle-buttons
2. Customizing the shortcut tooltip (replacing the default tooltip that simply repeats the callback string)
3. Customizing the shortcut callback (rather than using an eval-ed callback string)
4. Enabling/disabling shortcuts in run-time

Merry Christmas everyone!

To date I’ve reported dozens of bugs and as far as I can tell, few if any of them have actually been fixed, years after I’ve reported them. None of them appear on Matlab’s official bug parade, which is only a small subset of the full list that MathWorks keeps hidden for some unknown reason (update: see the discussion in the comments thread below, especially the input by Steve Eddins). Never mind, I don’t take it personally, I simply find a workaround and move on. I’ve already posted about this before. Today I’ll discuss two additional bugs I’ve run across once-too-often, and my workarounds:

• Saving non-Latin Command Window text using diary
• Printing GUIs reliably

Nothing really earth-shattering, but annoying nonetheless.

Saving non-Latin Command Window text using diary

The diary function is well-known for saving Matlab’s Command-Window (CW) text to a file. The function has existed for the past two decades at least, possibly even longer.

Unfortunately, perhaps the developer never thought that Matlab would be used outside the Americas and Western Europe, otherwise I cannot understand why to this day diary saves the text in ASCII format rather than the UTF-16 variant used by the CW. This works ok for basic Latin characters, but anyone who outputs Chinese, Japanese, Korean, Hindi, Arabic, Hebrew or other alphabets to the CW, and tries to save it using diary, will find the file unreadable.

Here is a sample illustrative script, that outputs the Arabic word salaam (peace, سلام) to the CW and then tries to save this using diary. If you try it, you will see it ok in the CW, but garbage text in the generated text file:

>> fname='diary_bug.txt'; diary(fname); disp(char([1587,1604,1575,1605])); diary off; winopen(fname)
سلام

The problem is that since diary assumes ASCII characters, any characters having a numeric value above 255 get truncated and are stored as invalid 1-byte characters, char(26) in this case.

Here’s my workaround:

% Output Command Window text to a text file
function saveCmdWinText(filename)
    cmdWinDoc = com.mathworks.mde.cmdwin.CmdWinDocument.getInstance;
    txt = char(cmdWinDoc.getText(0,cmdWinDoc.getLength));
    fid = fopen(filename,'W');
    fwrite(fid,txt,'uint16');  % store as 2-byte characters
    fclose(fid);
    %winopen(filename);  % in case you wish to verify...
end

This works well, saving the characters in their original 2-byte format, for those alphabets that use 2-bytes: non-basic Latins, Greek, Cyrillic, Armenian, Arabic, Hebrew, Coptic, Syriac and Tāna (I don’t think there are more than a handful of Matlab users who use Coptic, Syriac or Tāna but never mind). However, UTF-8 specifies that CJK characters need 3-4 bytes and this is apparently not supported in Matlab, whose basic char data type only has 2 bytes, so I assume that Chinese, Japanese and Korean will probably require a different solution (perhaps the internal implementation of char and the CW is different in the Chinese/Japanese versions of Matlab, I really don’t know. If this is indeed the case, then perhaps a variant of my workaround can also be used for CJK output).

Correction #1: I have learned since posting (see Steve Eddins’ comment below) that Matlab actually uses UTF-16 rather than UTF-8, solving the CJK issue. I humbly stand corrected.

Correction #2: The saveCmdWinText code above saves the CW text in UTF-16 format. This may be problematic in some text editors that are not UTF-savvy. For such editors (or if your editor get confused with the various BOM/endianness options), consider saving the data in UTF-8 format – again, assuming you’re not using an alphabet [such as CJK] outside the ASCII range (thanks Rob):

function saveCmdWinText_UTF8(filename)
    cmdWinDoc = com.mathworks.mde.cmdwin.CmdWinDocument.getInstance;
    txt = char(cmdWinDoc.getText(0,cmdWinDoc.getLength));
    fid = fopen(filename,'W','n','utf-8');
    fwrite(fid,txt,'char');
    fclose(fid);
    %winopen(filename);  % in case you wish to verify...
end

Also, this workaround is problematic in the sense that it’s a one-time operation that stores the entire CW text that is visible at that point. This is more limited than diary‘s ability to start and stop output recording in mid-run, and to record output on-the-fly (rather than only at the end). Still, it does provide a solution in case you output non-ASCII 2-byte characters to the CW.

Update: I plan to post a utility to the Matlab File Exchange in the near future that will mimic diary‘s ability to start/stop text recording, rather than simply dumping the entire CW contents to file. I’ll update here when this utility is ready for download.

There are various other bugs related to entering non-Latin (and specifically RTL) characters in the CW and the Matlab Editor. Solving the diary bug is certainly the least of these worries. Life goes on…

p.s. – I typically use this translator to convert from native script to UTF codes that can be used in Matlab. I’m sure there are plenty of other translators, but this one does the job well enough for me.

For people interested in learning more about the Command Window internals, take a look at my cprintf and setPrompt utilities.

cprintf usage examples

setPrompt usage examples

Printing GUIs reliably

Matlab has always tried to be far too sophisticated for its own good when printing figures. There’s plenty of internal code that tries to handle numerous circumstances in the figure contents for optimal output. Unfortunately, this code also has many bugs. Try printing even a slightly-complex GUI containing panels and/or Java controls and you’ll see components overlapping each other, not being printed, and/or being rendered incorrectly in the printed output. Not to mention the visible flicker that happens when Matlab modifies the figure in preparation for printing, and then modifies it back to the original.

All this when a simple printout of a screen-capture would be both much faster and 100% reliable.

Which is where my ScreenCapture utility comes in. Unlike Matlab’s print and getframe, ScreenCapture takes an actual screen-capture of an entire figure, or part of a figure (or even a desktop area outside any Matlab figure), and can then send the resulting image to a Matlab variable (2D RGB image), an image file, system clipboard, or the printer. We can easily modify the <Print> toolbar button and menu item to use this utility rather than the builtin print function:

Matlab's default toolbar Print action

hToolbar = findall(gcf,'tag','FigureToolBar');
hPrintButton = findall(hToolbar, 'tag','Standard.PrintFigure');
set(hPrintButton, 'ClickedCallback','screencapture(gcbf,[],''printer'')');
 
hPrintMenuItem = findall(gcf, 'type','uimenu', 'tag','printMenu');
set(hPrintMenuItem,      'Callback','screencapture(gcbf,[],''printer'')');

This prints the entire figure, including the frame, menubar and toolbar (if any). If you just wish to print the figure’s content area, then make sure to create a top-level uipanel that spans the entire content area and in which all the contents are included. Then simply pass this top-level container handle to ScreenCapture:

hTopLevelContainer = uipanel('BorderType','none', 'Parent',gcf, 'Units','norm', 'Pos',[0,0,1,1]);
...
hToolbar = findall(gcf,'tag','FigureToolBar');
hPrintButton = findall(hToolbar, 'tag','Standard.PrintFigure');
set(hPrintButton, 'ClickedCallback',@(h,e)screencapture(hTopLevelContainer,[],'printer'));
 
hPrintMenuItem = findall(gcf, 'type','uimenu', 'tag','printMenu');
set(hPrintMenuItem,      'Callback',@(h,e)screencapture(hTopLevelContainer,[],'printer'));

In certain cases (depending on platform/OS/Matlab-release), the result may capture a few pixels from the figure’s window frame. This can easily be corrected by specifying a small offset to ScreenCapture:

set(hPrintButton, 'ClickedCallback',@(h,e)printPanel(hTopLevelContainer));
set(hPrintMenuItem,      'Callback',@(h,e)printPanel(hTopLevelContainer));
 
function printPanel(hTopLevelContainer)
    pos = getpixelposition(hTopLevelContainer);
    screencapture(hTopLevelContainer, pos+[2,4,0,0], 'printer');
end

I was thrilled to see the answer in the recent Matlab Computational Finance Conference, London. I presented at last year’s conference, so I was excited to review this year’s presentations when I came across a nugget in Kevin Shea’s keynote presentation on new Matlab developments. Note that there were two separate Matlab computational-finance events in 2014 – in London (June 24) and NY (April 9); the interesting piece is from London. Unlike the NY conference, the London conference proceedings do not include a video recording, only the slides (perhaps the video will be added to the proceedings page later, after all it takes some time to process). The last slide of Kevin’s presentation shows a screenshot of a Chrome browser displaying what appears to be a full-fledged Matlab desktop (Workspace, Editor, Command Window, figures) at https://matlab.mathworks.com:

Matlab Online (click to zoom)

The benefits of using Matlab Online are potentially tremendous: excluding some limitations (e.g., limited GUI, run time and supported toolboxes), we could deploy our Matlab programs online with minimal to no additional dev effort.

Information on Matlab Online can be found at http://mathworks.com/products/matlab/online. It even includes detailed dedicated documentation for all the supported functions and features.

For some reason, MathWorks has chosen not to link this new (and in my humble opinion, great) product in its official list of products. Perhaps the reason is that MathWorks do not wish for Matlab Online to cannibalize its main offerings. This might also explain the limitations placed on the online version (some limitations are technical, the rest are business-related). Matlab Online requires a dedicated login access, provided by MathWorks. Unfortunately, it is still not fully released to the public and is only available to users with up to date student licenses.

Still, if you are an important enough client of MathWorks, you might ask for access. Even with its current limitations, Matlab Online may well be useful for your needs. If you press strongly enough, and if your account is large enough, perhaps MathWorks might even enable a dedicated subdomain for you (as in intel.mathworks.com or matlab.apple.com).

Here’s another screenshot, in case you’re not convinced yet:

Matlab Online (click to zoom)

A post by MathWorks on the Answers forum earlier this year lists Matlab Online along with some additional online-computing alternatives for Matlab.

And now for some guesswork/speculation: MathWorks placed a major bet on Java technology in the early 2000’s (actually the late 90’s, since R12 took a few years to develop). Java was indeed hot at that time, but then so were some other technologies over the years. Fortunately for MathWorks, Java proved agile, mature and portable enough to enable mobile and online porting. This could explain the lack of Simulink GUI support, since Simulink GUI is still C++-based to a large extent. It might also explain the extra work done in HG2 in the graphic infrastructure (previously C++-based). After all, the basic MCOS graphic classes were available years ago, so if HG2 was only about a transition from double-value handles to MCOS, and some nice beautifications (anti-aliasing, CVD-aware colors etc.) then HG2 could be released long ago. Working on the underlying engine to make it portable could well explain HG2’s belated arrival.

So what does all this mean about Matlab’s future? Well, it appears to me that MathWorks’ apparent move towards SaaS (software as a service) and cloud-based computing is slightly belated, but quite evidently follows a general industry trend. In my eyes it heralds a move by MathWorks from desktop to online services, perhaps even to pay-per-use computing (as by Techila for example). The Matlab desktop will still be MathWorks’ bread and butter for many years to come. But the ability of Matlab programs to work either locally (on a Matlab client, either thin or thick) or online would be an enormous productivity benefit, in essence being a “killer feature” over Matlab’s competitors.

Technically, Matlab’s online integration could enable closer integration of online content in Matlab programs (esp. GUI). I’ve already shown how active web content can be displayed in Matlab GUI, but a closer integration could mean this might all become fully documented and integrated (recall Windows’ failed Active Desktop on one hand, but smart-phones’ enormously-successful active widgets on the other). It could also enable closer integration with online graphing services such as Plotly.

I for one, can’t wait to see this dream being realized. It’s not too far down the road, it would seem. I just hope that licensing and cost considerations won’t keep us from using it. MathWork’s recent Home License scheme seems to indicate that this is well understood by MathWorks, so I am highly optimistic

Then again, all this might be pure baseless speculation. Time will tell.

Unfortunately, it turns out that a technical challenge was preventing this move: the other technology enabled data field names (identifiers) longer than Matlab’s namelengthmax=63 characters, and these names also sometimes contained illegal identifier characters, such as spaces or symbols. This prevented an easy code transition, indefinitely delaying the migration to Matlab.

Accessing improper fieldnames

I suggested to this client to use Matlab class objects that overloaded the subsref() and subsasgn() methods: the long original identifiers would be stored in some internal container (cell array or containers.Map etc.), and they would be accessed not directly (which would be impossible since they are not valid Matlab identifiers) but via the overloaded methods. Something along the following lines:

classdef ExtendedFieldsClass
    % The internal data implementation is not publicly exposed
    properties (Access = 'protected')
        props = containers.Map;
    end
 
    methods (Access = 'public', Hidden=true)
 
        % Overload property assignment
        function obj = subsasgn(obj, subStruct, value)
            if strcmp(subStruct.type,'.')
                try
                    obj.props(subStruct.subs) = value;
                catch
                    error('Could not assign "%s" property value', subStruct.subs);
                end
            else  % '()' or '{}'
                error('not supported');
            end
        end
 
        % Overload property retrieval (referencing)
        function value = subsref(obj, subStruct)
            if strcmp(subStruct.type,'.')
                try
                    value = obj.props(subStruct.subs);
                catch
                    error('"%s" is not defined as a property', subStruct.subs);
                end
            else  % '()' or '{}'
                error('not supported');
            end
        end
    end
end

This works splendidly, as the following snippet shows:

>> c = ExtendedFieldsClass
c = 
  ExtendedFieldsClass with no properties.
 
>> c.(' asd f @#$^$%&') = -13.5;  % no error
 
>> c.(' asd f @#$^$%&')
ans =
                     -13.5
 
>> c.(' asd f @#$^$%& xyz')  % note the extra "xyz"
Error using ExtendedFieldsClass/subsref (line 27)
" asd f @#$^$%& xyz" is not defined as a property

Note how we need to use the () parentheses in order to access the “properties” as dynamic fieldnames. We would naturally get an error if we tried to directly access the field:

>> c. asd f @#$^$%&
 c. asd f @#$^$%&
        |
Error: Unexpected MATLAB expression.

Tab completion

So far so good.

The problem is that we would also like to see the defined “properties” when in the desktop’s tab completion. i.e., when I type “c.” and then click <tab> in the Matlab command prompt, I’d like to see the list of defined “properties” in a tooltip (in the example above: ” asd f @#$^$%&”). Instead, I get the message “No Completions Found.”:

Missing tab completion

I described the hack for desktop tab-completion a few years ago. Unfortunately, that hack only works for functions. We need to find another solution for Matlab class objects.

The solution is to overload the fieldnames() function as well, such that it would return a cell-array of the relevant strings:

classdef ExtendedFieldsClass
    % The internal data implementation is not publicly exposed
    properties (Access = 'protected')
        props = containers.Map;
    end
 
    methods (Access = 'public', Hidden=true)
 
        % Overload property assignment
        function obj = subsasgn(obj, subStruct, value)
            ...  (as above)
        end
 
        % Overload property retrieval (referencing)
        function value = subsref(obj, subStruct)
            ...  (as above)
        end
 
        % Overload fieldnames retrieval
        function names = fieldnames(obj)            names = sort(obj.props.keys);  % return in sorted order        end    end
end

When we now run this in the command prompt, we get the expected behavior:

Working tab completion

R2014a

Unfortunately, this works only up to and including Matlab release R2013b. In R2014a, MathWorks made some internal change that prevents overloading the fieldnames function. To be more precise, we can still overload it as above, and it will indeed work if we directly call fieldnames(c), but it no longer has any effect on the tab completion. On R2014a, the tab-completion remains broken and returns “No Completions Found.” When this was reported to MathWorks some time ago, the official response was that the previous behavior was considered a “bug”, and this was “fixed” in R2014a (don’t bother searching for it in the official bugs parade). Go figure…

So what do you think I should now do? Remember: this is a large client, who knows how many licenses are at stake. Should I suggest to my client not to switch to Matlab? Or should I suggest that they keep using R2013b across the entire organization and cancel their annual maintenance? Or maybe I should simply tell them to accept the fact that some important functionality should be expected to get broken whenever Matlab is upgraded?

These sort of things just blow me away. Sometimes I feel as if I am swimming against the current, and that’s frustrating. I admit it doesn’t happen very often. Then again, I guess if things were straight-forward, nobody would need me to consult them…

Don’t mind me – just blowing off some steam. I’m allowed to, every now and then, aren’t I?

Addendum July 21, 2014: I found out today that on R2014a+ we can simply overload the properties method. This is a function that returns the properties of a class, and so it makes perfect sense for a class object’s tab-completion to use properties rather than fieldnames. So I can now indeed see why the past behavior was considered by MathWorks to be a bug that should be fixed. Still, it would have been nice if for backward-compatibility considerations, Matlab (or at least mlint) would have detected the fact that fieldnames is being overloaded in a user class and warned/alerted regarding the fact that we should now overload properties. In any case, to be fully backward compatible, simply overload both methods, and make one of them call the other. For example:

        % Overload property names retrieval
        function names = properties(obj)
            names = fieldnames(obj);
        end

My client would be quite happy to hear of this new development

Related odds and ends

Michal Kutil described a mechanism for overloading the methods function, which is also part of the tab-completion tooltip. The problem here is that we cannot simply overload methods in our class, since Matlab calls methods with the class name (not the class object reference) when it wants to determine the relevant methods to display in the tooltip. Michal’s solution was to create a wrapper function that calls the overloaded variant. This wrapper function can then be placed within a @char folder somewhere in the Matlab path. I used a similar trick for my profile_history utility last month.

Related newsgroup posts by Eric Salemi here and here.

Similarly, in order to overload the data-value tooltip (when hovering over the object in the editor), or when displaying the object in the Matlab command prompt, simply overload the disp() function (see related):

        % Overload class object display
        function disp(obj)
            disp([obj.props.keys', obj.props.values']);  % display as a cell-array
        end

In a related matter, we can limit the values that a property can accept using the matlab.system.StringSet class of the matlab.System package, as recently discovered by Oleg Komarov (additional details; a different way to constrict property data type):

classdef foo < matlab.System
    properties
        Coordinates
    end
    properties(Hidden,Transient)
        CoordinatesSet = matlab.system.StringSet({'north','south','east','west'});
    end
end

This blog will now take a short vacation for a few weeks, due to my U.S. trip. I will return with some fresh material in August – stay tuned!

A short while ago, Carsten Allefeld (a regular on this blog) alerted me to the fact that recent Matlab releases have added yet another undocumented feature, this time the ability to specify orange formatting to the output text. This made sense: after all, recent releases have started to display warnings as orange text on STDOUT. While errors have for ages been rendered red by the simple act of directing their output to STDERR, warnings are output to STDOUT and so a bypass was naturally devised. This is similar to the <strong> bypass that I explained last year, for the bold formatting. By redirecting STDOUT, Carsten was able to see that by adding '[\b...]\b' to the output text, anything between the brackets was given an orange color. Of course, we can still use cprintf to display any color in the rainbow, orange included:

Orange color in Command Window text

The \b hack relies on the fact that \b is actually the standard back-space control-character (BS, or ASCII 8), which erases the preceding [ (bracket) character. Matlab’s implementation of fprintf simply has a back-door that switches to orange color when it sees a [\b – ]\b couple. By the way, it won’t work if you use any other combination, or if there are not exactly two such in the displayed text. MathWorks made the hack very specific so that it would be close to impossible to come by it by mistake.

Since ‘\b’ translates into the BS character, we can directly use char(8), as follows:

disp(['this is [' 8 'orange]' 8 ' text'])

The [\b sequence is reminiscent of the CSI ANSI sequence used in days of yore, on non-graphic consoles. In those old days, we used the ESC-[ sequence to add text formatting to the console text. Perhaps some MathWorker was nostalgic. Note the related FEX utility tcprintf, which implements ANSI sequences for color-coding the Matlab console output.

<editorial>
If you ask me, this is an example of bad design: instead of modifying fprintf to enable generic color and text formatting, the developer hard-coded a specific [\b hack for a specific color, and a different specific <strong> hack for bold formatting. If tomorrow a green color or underlined text will be needed, they’ll need to modify the hard-coding and extend it with yet new hard-coding. Moreover, the [\b hack does not enable multiple segments of orange colors in the same text, a natural extension. In their shoes, I’d probably have extended the <strong> paradigm with something like <font color="orange"> or: <span style="color:#FF8000;"> (I wonder where I got this idea from? hmm…) Maybe I’m just being too harsh: it’s obviously easier to bitch from the benches than to score at the field…
</editorial>

Anyway, if you only need black, red and orange colors in your text, you need look no further than fprintf. If, however, you require your outputs to be a bit more lively, then consider using cprintf:

cpintf demo

Do you know of any additional undocumented Command Window hack? If so, please do leave a comment below.

Introduction

I often work with datasets that in one way or another have millions of rows and several columns. Although the dimensions preclude visual-based approaches, e.g. imagine keeping all data on a spreadsheet, especially during the development and testing phases, I need to inspect the dataset at given ‘coordinates’.

As a concrete example, in my work on financial time-series I sometimes encounter a subtle irregularity that might arise from a bug in the code when you stack different series of prices. Then you calculate returns and wonder why you get -99% or +2000%.

Thankfully, the Variables Editor (VE) is very much what I would need for such inspection tasks, if it wasn’t that its usefulness is inversely proportional to the size of the data. An example would better clarify what I mean. Suppose you need to scroll to position 5677545 of your 1e8-by-1 data variable, the following screenshot illustrates how the scrollbar, pgdown/pgup or the arrow keys are not fine-tuned for such task.

Problematic Variables Editor scrolling (and the scrollto solution)

You can easily waste minutes to mindless scrolling!

We could argue that the task is painlessly accomplished through basic logical indexing in the Command Window (CW), as the following image shows (note that I already set the display to format compact):

Sub-indexing into a large data matrix

However, this approach is limited because:

1. it displays a static snapshot of the selected region, and often precludes interaction with the data, e.g. copy-paste
2. it requires more visual space than the VE, which either means that you need to modify your layout or scroll the CW
3. typing in the CW can be even more tedious than scrolling the VE…

Why openvar is not a good-enough solution

In search of a solution that would retain the VE interactivity and would speed up the task of scrolling to the point of interest, I first checked if there was an API for the VE similar to that for the documents Editor. As a side-note: to date, the Editor’s API remains undocumented/unsupported, although having been published in what was then called the Desktop Blog (renamed MATLAB Spoken Here since Ken and Mike have left MathWorks and the blog focus has changed).

In any case, it turns out that there is a stub of an API for the Variables Editor, matlab.desktop.vareditor, which however comes much short of my expectations. In fact, it only goes as far as visualizing the data of some variable within a completely separate basic version of the VE. This could prove to be a valid embeddable alternative to the uitable, but it does not help us with the need to scroll to a specific data cell.

Unlucky with the VE API, I looked into openvar for an undocumented option that would let me open some variable in the VE and then scroll to the desired ‘coordinates’ – this effort proved fruitless.

A description of the solution that did work

Since the standard Matlab did not offer a programmatic solution, I started digging into the underlying Java components.

I found that one simple direct way is to open our variable of interest in the VE with openvar, retrieve the Java Desktop instance (in the same manner as we do to get the Desktop instance to customize the Editor). From this instance we can find the handle to the appropriate com.mathworks.mlwidgets.array.ArrayTable with findjobj (remember that there could be multiple ArrayTables, one for each inspected variable). Finally, we scroll to the desired position using the scrollCellToVisible() method.

Note that the Matlab object builds on the javax.swing.JViewport and provides a convenient interface as in the case of the scrolling function, since its java counterpart scrollRectToVisible() requires as input a java.awt.rectangle object rather than scalar doubles.

A succinct usage example would be:

% Create example variable
a = randn(1e8,1);
name = 'a';
 
% Open 'a' in the VE or grab focus
openvar(name)
 
% Retrieve Desktop instance and handle to client
desktop = com.mathworks.mde.desk.MLDesktop.getInstance;
varclient = desktop.getClient(name);
 
% Retrieve handle to scrollable table with findjobj
jVarTable = findjobj(varclient,'property',{'name','VariableTable'});
 
% Select, scroll and update the UI (note the zero-indexing, i.e. row-1)
row = 5677545;
col = 1;
jVarTable.setRowSelectionInterval(row-1, row-1)
jVarTable.setColumnSelectionInterval(col-1, col-1)
jVarTable.scrollCellToVisible(row-1, col-1)
jVarTable.updateUI

The scrollto utility

As most of the lines of the usage example above should already be familiar and/or self-explanatory to the followers of this blog, the next logical step is to encapsulate the snippet into a function which you can already find on the File Exchange: scrollto.

The scrollto function provides the following features:

• Accepts subs or indices
• You can scroll indexed variables, i.e. ‘a{1}’ or ‘a.field{2,3}’ etc.
• Scrolling is also supported in debug mode (from ver2.00), i.e. on variables of the ‘caller’ workspace
• Handles asynchronous rendering of the VE (see below)

and supports the following classes (should be 2D arrays):

• Numeric and logical data – ‘VariableTable’
• Cell arrays – ‘CellTable’
• Timeseries – ‘TimeSeriesArrayEditorTablePanel:fDataTable’
• Datasets (Statistics Toolbox) – ‘DatasetVariableTable’

Matlab handles different classes in the VE through different interfaces. For this reason, for each supported class I reported the ‘name’ property to use with findjobj.

Synchronization issues

Asynchronous rendering of the VE can happen in either of two cases:

1. if the variable has never been opened in the VE, or the variable was opened but it does not exist in the workspace anymore
2. in code that opens and interacts with the VE

Writing scrollto proved to be more than a simple wrapping effort and it is worth mentioning the workaround implemented to allow a smooth workflow. The biggest issue I faced is the asynchronous rendering of the VE. As Yair reports in his book Undocumented Secrets of Matlab-Java Programming, p. 538:

“The tight-coupling of the Variable Editor to the Desktop Workspace variables is unfortunate in some respects. … Matlab only has a single computational thread, so Matlab code has to finish before the JMI request can be handled. This means that the Variables Editor contents cannot be displayed synchronously by the Matlab code that invokes it.”

In other words, we cannot retrieve the handle to e.g. the VariableTable until the function has finished executing.

A workaround is to call openvar, wait until the focus is back to the CW, and then call scrollto. I cannot tell you how this workflow made me feel so close and yet so far from a satisfactory implementation.

The ideal flowchart of a basic wrapper around the example (see above) would have been:

Ideal scrollto workflow

Now, since we cannot retrieve the handle to the VariableTable if the VE has not rendered yet, I implemented an asynchronous second call to scrollto through a timer object that fires after the first call to scrollto has finished executing:

Asynchronous scrollto workflow

The result is a better experience without unexpected interruptions or breaks in the workflow.

I would like to thank Yair for his comments and guidance. Without his support, scrollto would not exist and humankind would be doomed to scroll the VE manually!

Public-service announcement

(This is Yair again):

I wish to invite you to join my online presentation (webinar) on “A Real-Time Trading System in MATLAB” in the upcoming MATLAB Computational Finance Virtual Conference next Thursday, September 19, 2013. I will be speaking at 2pm EST (8pm CEST). Registration is free and it’s a virtual conference, so there’s no need for a tie and jacket… Following a half-hour presentation, I will be answering audience questions online.

I gave an earlier version of this presentation at the Computational Finance Conference in New York on May 23, and you are welcome to look there for a preview. The presentation slides can be downloaded here. Even if you’re not interested in real-time financial trading with Matlab, you might find it interesting to see the neat things that Matlab can do using a Java API interface and a few undocumented GUI tricks.

Introduction

JGit logo

The first step is to download and install JGit4MATLAB in your Matlab path. The first time that the JGit Matlab class is called, it downloads the latest version of org.eclipse.jgit.jar, adds it to Matlab’s static Java classpath and makes a backup of the existing javaclasspath.txt file as javaclasspath.JGitSaved. Matlab must then be restarted for the changes to the static Java class path to take effect. From now on, you can use the commands in the JGit Matlab class or the jgit.m wrapper function. These use basic JGit commands, which shall be described below:

Most of the important JGit methods are encapsulated in the Matlab JGit wrapper class and jgit.m function, but you can also use them directly if you wish. The Matlab wrapper currently supports ADD, BRANCH, CHECKOUT, CLONE, COMMIT, DIFF, INIT, LOG, MERGE & STATUS, but there are many more in org.eclipse.jgit.api.Git that you can use directly. In fact you don’t really need to use the wrapper at all if you don’t want to – all you really need is the JAR file, which the wrapper downloads and installs for you. You can also download the JAR file independently from here (latest only, which is what JGit4MATLAB does), or here (latest and previous releases). If you’re using JGit4MATLAB, you can update the JAR file to the latest version at will, by issuing JGit.downloadJGitJar('@JGit\org.eclipse.jgit.jar').

The most basic Git commands are clone, init, add and commit. Let us explore how to use JGit’s “Git-Porcelain API” class in Matlab to perform these tasks.

Clone and Init

Clone and Init are both class methods of the Git-Porcelain API class and return a Command API class object, which is what nearly all of the Git-Porcelain methods do:

import org.eclipse.jgit.api.Git   % import the package
cloneCMD = Git.cloneRepository;   % return a CloneCommand object
URI = 'git://github.com/mikofski/JGit4Matlab.git';   % a Git repository
cloneCMD.setURI(URI)   % set the repository to clone
g = cloneCMD.call;   % clone the repository

or more compactly:

g = org.eclipse.jgit.api.Git.cloneRepository.setURI(URI).call;

The call method executes the command with the options and arguments set by the other methods. Calling the clone method also returns a Git instance. The full API for the CloneCommand class lists all the possible options and arguments for cloning. One option lets you set an optional progress monitor; there are some already included in JGit, but unfortunately, they use the carriage return (or ‘\r’) escape sequence which in Windows is the same as a newline (read here). So, I implemented a customized ProgressMonitor class for Matlab that uses backspace (or ‘\b’) instead.

Init is very similar to clone:

g = org.eclipse.jgit.api.Git.init.call;

Configuring author information

Before moving on, it is important to set the “author” and “email” fields in the Git config. JGit will use your computer’s info if you provide nothing. You probably want to set these values globally, so the place to put them is in your $HOME folder in the “.gitconfig” file.

file = java.io.File(fullfile(getenv('home'),'.gitconfig'));
gitconfig = org.eclipse.jgit.storage.file.FileBasedConfig(file, org.eclipse.jgit.util.FS.DETECTED);
gitconfig.setString('user',[], 'name','John Doe');       % place your name in here
gitconfig.setString('user',[], 'email','john@doe.com');  % place your email in here
gitconfig.save;

Hopefully your commits will all have the right name and email address in them from now on.

Typical JGit Workflow

Now that you have a Git repository, you will want to start working with it. The typical workflow is to create a feature branch in which to work. But first JGit needs a Git instance. If you saved it from cloning or initializing a new repo then good for you! But if you need to recreate it, just pass the constructor a File object that points to the .git folder inside your Git repo:

JGit workflow

file = java.io.File(fullfile('path', 'to', 'repo', '.git'));
g = org.eclipse.jgit.api.Git.open(file);
g.checkout.setCreateBranch(true).setName('newbranch').call;

Now you are no longer on the “master” branch, which is usually reserved for buildable/deployable code. Start working. But before you do too much, commit your work to your branch:

g.add.addFilepattern('file1.m').addFilepattern('file2.m').call;
g.commit.setMessage('initial dump').call;

JGit commit log

Everything else is pretty much the same as above: Get a Git instance; create a command; set the options; and finally call the command:

file = java.io.File(fullfile('path', 'to', 'repo', '.git'));
g = org.eclipse.jgit.api.Git.open(file);
r = g.getRepository;  % a Repository instance can be very useful
g.status.call;
g.log.call;
g.checkout.setName('master').call;
g.merge.include(r.resolve('newbranch')).call;
g.push.call;
g.pull.call;

Note: Repository.resolve() is one of the most powerful methods.

The JGit log window on the left can be gotten using glog – the standalone JGit command line log GUI (slightly similar to gitk) that can be downloaded as a binary for systems with /bin/sh from the eclipse site (same place as the jar download).

Some tricks can be gleaned by what little work I’ve done so far on the JGit4MATLAB wrapper on the Matlab File Exchange. I’ll be happy to answer whatever questions I can there, or comments here. I’m also happy to accept pull requests on Github, of course using JGit4MATLAB!

Authentication

You can only use an SSH paired keys to authenticate remote repositories, the keys must **not** have a pass-phrase, they must be in the openSSH format and they must be in $HOME/.ssh. PuTTYgen let’s you do all of this, so it’s really not an issue. Of course you can still use remotes that do not require authentication with no problems.

All commands that deal with remotes and that require authentication, e.g. CLONE, FETCH, PULL, PUSH, will only work

• with SSH
• with *no* passphrase
• with keys in the openSSH format
• with keys and known hosts in $HOME\.ssh

Obviously remotes that do not require authentication will work fine. e.g. public, read-only and local repositories.

SSH is very easy to set up:

1. Download puttygen (Intel x86). Disregard the “Intel x86” business, it doesn’t matter what processor you have or whether your os is 64-bit or 32-bit. Puttygen is a very mature well establish application that is used by many other applications, for example all of the TortoiseXXX scm clients use Putty, Plink and Pageant. Here is the actual download here (click to download then click to install): http://the.earth.li/~sgtatham/putty/latest/x86/puttygen.exe
2. Create a folder called .ssh in your %– USERPROFILE%. On Windows 7 this is C:\Users\ and on XP it is C:\Documents and Settings\. You can do this in Matlab: mkdir(fullfile(getenv('userprofile'), '.ssh')).
3. Create a key using puttygen, this is fairly self explanatory, and kind of fun, but do *not* set a pass-phrase; leave those fields blank!
4. Using puttygen, find the conversion menu tab and export the key in the openSSH format to the .ssh folder that you created earlier.
5. Now copy and paste your public key to your remote repositories (Github, Bitbucket, etc.) as needed. Jsch, which is the ssh client that JGit uses, should now work out of the box.

Unfortunately using SSL (HTTPS) isn’t possible with JGit4MATLAB. Sorry. but hopefully you’ll love the excitement of using ssh key pairs!

Further development

Possible future work on the Matlab-JGit integration includes assigning standard keyboard-shortcuts to common source-control tasks, integration as toolbar/toolstrip icons for easier access, and in the Matlab Editor in general. Only a few days ago, a reader on this blog has shown interest in this.

In the far future, integration with Matlab’s existing source-control system. Anyone wish to pick up the glove?

JGit is covered by the BSD 3-clause license which is very permissive, and should allow MathWorks or anyone for that matter to develop an integrated JGit application for Matlab. There are more packages in JGit than just the porcelain API, including Swing utilities and other utilities that can be used to create rich user interfaces and experiences, but the development would have to be in Java. Developers should look at the developer’s section of the JGit website and use either the Maven or Github repository to set up their forks.

Conclusion

JGit and Matlab are a natural combination. JGit is a mature and thorough open-source project. Most porcelain commands are available without any additional work. There are some arcanae, but the API documentation is sufficient to figure out most things using short high-level code. If you find yourself diving too deep, then scan the docs some more, and you will probably find a shortcut.

The functionality in question is the Profiler’s documented ability to display results in either wall-clock (profile -timer real) or CPU-time (profile -timer cpu) modes. In fact, at least since R2008a and possibly sooner (I’m unsure about earlier releases), at least on Windows (I’m unsure about other platforms), both of these Profiler options provide wall-clock measurements – CPU-time measurements is not available. The irony is that CPU mode is stated in the documentation to be the Profiler’s default mode, and has been this way ever since this option was introduced about a decade ago.

I used a very simple test harness – prepare the following function in test.m:

function test
    pause(2)
end

And now run it in the Profiler:

>> profile on -timer cpu
>> test
>> profile off; profile report

If everything had worked as documented, the Profiler report would show the test function as taking no time at all, or maybe only a few millisecs of overhead. In fact, this is exactly what we see in Matlab release R14 SP3 (7.1, dated 2005):

Correct CPU-time profiling report (R14 SP3)

Unfortunately, on R2008a and later (and possibly also some earlier releases), the same profiling run reports the timing as taking 2 full seconds (=wall-clock time):

Bug: wall-clock timing instead of CPU-time (R2008a+)

You might assume that the built-in pause function has a bug that causes it to churn the CPU, and in that case the Profiler’s report would be correct. However, a quick check in the Task Manager (I actually use Process Explorer that I find much better) proves that this is not the case – no discernible CPU-time is used by the Matlab process during the pause.

Somewhere between R2006a and R2008a a bug was introduced that caused the Profiler to only use wall-clock mode.

Note that even in R14 SP3 the situation is not perfect: I have found that the default mode (i.e., when not specifying the -timer mode explicitly) is sometimes wall-clock and sometimes CPU, with no apparent consistency. However, if we explicitly use the -timer option, it works ok.

It is strange that such a feature has escaped everyone’s attention (the community’s no less than MathWorks’) for so many years. It is true that this is not an often-used functionality, and it is also true that the profile function itself it way under-utilized by the general public. Still, anyone who needs to do serious performance tuning must have used this feature at some point in time. When performance-tuning code, it is very important to be aware of both the wall-clock and the CPU-time of the profiled program. Different tuning strategies apply to these two modes. Relying only on wall-clock timing can lead to incorrect conclusions about the application’s hotspots, and therefore to sub-optimal tuning. So I am very surprised that this bug was not reported until now. I’ve reported it (1-K1I06V) lately, and I really hope that it will be fixed in R2013a.

Profiling built-in functions

Built-in functions (e.g., isempty, length, find) are not monitored or displayed in the profiling summary by default. They are also not linkable to a detailed profiling report in the detailed report of any user function. So, if we have a code line such as the following, we cannot know how much time is spent in length, find as opposed to being spent in userFunction:

numResults = length(find(userFunction()));

However, we can tell the profile to monitor and display built-in functions using the undocumented optional -detail builtin switch of the profile function:

profile on –detail builtin
profile('on','-detail','builtin')   % an equivalent alternative

Alternately, we can use the undocumented built-in function callstats directly (as I already explained, the profile function is simply a convenient wrapper function to callstats):

callstats('level', 2);  % 1=mmex, 2=builtin

The built-in functions will now be reported in the profiling summary and detailed reports, and will be linkable within the code lines of the detailed report.

Note: old MATLAB releases also supported the ‘operator’ detail level, which provided profiling information about built-in operators such as +. However, this level of detail is no longer supported in the latest MATLAB releases.

The -detail switch can also be used with other profile switches. For example:

>> profile('on','-detail','builtin','-timestamp'); surf(peaks); profile('off')

To return to the standard profiling level (‘mmex’), run one of the following:

profile on –detail mmex
profile('on','-detail','mmex')   % an equivalent alternative
callstats('level', 1);           % an equivalent alternative

Profiling overhead removal

The -remove_overhead switch checks the profiler’s own overhead and knows to remove this overhead from all relevant timing values. This is normally off by default, and returns profData.overhead=0. When turned on, it stores the computed profiling overhead in profData.overhead:

profile -remove_overhead on
profile('-remove_overhead','on')        % equivalent alternative
callstats('remove_sample_overhead',1);  % equivalent alternative

We need to make a small fix to the profview function (%matlabroot%/toolbox/matlab/codetools/profview.m), otherwise the profiling report will croak on an error. The fix is simple: replace the following code segment (lines #351-355 in the profview.m file of R2012a)

if profileInfo.Overhead==0
    s{end+1} = sprintf(['<p><a name="selftimedef"></a>', getString(message('MATLAB:profiler:SelfTime1st')) ' ']);
else                        
    s{end+1} = sprintf(['<p><a name="selftimedef"></a>', getString(message('MATLAB:profiler:SelfTime2nd', profileInfo.Overhead))]);
end

with this (changed line highlighted – note the extra num2str):

if profileInfo.Overhead==0
    s{end+1} = sprintf(['<p><a name="selftimedef"></a>', getString(message('MATLAB:profiler:SelfTime1st')) ' ']);
else
    s{end+1} = sprintf(['<p><a name="selftimedef"></a>', getString(message('MATLAB:profiler:SelfTime2nd', num2str(profileInfo.Overhead)))]);end

Turning the overhead switch on causes the message at the bottom of the profiling summary report to change, from something that says:

Self time is the time spent in a function excluding the time spent in its child functions. Self time also includes overhead resulting from the process of profiling.

To this (the numeric value is the computed overhead, also reported in profData.overhead):

Self time is the time spent in a function excluding both the time spent in its child functions and most of the overhead resulting from the process of profiling.
In the present run, self time excludes 0.80601 seconds of profiling overhead. The amount of remaining overhead reflected in self time cannot be determined.

A similar customization needs to be done to fix an error arising from the use of the undocumented profile -timer none option. I don’t really see great value in running a Profiler without timing information, so I’ll skip this part here. If anyone has an idea what this could be used for, I’d be happy to hear.

[As-yet] unsolved mysteries

To conclude my series on undocumented profiling options, a couple of unsolved mysteries:

hardware profiling

The profview.m function has several references to hardware performance counter data, which, if available as fields in profData.FunctionTable, are displayed as separate columns to the left of the time column field in the code section, and also enabled for the sorting and highlighting drop-downs in the detailed profiling report. Separate monitored hardware events are stored in separate fields named profData.FunctionTable.HWevent (see the getHwFields() sub-function on lines 1557-1564). The mystery: how can we tell the Profiler to start hardware monitoring in a way that would add these HW fields to the collected profData.FunctionTable? Lines 1652-1653 give a hint:

hw_events = callstats('hw_events');
match = strcmpi(['hw_' str],hw_events);

Unfortunately, at least on my system, callstats('hw_events') returns an empty cell array:

>> hw_events = callstats('hw_events')
hw_events = 
   Empty cell array: 1-by-0

So if anyone has an idea how to use these HW events, I’d be happy to hear – please leave a comment below.

parallel profiling

Another mystery relates to the question of how to profile parallel Matlab code. The underlying Java object of the Profiler tool (com.mathworks.mde.profiler.Profiler.getInstance) contains the following tell-tale methods:

• getHtmlTextParallel(), setHtmlTextParallel()
• getInstanceWithParallelOpts()
• getSelectedLabsFromHtml()
• setNumLabs()
• setNumLabsParallel()
• setSelectedLab()

I would have expected the parallel-enabled Profiler to be started using com.mathworks.mde.profiler.Profiler.getInstanceWithParallelOpts.invoke(). This does indeed start the Profiler, but I can’t see any difference between its output and that of the normal Profiler, even when I run parallel (parfor) code. It definitely does not look like output from the Parallel Computing Toolbox (PCT)’s mpiprofile function.

Perhaps these methods are used by mpiprofile internally? It could make sense for both profile and mpiprofile to share the same underlying profiler tool. On the other hand, if this were indeed the case then why would there be a reason for a separate mpiprofile function? Wouldn’t the standard profile function be sufficient?

Anyway, if anyone has an idea how to use these methods in the standard profiler, perhaps to profile parallel applications without the need for pmode, I’d be happy to hear – please leave a comment below.

Conclusion

The Matlab Profiler has quite a few important features, which for some unknown reason have not been made public. Matlab’s Profiler has remained stagnant since Matlab 7 was released last decade. In fact, some functionality, like the JIT information feature, was actually removed. Hopefully, future Matlab releases will improve the Profiler, by making these undocumented features officially supported, as well as by adding other important functionality.

This concludes my series on undocumented profiling features in Matlab. If anyone knows any other Profiler tricks, please share them in a comment below.

Simple call history

One of the features that the programmatic profiling interface provides, and which is not represented in the Profiler GUI, is a report of the exact order in which different functions were called during the profiling session. This profiling history can be turned on using the –history switch. There is also an optional switch of –historysize, which enables us to modify the history size from a default maximum of 1 million function entry and exit items. Here is a sample usage of this history feature:

>> profile('on','-history'); surf(peaks); profile('off')
>> profData = profile('info');
>> history = profData.FunctionHistory
history =
  Columns 1 through 11
     0     0     0     1     0     1     0     1     1     1     0     
    19     1    17    17    17    17    18    18     1    19     5    
  ...

The history data is actually a numeric matrix, where the first row contains the values 0 (=function entry) or 1 (=function exit), and the second row is the corresponding index into profData.FunctionTable, indicating the called function. We can easily convert this matrix into human-readable form using the following code snippet:

offset = cumsum(1-2*history(1,:)) - 1;  % calling depth
entryIdx = history(1,:)==1;     % history items of function entries
funcIdx = history(2,entryIdx);  % indexes of relevant functions
funcNames = {profData.FunctionTable(funcIdx).FunctionName};
for idx = 1: length(funcNames);
   disp([repmat(' ',1,offset(idx)) funcNames{idx}]);
end

which generates the following calling list in the MATLAB Command Window:

isempty
 isempty
  transpose
 meshgrid
  peaks
 nargchk
  error
 ishg2parent
...

Unfortunately, the history information does not by default contain specific timing of each function entry/exit, but we can still make good use of it by looking at the sequence in which the functions were called from each other.

Detailed call history, with timing information

In order to retrieve actual history timing information, we can run profile with the undocumented/unsupported –timestamp switch, which stores the CPU clock next to the history information. The reported history matrix now has 4 rows rather than 2, where the extra rows represents the timestamp of each function entry/exit:

>> profile('on','-timestamp'); surf(peaks); profile('off')
>> profData = profile('info');
>> profData.FunctionHistory(:,1:3)
ans =
                0                0                1
                1                2                2
       1347473710       1347473710       1347473710
           453000           453000           468000

In this report, the 3rd row represents the timestamp in seconds, while the 4th row represents the fractional portion of the timestamp, in microseconds. In the example above, the first timestamp item corresponds to 1347473710.453 seconds.

The seconds value appears to be related to the number of seconds since midnight Jan 1, 1970 (so-called Epoch), which is a standard time representation in computing systems. However, the actual value appears to be off by slightly over a day from the expected value (which is retrievable via getTime(java.util.Date)) for some unknown reason. Since we are only interested in relative, rather than absolute times when profiling, this minor difference does not affect us.

The timeline of the profiling session can be visualized as follows:

histData = profData.FunctionHistory;
startTime     = histData(3,1) + histData(4,1)/1e6;
relativeTimes = histData(3,:) + histData(4,:)/1e6 - startTime;
plot(relativeTimes);

Profiling history timeline

This report helps us see that a particular set of function calls, around the 100th call mark, is responsible for about 0.5 seconds, a prime candidate for tuning investigation. If we only relied on the standard profiling report we might have missed this because it might have been meshed into the same “bucket” as other invocations of the same function. As illustration, take the following simulated example:

Invocation  #1 of func():  0.500 secs
Invocation  #2 of func():  0.013 secs
Invocation  #3 of func():  0.011 secs
   ...
Invocation #10 of func():  0.012 secs
_____________________________________
Total invocation time:     0.600 secs

In this simulation, we would not have known that the 0.6 secs invocation time of func() is not really evenly distributed across all 10 invocations, and this would lead us to incorrect conclusions. For example, we could spend time unnecessarily on tuning the steady-state run-time performance of the function, whereas we should really have concentrated on only the first invocation. By looking at the actual timestamps we could see the large run-time used by the first invocation and this information can possibly be used to tune this first invocation and significantly reduce the overall time taken by the function.

Note: instead of using profile -timestamp, we could also have used the undocumented built-in function callstats, which is the underlying profiling engine (the profile function is simply a convenient wrapper function to callstats – take a look within profile.m):

callstats('history',2);  % 0 = -nohistory, 1 = -history, 2 = -timestamp

History collection has an overhead, so if you don’t need it then you should turn it off:

profile -nohistory
profile('-nohistory')    % equivalent alternative
callstats('history',0);  % equivalent alternative

Addendum (June 16th, 2014): I have created a utility (profile_history, available on the Matlab File Exchange) that parses all this profile data and presents it in an interactive GUI. See this article for details.

Function call timeline profiling (click for full-size image)

Fixing a Profiler font problem

On my system, starting in R2012a, the code lines in the detailed profiling report look awful. The reason is that they are rendered using a proportional font, rather than the monospaced (fixed-width) font that is specified in the Matlab Preferences and used everywhere else (the Editor, Command Window etc.):

Before: ugly (proportional-font) profiling report

I have seen this phenomenon on several of my clients’ systems, so I’m pretty sure that this is not an isolated issue. However, I do not know whether this problem is wide-spread.

In any case, the solution is very simple: Edit the %matlabroot%/toolbox/matlab/codetools/matlab-report-styles.css files and replace the following CSS directives near the top of the file:

PRE {
  font-size: 100%;
}

with this one:

pre, tt {
  font-family: monospace
  font-size: 100%;
}

After: monospaced profiling report
(note the 1-millisecond timing resolution)

A very similar problem seems to afflict the Help browser. I could not add the CSS fix there because the help pages use the docstyle.css files that are bundled within the \help\techdoc\help.jar file. If you wish, you could update them there (a JAR is a simple zip file after all).

You may perhaps recall my related article last year, where I showed how to convert a proportional-font multi-line tooltip into a nicely-formatted monospaced tooltip.

Increasing the reported timing resolution

The Profiler records information at a system-dependent time resolution, typically 1 millisecond. However, the profiling report only displays information at a granularity of 10 ms. I often find it useful to increase the reported resolution to 1 ms, and in fact the screenshots above reflect this change. This can easily be done by editing the profview function (%matlabroot%/toolbox/matlab/codetools/profview.m. This function can be edited in the MATLAB editor via edit(‘profview’)).

The following is from lines 1410-1418 of profview.m in R2012a:

% Display the time
if timePerLine > 0.01,
    s{end+1} = sprintf('<span style="color: #FF0000"> %5.2f </span>',...
        timePerLine);
elseif timePerLine > 0
    s{end+1} = '<span style="color: #FF0000">&lt; 0.01 </span>';
else
    s{end+1} = '       ';
end

which could be replaced by the following (modified lines highlighted):

if timePerLine >= 0.005,    s{end+1} = sprintf('<span style="color: #FF0000"> %6.3f</span>',...        timePerLine);
elseif timePerLine > 0
    s{end+1} = '<span style="color: #FF0000">&lt; 0.005</span>';else
    s{end+1} = '       ';
end