Variables Editor scrolling

I would like to introduce guest blogger Oleg Komarov. Oleg has gathered a reputation as a Matlab veteran with plenty of insight into the arcane corners of Matlab, originally on the CSSM newsgroup, and more recently on StackExchange (which admittedly contains more Q&As of the challenging type). Today Oleg discusses a non-trivial solution to a deceivingly-simple problem.

Introduction

I often work with datasets that in one way or another have millions of rows and several columns. Although the dimensions preclude visual-based approaches, e.g. imagine keeping all data on a spreadsheet, especially during the development and testing phases, I need to inspect the dataset at given ‘coordinates’.
As a concrete example, in my work on financial time-series I sometimes encounter a subtle irregularity that might arise from a bug in the code when you stack different series of prices. Then you calculate returns and wonder why you get -99% or +2000%.
Thankfully, the Variables Editor (VE) is very much what I would need for such inspection tasks, if it wasn’t that its usefulness is inversely proportional to the size of the data. An example would better clarify what I mean. Suppose you need to scroll to position 5677545 of your 1e8-by-1 data variable, the following screenshot illustrates how the scrollbar, pgdown/pgup or the arrow keys are not fine-tuned for such task.

1. it displays a static snapshot of the selected region, and often precludes interaction with the data, e.g. copy-paste
2. it requires more visual space than the VE, which either means that you need to modify your layout or scroll the CW
3. typing in the CW can be even more tedious than scrolling the VE…

Why openvar is not a good-enough solution

In search of a solution that would retain the VE interactivity and would speed up the task of scrolling to the point of interest, I first checked if there was an API for the VE similar to that for the documents Editor. As a side-note: to date, the Editor’s API remains undocumented/unsupported, although having been published in what was then called the Desktop Blog (renamed MATLAB Spoken Here since Ken and Mike have left MathWorks and the blog focus has changed).
In any case, it turns out that there is a stub of an API for the Variables Editor, matlab.desktop.vareditor, which however comes much short of my expectations. In fact, it only goes as far as visualizing the data of some variable within a completely separate basic version of the VE. This could prove to be a valid embeddable alternative to the uitable, but it does not help us with the need to scroll to a specific data cell.
Unlucky with the VE API, I looked into openvar for an undocumented option that would let me open some variable in the VE and then scroll to the desired ‘coordinates’ – this effort proved fruitless.

A description of the solution that did work

Since the standard Matlab did not offer a programmatic solution, I started digging into the underlying Java components.
I found that one simple direct way is to open our variable of interest in the VE with openvar, retrieve the Java Desktop instance (in the same manner as we do to get the Desktop instance to customize the Editor). From this instance we can find the handle to the appropriate com.mathworks.mlwidgets.array.ArrayTable with findjobj (remember that there could be multiple ArrayTables, one for each inspected variable). Finally, we scroll to the desired position using the scrollCellToVisible() method.
Note that the Matlab object builds on the javax.swing.JViewport and provides a convenient interface as in the case of the scrolling function, since its java counterpart scrollRectToVisible() requires as input a java.awt.rectangle object rather than scalar doubles.
A succinct usage example would be:

% Create example variable
a = randn(1e8,1);
name = 'a';
% Open 'a' in the VE or grab focus
openvar(name)
% Retrieve Desktop instance and handle to client
desktop = com.mathworks.mde.desk.MLDesktop.getInstance;
varclient = desktop.getClient(name);
% Retrieve handle to scrollable table with findjobj
jVarTable = findjobj(varclient,'property',{'name','VariableTable'});
% Select, scroll and update the UI (note the zero-indexing, i.e. row-1)
row = 5677545;
col = 1;
jVarTable.setRowSelectionInterval(row-1, row-1)
jVarTable.setColumnSelectionInterval(col-1, col-1)
jVarTable.scrollCellToVisible(row-1, col-1)
jVarTable.updateUI

The scrollto utility

As most of the lines of the usage example above should already be familiar and/or self-explanatory to the followers of this blog, the next logical step is to encapsulate the snippet into a function which you can already find on the File Exchange: scrollto.
The scrollto function provides the following features:

• Accepts subs or indices
• You can scroll indexed variables, i.e. ‘a{1}’ or ‘a.field{2,3}’ etc.
• Scrolling is also supported in debug mode (from ver2.00), i.e. on variables of the ‘caller’ workspace
• Handles asynchronous rendering of the VE (see below)

and supports the following classes (should be 2D arrays):

• Numeric and logical data – ‘VariableTable’
• Cell arrays – ‘CellTable’
• Timeseries – ‘TimeSeriesArrayEditorTablePanel:fDataTable’
• Datasets (Statistics Toolbox) – ‘DatasetVariableTable’

Matlab handles different classes in the VE through different interfaces. For this reason, for each supported class I reported the ‘name’ property to use with findjobj.

Synchronization issues

Asynchronous rendering of the VE can happen in either of two cases:

1. if the variable has never been opened in the VE, or the variable was opened but it does not exist in the workspace anymore
2. in code that opens and interacts with the VE

Writing scrollto proved to be more than a simple wrapping effort and it is worth mentioning the workaround implemented to allow a smooth workflow. The biggest issue I faced is the asynchronous rendering of the VE. As Yair reports in his book Undocumented Secrets of Matlab-Java Programming, p. 538:

“The tight-coupling of the Variable Editor to the Desktop Workspace variables is unfortunate in some respects. … Matlab only has a single computational thread, so Matlab code has to finish before the JMI request can be handled. This means that the Variables Editor contents cannot be displayed synchronously by the Matlab code that invokes it.”

In other words, we cannot retrieve the handle to e.g. the VariableTable until the function has finished executing.
A workaround is to call openvar, wait until the focus is back to the CW, and then call scrollto. I cannot tell you how this workflow made me feel so close and yet so far from a satisfactory implementation.
The ideal flowchart of a basic wrapper around the example (see above) would have been:

Public-service announcement

(This is Yair again):

I gave an earlier version of this presentation at the Computational Finance Conference in New York on May 23, and you are welcome to look there for a preview. The presentation slides can be downloaded here. Even if you’re not interested in real-time financial trading with Matlab, you might find it interesting to see the neat things that Matlab can do using a Java API interface and a few undocumented GUI tricks.