GUI automation utilities

Once again, I welcome guest writer Takeshi (Kesh) Ikuma. Last week, Kesh described how Java’s Robot class can be used to automate Matlab GUI. Today, Kesh will describe two Matlab utilities that help this automation process.

jMouseEmu

The jMouseEmu utility is an interface function to simplify the programmatic execution of mouse actions (movements, clicks and wheel scrolls). jmouseemu features include accepting relative position with regards to a specified Handle Graphics control, supporting the Shift and Control click modifier keys, supporting double clicking and mouse dragging, controlling mouse wheel turns and allowing multiple command execution with interval timing specification.
jmouseemu has two operating modes: single- or multi-command. In this respect, a single jmouseemu command consists of a cursor position and a click option – multiple such commands can be specified to jmouseemu in the multi-command mode.
In the single-command mode, the function takes the cursor position information and click options. The click options can be any of the possible SelectionType property strings, or “none” if no clicking is desired. For example,

jmouseemu([500,250],'normal');

places the mouse cursor at the coordinate (500, 250) with respect to the current figure (in Matlab coordinates) and then clicks the left mouse button. If no figure is currently open, the cursor is placed at (500,250) with respect to the bottom-left corner of the screen.
The reference graphics handle can be added as the first argument. For example, if hEdit is a uicontrol editbox handle, then

jmouseemu(hEdit,[],'open');
jmouseemu(hEdit,[],'normal');

provides a simple way to select the entire text in the edit box: the double-click jmouseemu call selects the first word, then the following single-click jmouseemu extends the selection to the entire text. If the position argument omitted, the cursor is placed on the upper-left hand corner of the object, unless hEdit=0, in which case the cursor stays at the current position.
To turn the mouse wheel 600 notches down (towards you), run:

jmouseemu(-600,'wheel');

Multi-command mode extends the single-command mode by accepting a cell array of commands along with the update interval T in seconds. The command array must have three columns, and its k-th row specifies the mouse command at time (k–1)T second.
Using the multi-command mode, the above example can be executed with a single jmouseemu call, placing the commands in a cell array:

jmouseemu({h,[],'open'; [],[],'normal'},0);  % T=0

Here, if you omit handles (i.e., the first column of command array) for the second or later commands, it defaults to the handle of the previous command.
Multi-command mode enables mouse cursor to follow a predefined path. For example, to make the mouse cursor travel in a full circle:

T = 0.01; % update every 10 milliseconds
x = pi * (0:T:2)';
pos = 100*[cos(x),sin(x)] + 250;
N = numel(x);
cmds = cell(N,3); % empty click option -> no click
cmds{1,1} = gcf; % position w.r.t. current figure
cmds(:,2) = mat2cell(pos,ones(N,1),2);
jmouseemu(cmds,T);

The command interval is realized by calling robot.delay(T*1e3) between commands. This example updates the mouse position every 10 msecs and completes the circle in 2 seconds.
In multi-command mode, there are two additional click options, “drag_on” and “drag_off”, available to enable mouse dragging. Dragging must be initiated and terminated within a single jmouseemu call, and “drag_off” and “none” are the only two acceptable click options during dragging.
Lastly, to pause mouse movement for a moment (longer than the update interval) , a “delay” command may be used:

jmouseemu({...; [],1.0,'delay'; ...}, T);  % pause for 1 second

Using jMouseEmu to trigger multiple Matlab HG callbacks

jmouseemu (or more broadly the Matlab handling of Java Robot commands) has a major drawback: If jmouseemu is used to trigger a Matlab GUI callback during a sequence of mouse maneuver, the callback will not be executed until the entire maneuver is completed. Moreover, since SelectionType property is not logged for every callback event, the incorrect SelectionType property may be reported during callback execution.
For example, add a double click at the beginning of the circular movement:

cmds{:,3} = 'open';
jmouseemu(cmds,T);

Running this code, you can observe the figure’s WindowsButtonDownFcn callback, which has been set to echo SelectionType property from earlier, does not kick in until the circle is completed. Now, introduce “extend” click at the end of the circle:

cmds{end,3} = 'extend';
jmouseemu(cmds,T);

Again, no callback displaying SelectionType echo occurs during the mouse movement. When the callbacks are finally executed, they return in the order – “normal”, “extend”, “open” – instead of expected “normal”, “open”, “extend”. To correct this behavior, jmouseemu must be called twice:

jmouseemu(cmds(1,:),T);
jmouseemu(cmds(2:end,:),T);

The key here is for each jmouseemu call to have only one callback triggering mouse click command and to place it at the end of the command sequence.

inputEmu

The inputEmu utility is similar to jMouseEmu, with extended support for keyboard inputs. The support for relative mouse cursor position with respect to an HG object, however, has been turned off in the current version of inputEmu (it may get turned on in a future version). Position information in inputEmu is always given in pixels with respect to the lower-left corner of the main display screen. Also, the user must note that the input arguments are flipped between jMouseEmu and inputEmu.
In addition to the left mouse click commands from jMouseEmu, inputEmu can emulate the right and middle mouse button clicks. For example, to perform the right click (e.g., to open a context menu):

inputemu({'right_down' []; 'right_up' []}'); % execute right button click

inputEmu’s main feature is keyboard support. There are 4 basic commands (the behavior may be somewhat different on non-windows platforms):

• key_normal – Normal key press
• key_ctrl – CONTROL + key press
• key_alt – ALT + key press
• key_win – WIN + key press

One other obvious key press, SHIFT + key press, is conspicuously missing from this list. This is because the SHIFT key is automatically pressed when a SHIFT modified character (e.g., an upper-case alphabet) is entered. For example, executing

inputemu('key_normal','Hello World'); % emulates typing "Hello World"

directly in Matlab prompt displays “Hello World” in the next command line. The text is first decoded to a sequence of key press and release pairs, including SHIFT key presses and releases as needed, then the sequence of key commands are sent to the java Robot object.
In addition to all the characters on the keyboard, non-character keys can be stroked by using escape characters:

• ‘\SHIFT’
• ‘\CTRL’
• ‘\ALT’
• ‘\WINDOWS’
• ‘\BACKSPACE’
• ‘\TAB’
• ‘\ENTER’
• ‘\ESC’
• ‘\PAGEUP’
• ‘\PAGEDOWN’
• ‘\END’
• ‘\HOME’
• ‘\LEFT’
• ‘\UP’
• ‘\RIGHT’
• ‘\DOWN’
• ‘\INSERT’
• ‘\DELETE’
• ‘\CAPSLOCK’
• ‘\NUMLOCK’
• ‘\SCROLLLOCK’
• ‘\PRINTSCREEN’
• ‘\PA– USE’

Also, use ‘\\’ for the forward slash character and ‘\F1’ – ‘\F12’ for the Function keys. For example, to execute a Matlab “ver” command automatically via command window, run

inputemu('key_normal','ver\ENTER'); % runs the ver command

Lastly, as mentioned in the previous article, java.awt.Robot is sensitive to the keyboard locale, and the inputEmu utility is programmed specifically for the US QWERTY keyboard. To use the inputEmu for another keyboard layout, the program has to be modified accordingly. If anyone’s interested using this utility for different keyboard layout, I’d be glad to provide assistance.
Do you use Java’s Robot class or any of the utilities above in your work? Yair and I would love to hear about your experience in a short comment below.