Once again, I welcome guest writer Takeshi (Kesh) Ikuma. Last week, Kesh described how Java’s Robot class can be used to automate Matlab GUI. Today, Kesh will describe two Matlab utilities that help this automation process.
jMouseEmu and inputEmu are both available on the Matlab File Exchange. There is a large overlap between these utilities. As their name suggests, jMouseEmu is geared to control the mouse, while inputEmu emulates both mouse and keyboard inputs.
jMouseEmu
The jMouseEmu utility is an interface function to simplify the programmatic execution of mouse actions (movements, clicks and wheel scrolls). jmouseemu features include accepting relative position with regards to a specified Handle Graphics control, supporting the Shift and Control click modifier keys, supporting double clicking and mouse dragging, controlling mouse wheel turns and allowing multiple command execution with interval timing specification.
jmouseemu has two operating modes: single- or multi-command. In this respect, a single jmouseemu command consists of a cursor position and a click option – multiple such commands can be specified to jmouseemu in the multi-command mode.
In the single-command mode, the function takes the cursor position information and click options. The click options can be any of the possible SelectionType property strings, or “none” if no clicking is desired. For example,
jmouseemu([500,250],'normal'); |
places the mouse cursor at the coordinate (500, 250) with respect to the current figure (in Matlab coordinates) and then clicks the left mouse button. If no figure is currently open, the cursor is placed at (500,250) with respect to the bottom-left corner of the screen.
The reference graphics handle can be added as the first argument. For example, if hEdit is a uicontrol editbox handle, then
jmouseemu(hEdit,[],'open'); jmouseemu(hEdit,[],'normal'); |
provides a simple way to select the entire text in the edit box: the double-click jmouseemu call selects the first word, then the following single-click jmouseemu extends the selection to the entire text. If the position argument omitted, the cursor is placed on the upper-left hand corner of the object, unless hEdit=0, in which case the cursor stays at the current position.
To turn the mouse wheel 600 notches down (towards you), run:
jmouseemu(-600,'wheel'); |
Multi-command mode extends the single-command mode by accepting a cell array of commands along with the update interval T in seconds. The command array must have three columns, and its k-th row specifies the mouse command at time (k–1)T second.
Using the multi-command mode, the above example can be executed with a single jmouseemu call, placing the commands in a cell array:
jmouseemu({h,[],'open'; [],[],'normal'},0); % T=0 |
Here, if you omit handles (i.e., the first column of command array) for the second or later commands, it defaults to the handle of the previous command.
Multi-command mode enables mouse cursor to follow a predefined path. For example, to make the mouse cursor travel in a full circle:
T = 0.01; % update every 10 milliseconds x = pi * (0:T:2)'; pos = 100*[cos(x),sin(x)] + 250; N = numel(x); cmds = cell(N,3); % empty click option -> no click cmds{1,1} = gcf; % position w.r.t. current figure cmds(:,2) = mat2cell(pos,ones(N,1),2); jmouseemu(cmds,T); |
The command interval is realized by calling robot.delay(T*1e3) between commands. This example updates the mouse position every 10 msecs and completes the circle in 2 seconds.
In multi-command mode, there are two additional click options, “drag_on” and “drag_off”, available to enable mouse dragging. Dragging must be initiated and terminated within a single jmouseemu call, and “drag_off” and “none” are the only two acceptable click options during dragging.
Lastly, to pause mouse movement for a moment (longer than the update interval) , a “delay” command may be used:
jmouseemu({...; [],1.0,'delay'; ...}, T); % pause for 1 second |
Using jMouseEmu to trigger multiple Matlab HG callbacks
jmouseemu (or more broadly the Matlab handling of Java Robot commands) has a major drawback: If jmouseemu is used to trigger a Matlab GUI callback during a sequence of mouse maneuver, the callback will not be executed until the entire maneuver is completed. Moreover, since SelectionType property is not logged for every callback event, the incorrect SelectionType property may be reported during callback execution.
For example, add a double click at the beginning of the circular movement:
cmds{:,3} = 'open'; jmouseemu(cmds,T); |
Running this code, you can observe the figure’s WindowsButtonDownFcn callback, which has been set to echo SelectionType property from earlier, does not kick in until the circle is completed. Now, introduce “extend” click at the end of the circle:
cmds{end,3} = 'extend'; jmouseemu(cmds,T); |
Again, no callback displaying SelectionType echo occurs during the mouse movement. When the callbacks are finally executed, they return in the order – “normal”, “extend”, “open” – instead of expected “normal”, “open”, “extend”. To correct this behavior, jmouseemu must be called twice:
jmouseemu(cmds(1,:),T); jmouseemu(cmds(2:end,:),T); |
The key here is for each jmouseemu call to have only one callback triggering mouse click command and to place it at the end of the command sequence.
inputEmu
The inputEmu utility is similar to jMouseEmu, with extended support for keyboard inputs. The support for relative mouse cursor position with respect to an HG object, however, has been turned off in the current version of inputEmu (it may get turned on in a future version). Position information in inputEmu is always given in pixels with respect to the lower-left corner of the main display screen. Also, the user must note that the input arguments are flipped between jMouseEmu and inputEmu.
In addition to the left mouse click commands from jMouseEmu, inputEmu can emulate the right and middle mouse button clicks. For example, to perform the right click (e.g., to open a context menu):
inputemu({'right_down' []; 'right_up' []}'); % execute right button click |
inputEmu’s main feature is keyboard support. There are 4 basic commands (the behavior may be somewhat different on non-windows platforms):
- key_normal – Normal key press
- key_ctrl – CONTROL + key press
- key_alt – ALT + key press
- key_win – WIN + key press
One other obvious key press, SHIFT + key press, is conspicuously missing from this list. This is because the SHIFT key is automatically pressed when a SHIFT modified character (e.g., an upper-case alphabet) is entered. For example, executing
inputemu('key_normal','Hello World'); % emulates typing "Hello World" |
directly in Matlab prompt displays “Hello World” in the next command line. The text is first decoded to a sequence of key press and release pairs, including SHIFT key presses and releases as needed, then the sequence of key commands are sent to the java Robot object.
In addition to all the characters on the keyboard, non-character keys can be stroked by using escape characters:
- ‘\SHIFT’
- ‘\CTRL’
- ‘\ALT’
- ‘\WINDOWS’
- ‘\BACKSPACE’
- ‘\TAB’
- ‘\ENTER’
- ‘\ESC’
- ‘\PAGEUP’
- ‘\PAGEDOWN’
- ‘\END’
- ‘\HOME’
- ‘\LEFT’
- ‘\UP’
- ‘\RIGHT’
- ‘\DOWN’
- ‘\INSERT’
- ‘\DELETE’
- ‘\CAPSLOCK’
- ‘\NUMLOCK’
- ‘\SCROLLLOCK’
- ‘\PRINTSCREEN’
- ‘\PA– USE’
Also, use ‘\\’ for the forward slash character and ‘\F1’ – ‘\F12’ for the Function keys. For example, to execute a Matlab “ver” command automatically via command window, run
inputemu('key_normal','ver\ENTER'); % runs the ver command |
Lastly, as mentioned in the previous article, java.awt.Robot is sensitive to the keyboard locale, and the inputEmu utility is programmed specifically for the US QWERTY keyboard. To use the inputEmu for another keyboard layout, the program has to be modified accordingly. If anyone’s interested using this utility for different keyboard layout, I’d be glad to provide assistance.
Do you use Java’s Robot class or any of the utilities above in your work? Yair and I would love to hear about your experience in a short comment below.
Hey! Im using the inputEmu and its working great, although I had to modify it slightly since my keyboard isnt US standard. I have a quick question- do you know how to make the keyboard type the content of a variable you have set?
For instance:
variable= test 1
inputemuu(‘key_normal’,variable)
I would like this to type “test 1”
Any help would be appreciated 🙂
[…] Java Robot in two past articles on this blog, where guest blogger Kesh Ikuma explained (here and here) how it can be used to simulate mouse and keyboard actions programmatically. […]
This feels like a horrible abuse of everything, but I wrapped my project in a nice GUI so it’s easy to experiment. But now I need a systematic test so I can make figures for the paper. I need them yesterday of course, and the important part of the code is REALLY slow. So I’m writing a script using jMouseEmu and inputEmu, so that the robot can operate the GUI while I sleep. Here’s hoping it works after my laptop screen falls asleep!
Hey!
its a wonderful invention. I have been using it for my research work.
I have one question. “Is there any way to find current location of TAB cursor ???”
@Hari – you are not making any sense. What do you mean by “Tab cursor”???
As we press TAB button on keyboard, the control moves from one icon to next. the “TAB Cursor” term is being used for this control. As there are some means to find current location of mouse pointer, there might be some commands to find current position of TAB Cursor. i want to know about those commands.
thank u!
@Hari – there is no technical term called “TAB cursor”. You probably mean “focus”. You can check whether a certain component is currently in focus using the isFocusOwner() method of the underlying Java component (which you can find using the findjobj utility. For example: