Undocumented Matlab
  • SERVICES
    • Consulting
    • Development
    • Training
    • Gallery
    • Testimonials
  • PRODUCTS
    • IQML: IQFeed-Matlab connector
    • IB-Matlab: InteractiveBrokers-Matlab connector
    • EODML: EODHistoricalData-Matlab connector
    • Webinars
  • BOOKS
    • Secrets of MATLAB-Java Programming
    • Accelerating MATLAB Performance
    • MATLAB Succinctly
  • ARTICLES
  • ABOUT
    • Policies
  • CONTACT
  • SERVICES
    • Consulting
    • Development
    • Training
    • Gallery
    • Testimonials
  • PRODUCTS
    • IQML: IQFeed-Matlab connector
    • IB-Matlab: InteractiveBrokers-Matlab connector
    • EODML: EODHistoricalData-Matlab connector
    • Webinars
  • BOOKS
    • Secrets of MATLAB-Java Programming
    • Accelerating MATLAB Performance
    • MATLAB Succinctly
  • ARTICLES
  • ABOUT
    • Policies
  • CONTACT

GUI automation utilities

September 22, 2010 7 Comments

Once again, I welcome guest writer Takeshi (Kesh) Ikuma. Last week, Kesh described how Java’s Robot class can be used to automate Matlab GUI. Today, Kesh will describe two Matlab utilities that help this automation process.


jMouseEmu and inputEmu are both available on the Matlab File Exchange. There is a large overlap between these utilities. As their name suggests, jMouseEmu is geared to control the mouse, while inputEmu emulates both mouse and keyboard inputs.

jMouseEmu

The jMouseEmu utility is an interface function to simplify the programmatic execution of mouse actions (movements, clicks and wheel scrolls). jmouseemu features include accepting relative position with regards to a specified Handle Graphics control, supporting the Shift and Control click modifier keys, supporting double clicking and mouse dragging, controlling mouse wheel turns and allowing multiple command execution with interval timing specification.
jmouseemu has two operating modes: single- or multi-command. In this respect, a single jmouseemu command consists of a cursor position and a click option – multiple such commands can be specified to jmouseemu in the multi-command mode.
In the single-command mode, the function takes the cursor position information and click options. The click options can be any of the possible SelectionType property strings, or “none” if no clicking is desired. For example,

jmouseemu([500,250],'normal');

jmouseemu([500,250],'normal');

places the mouse cursor at the coordinate (500, 250) with respect to the current figure (in Matlab coordinates) and then clicks the left mouse button. If no figure is currently open, the cursor is placed at (500,250) with respect to the bottom-left corner of the screen.
The reference graphics handle can be added as the first argument. For example, if hEdit is a uicontrol editbox handle, then

jmouseemu(hEdit,[],'open');
jmouseemu(hEdit,[],'normal');

jmouseemu(hEdit,[],'open'); jmouseemu(hEdit,[],'normal');

provides a simple way to select the entire text in the edit box: the double-click jmouseemu call selects the first word, then the following single-click jmouseemu extends the selection to the entire text. If the position argument omitted, the cursor is placed on the upper-left hand corner of the object, unless hEdit=0, in which case the cursor stays at the current position.
To turn the mouse wheel 600 notches down (towards you), run:

jmouseemu(-600,'wheel');

jmouseemu(-600,'wheel');

Multi-command mode extends the single-command mode by accepting a cell array of commands along with the update interval T in seconds. The command array must have three columns, and its k-th row specifies the mouse command at time (k–1)T second.
Using the multi-command mode, the above example can be executed with a single jmouseemu call, placing the commands in a cell array:

jmouseemu({h,[],'open'; [],[],'normal'},0);  % T=0

jmouseemu({h,[],'open'; [],[],'normal'},0); % T=0

Here, if you omit handles (i.e., the first column of command array) for the second or later commands, it defaults to the handle of the previous command.
Multi-command mode enables mouse cursor to follow a predefined path. For example, to make the mouse cursor travel in a full circle:

T = 0.01; % update every 10 milliseconds
x = pi * (0:T:2)';
pos = 100*[cos(x),sin(x)] + 250;
N = numel(x);
cmds = cell(N,3); % empty click option -> no click
cmds{1,1} = gcf; % position w.r.t. current figure
cmds(:,2) = mat2cell(pos,ones(N,1),2);
jmouseemu(cmds,T);

T = 0.01; % update every 10 milliseconds x = pi * (0:T:2)'; pos = 100*[cos(x),sin(x)] + 250; N = numel(x); cmds = cell(N,3); % empty click option -> no click cmds{1,1} = gcf; % position w.r.t. current figure cmds(:,2) = mat2cell(pos,ones(N,1),2); jmouseemu(cmds,T);

The command interval is realized by calling robot.delay(T*1e3) between commands. This example updates the mouse position every 10 msecs and completes the circle in 2 seconds.
In multi-command mode, there are two additional click options, “drag_on” and “drag_off”, available to enable mouse dragging. Dragging must be initiated and terminated within a single jmouseemu call, and “drag_off” and “none” are the only two acceptable click options during dragging.
Lastly, to pause mouse movement for a moment (longer than the update interval) , a “delay” command may be used:

jmouseemu({...; [],1.0,'delay'; ...}, T);  % pause for 1 second

jmouseemu({...; [],1.0,'delay'; ...}, T); % pause for 1 second

Using jMouseEmu to trigger multiple Matlab HG callbacks

jmouseemu (or more broadly the Matlab handling of Java Robot commands) has a major drawback: If jmouseemu is used to trigger a Matlab GUI callback during a sequence of mouse maneuver, the callback will not be executed until the entire maneuver is completed. Moreover, since SelectionType property is not logged for every callback event, the incorrect SelectionType property may be reported during callback execution.
For example, add a double click at the beginning of the circular movement:

cmds{:,3} = 'open';
jmouseemu(cmds,T);

cmds{:,3} = 'open'; jmouseemu(cmds,T);

Running this code, you can observe the figure’s WindowsButtonDownFcn callback, which has been set to echo SelectionType property from earlier, does not kick in until the circle is completed. Now, introduce “extend” click at the end of the circle:

cmds{end,3} = 'extend';
jmouseemu(cmds,T);

cmds{end,3} = 'extend'; jmouseemu(cmds,T);

Again, no callback displaying SelectionType echo occurs during the mouse movement. When the callbacks are finally executed, they return in the order – “normal”, “extend”, “open” – instead of expected “normal”, “open”, “extend”. To correct this behavior, jmouseemu must be called twice:

jmouseemu(cmds(1,:),T);
jmouseemu(cmds(2:end,:),T);

jmouseemu(cmds(1,:),T); jmouseemu(cmds(2:end,:),T);

The key here is for each jmouseemu call to have only one callback triggering mouse click command and to place it at the end of the command sequence.

inputEmu

The inputEmu utility is similar to jMouseEmu, with extended support for keyboard inputs. The support for relative mouse cursor position with respect to an HG object, however, has been turned off in the current version of inputEmu (it may get turned on in a future version). Position information in inputEmu is always given in pixels with respect to the lower-left corner of the main display screen. Also, the user must note that the input arguments are flipped between jMouseEmu and inputEmu.
In addition to the left mouse click commands from jMouseEmu, inputEmu can emulate the right and middle mouse button clicks. For example, to perform the right click (e.g., to open a context menu):

inputemu({'right_down' []; 'right_up' []}'); % execute right button click

inputemu({'right_down' []; 'right_up' []}'); % execute right button click

inputEmu’s main feature is keyboard support. There are 4 basic commands (the behavior may be somewhat different on non-windows platforms):

  • key_normal – Normal key press
  • key_ctrl – CONTROL + key press
  • key_alt – ALT + key press
  • key_win – WIN + key press

One other obvious key press, SHIFT + key press, is conspicuously missing from this list. This is because the SHIFT key is automatically pressed when a SHIFT modified character (e.g., an upper-case alphabet) is entered. For example, executing

inputemu('key_normal','Hello World'); % emulates typing "Hello World"

inputemu('key_normal','Hello World'); % emulates typing "Hello World"

directly in Matlab prompt displays “Hello World” in the next command line. The text is first decoded to a sequence of key press and release pairs, including SHIFT key presses and releases as needed, then the sequence of key commands are sent to the java Robot object.
In addition to all the characters on the keyboard, non-character keys can be stroked by using escape characters:

  • ‘\SHIFT’
  • ‘\CTRL’
  • ‘\ALT’
  • ‘\WINDOWS’
  • ‘\BACKSPACE’
  • ‘\TAB’
  • ‘\ENTER’
  • ‘\ESC’
  • ‘\PAGEUP’
  • ‘\PAGEDOWN’
  • ‘\END’
  • ‘\HOME’
  • ‘\LEFT’
  • ‘\UP’
  • ‘\RIGHT’
  • ‘\DOWN’
  • ‘\INSERT’
  • ‘\DELETE’
  • ‘\CAPSLOCK’
  • ‘\NUMLOCK’
  • ‘\SCROLLLOCK’
  • ‘\PRINTSCREEN’
  • ‘\PA– USE’

Also, use ‘\\’ for the forward slash character and ‘\F1’ – ‘\F12’ for the Function keys. For example, to execute a Matlab “ver” command automatically via command window, run

inputemu('key_normal','ver\ENTER'); % runs the ver command

inputemu('key_normal','ver\ENTER'); % runs the ver command

Lastly, as mentioned in the previous article, java.awt.Robot is sensitive to the keyboard locale, and the inputEmu utility is programmed specifically for the US QWERTY keyboard. To use the inputEmu for another keyboard layout, the program has to be modified accordingly. If anyone’s interested using this utility for different keyboard layout, I’d be glad to provide assistance.
Do you use Java’s Robot class or any of the utilities above in your work? Yair and I would love to hear about your experience in a short comment below.

Related posts:

  1. GUI automation using a Robot – This article explains how Java's Robot class can be used to programmatically control mouse and keyboard actions...
  2. PlotEdit context-menu customization – A combination of Matlab and Java Robot commands to automate a certain animation can be used when we cannot access underlying GUI/graphics code. ...
  3. Figure keypress modifiers – The figure's CurrentModifier property provides a simple and consistent way to retrieve keypress modifiers: alt-, control- and shift-clicks. ...
  4. Uicontrol callbacks – This post details undocumented callbacks exposed by the underlying Java object of Matlab uicontrols, that can be used to modify the control's behavior in a multitude of different events...
  5. Hyperlink text labels – Hyperlink labels can easily be added to Matlab GUIs. ...
  6. ScreenCapture utility – The ScreenCapture utility uses purely-documented Matlab for capturing a screen region as an image from within Matlab. ...
GUI Java Kesh Ikuma
Print Print
« Previous
Next »
7 Responses
  1. Hanna March 29, 2012 at 02:45 Reply

    Hey! Im using the inputEmu and its working great, although I had to modify it slightly since my keyboard isnt US standard. I have a quick question- do you know how to make the keyboard type the content of a variable you have set?
    For instance:
    variable= test 1
    inputemuu(‘key_normal’,variable)

    I would like this to type “test 1”
    Any help would be appreciated 🙂

  2. ScreenCapture utility | Undocumented Matlab September 24, 2012 at 03:14 Reply

    […] Java Robot in two past articles on this blog, where guest blogger Kesh Ikuma explained (here and here) how it can be used to simulate mouse and keyboard actions programmatically. […]

  3. Alex March 11, 2013 at 21:44 Reply

    This feels like a horrible abuse of everything, but I wrapped my project in a nice GUI so it’s easy to experiment. But now I need a systematic test so I can make figures for the paper. I need them yesterday of course, and the important part of the code is REALLY slow. So I’m writing a script using jMouseEmu and inputEmu, so that the robot can operate the GUI while I sleep. Here’s hoping it works after my laptop screen falls asleep!

  4. Hari April 13, 2015 at 02:57 Reply

    Hey!

    its a wonderful invention. I have been using it for my research work.

    I have one question. “Is there any way to find current location of TAB cursor ???”

    • Yair Altman April 13, 2015 at 03:23 Reply

      @Hari – you are not making any sense. What do you mean by “Tab cursor”???

      • Hari April 15, 2015 at 01:46

        As we press TAB button on keyboard, the control moves from one icon to next. the “TAB Cursor” term is being used for this control. As there are some means to find current location of mouse pointer, there might be some commands to find current position of TAB Cursor. i want to know about those commands.

        thank u!

      • Yair Altman April 15, 2015 at 09:54

        @Hari – there is no technical term called “TAB cursor”. You probably mean “focus”. You can check whether a certain component is currently in focus using the isFocusOwner() method of the underlying Java component (which you can find using the findjobj utility. For example:

        hButton = uicontrol(...);
        jButton = findjobj(hButton);
        ...
        isInFocus = jButton.isFocusOwner();  % =logical true/false

        hButton = uicontrol(...); jButton = findjobj(hButton); ... isInFocus = jButton.isFocusOwner(); % =logical true/false

Leave a Reply
HTML tags such as <b> or <i> are accepted.
Wrap code fragments inside <pre lang="matlab"> tags, like this:
<pre lang="matlab">
a = magic(3);
disp(sum(a))
</pre>
I reserve the right to edit/delete comments (read the site policies).
Not all comments will be answered. You can always email me (altmany at gmail) for private consulting.

Click here to cancel reply.

Useful links
  •  Email Yair Altman
  •  Subscribe to new posts (feed)
  •  Subscribe to new posts (reader)
  •  Subscribe to comments (feed)
 
Accelerating MATLAB Performance book
Recent Posts

Speeding-up builtin Matlab functions – part 3

Improving graphics interactivity

Interesting Matlab puzzle – analysis

Interesting Matlab puzzle

Undocumented plot marker types

Matlab toolstrip – part 9 (popup figures)

Matlab toolstrip – part 8 (galleries)

Matlab toolstrip – part 7 (selection controls)

Matlab toolstrip – part 6 (complex controls)

Matlab toolstrip – part 5 (icons)

Matlab toolstrip – part 4 (control customization)

Reverting axes controls in figure toolbar

Matlab toolstrip – part 3 (basic customization)

Matlab toolstrip – part 2 (ToolGroup App)

Matlab toolstrip – part 1

Categories
  • Desktop (45)
  • Figure window (59)
  • Guest bloggers (65)
  • GUI (165)
  • Handle graphics (84)
  • Hidden property (42)
  • Icons (15)
  • Java (174)
  • Listeners (22)
  • Memory (16)
  • Mex (13)
  • Presumed future risk (394)
    • High risk of breaking in future versions (100)
    • Low risk of breaking in future versions (160)
    • Medium risk of breaking in future versions (136)
  • Public presentation (6)
  • Semi-documented feature (10)
  • Semi-documented function (35)
  • Stock Matlab function (140)
  • Toolbox (10)
  • UI controls (52)
  • Uncategorized (13)
  • Undocumented feature (217)
  • Undocumented function (37)
Tags
ActiveX (6) AppDesigner (9) Callbacks (31) Compiler (10) Desktop (38) Donn Shull (10) Editor (8) Figure (19) FindJObj (27) GUI (141) GUIDE (8) Handle graphics (78) HG2 (34) Hidden property (51) HTML (26) Icons (9) Internal component (39) Java (178) JavaFrame (20) JIDE (19) JMI (8) Listener (17) Malcolm Lidierth (8) MCOS (11) Memory (13) Menubar (9) Mex (14) Optical illusion (11) Performance (78) Profiler (9) Pure Matlab (187) schema (7) schema.class (8) schema.prop (18) Semi-documented feature (6) Semi-documented function (33) Toolbar (14) Toolstrip (13) uicontrol (37) uifigure (8) UIInspect (12) uitools (20) Undocumented feature (187) Undocumented function (37) Undocumented property (20)
Recent Comments
  • Nicholas (6 days 14 hours ago): Hi Yair, Thanks for the reply. I am on Windows 10. I also forgot to mention that this all works wonderfully out of the editor. It only fails once compiled. So, yes, I have tried a...
  • Nicholas (6 days 14 hours ago): Hi Yair, Thanks for the reply. I am on Windows 10. I also forgot to mention that this all works wonderfully out of the editor. It only fails once compiled. So, yes, I have tried a...
  • Yair Altman (6 days 21 hours ago): Nicholas – yes, I used it in a compiled Windows app using R2022b (no update). You didn’t specify the Matlab code location that threw the error so I can’t help...
  • Nicholas (7 days 17 hours ago): Hi Yair, Have you attempted your displayWebPage utility (or the LightweightHelpPanel in general) within a compiled application? It appears to fail in apps derived from both R2022b...
  • João Neves (10 days 22 hours ago): I am on matlab 2021a, this still works: url = struct(struct(struct(struct(hF ig).Controller).PlatformHost). CEF).URL; but the html document is empty. Is there still a way to do...
  • Yair Altman (13 days 20 hours ago): Perhaps the class() function could assist you. Or maybe just wrap different access methods in a try-catch so that if one method fails you could access the data using another...
  • Jeroen Boschma (13 days 23 hours ago): Never mind, the new UI components have an HTML panel available. Works for me…
  • Alexandre (14 days 0 hours ago): Hi, Is there a way to test if data dictionnatry entry are signal, simulink parameters, variables … I need to access their value, but the access method depends on the data...
  • Nicholas (14 days 14 hours ago): In case anyone is looking for more info on the toolbar: I ran into some problems creating a toolbar with the lightweight panel. Previously, the Browser Panel had an addToolbar...
  • Jeroen Boschma (17 days 21 hours ago): I do not seem to get the scrollbars (horizontal…) working in Matlab 2020b. Snippets of init-code (all based on Yair’s snippets on this site) handles.text_explorer...
  • Yair Altman (45 days 23 hours ago): m_map is a mapping tool, not even created by MathWorks and not part of the basic Matlab system. I have no idea why you think that the customizations to the builtin bar function...
  • chengji chen (46 days 6 hours ago): Hi, I have tried the method, but it didn’t work. I plot figure by m_map toolbox, the xticklabel will add to the yticklabel at the left-down corner, so I want to move down...
  • Yair Altman (53 days 23 hours ago): @Alexander – this is correct. Matlab stopped including sqlite4java in R2021b (it was still included in 21a). You can download the open-source sqlite4java project from...
  • Alexander Eder (59 days 18 hours ago): Unfortunately Matlab stopped shipping sqlite4java starting with R2021(b?)
  • K (66 days 5 hours ago): Is there a way to programmatically manage which figure gets placed where? Let’s say I have 5 figures docked, and I split it into 2 x 1, I want to place 3 specific figures on the...
Contact us
Captcha image for Custom Contact Forms plugin. You must type the numbers shown in the image
Undocumented Matlab © 2009 - Yair Altman
This website and Octahedron Ltd. are not affiliated with The MathWorks Inc.; MATLAB® is a registered trademark of The MathWorks Inc.
Scroll to top