- Undocumented Matlab - https://undocumentedmatlab.com -

GUI automation using a Robot

Posted By Yair Altman On September 15, 2010 | 32 Comments

I would like to welcome guest writer Takeshi (Kesh) Ikuma [3]. Kesh has posted several interesting utilities on the Matlab File Exchange, including the award-winning Enhanced Input Dialog Box [4]. Today, Kesh will describe how we can automate GUI actions programmatically.

Automating GUI actions, including controlling a mouse and keyboard programmatically, is often useful. This can be used, for example, to demonstrate GUI usage or to perform a recorded macro.

Matlab’s Handle Graphics interface provides a simple way to manipulate the mouse cursor position, but not to emulate mouse or keyboard clicks. However, we can utilize Java’s java.awt.Robot [5] class.

This article provides an overview of the Robot class and how it can be used to program mouse movement, button clicks and keyboard strikes.

java.awt.Robot class

The online Java documentation describes the purpose of the Robot class as follows:
This class is used to generate native system input events for the purposes of test automation, self-running demos, and other applications where control of the mouse and keyboard is needed.

This class has three main functionalities: mouse control, keyboard control, and screen capture. Here are some of the important member functions:

Mouse control functions

void mouseMove [6](int x, int y)

This function moves the cursor to the coordinate (x, y) which is defined with respect to the top-left screen corner (in contrast to Matlab’s coordinate origin at the bottom-left corner).

void mousePress [7](int buttons)
void mouseRelease [8](int buttons)

This pair of functions performs the button click. Their input argument is an OR’ed combination of java.awt.event.InputEvent [9]s:

java.awt.event.InputEvent.BUTTON1_MASK   // left mouse button 
java.awt.event.InputEvent.BUTTON2_MASK   // middle mouse button
java.awt.event.InputEvent.BUTTON3_MASK   // right mouse button

Keyboard control functions

Keyboard action is emulated by the following pair of functions. Their keycodes are defined in java.awt.event.KeyEvent [10]:

void keyPress [11](int keycode)
void keyRelease [12](int keycode)

NOTE: Although java.awt.event.KeyEvent constants defines most of the US QWERTY keys, not all can actually be used with java.awt.Robot. Specifically, it appears that only the KeyEvent constants for unmodified keys can be used. See the following section for an example.

Utility functions

The robot can be put to sleep for a desired duration (in milliseconds). Also, the calling routine can be blocked until the robot exhausts its command queue.

void delay [13](int ms)
void waitForIdle [14]()

Readers interested in Robot’s screen-capture capability are invited to take a look at Yair’s ScreenCapture utility [15] on the File Exchange.

Using java.awt.Robot in Matlab

To create the Robot object in Matlab, simply run

robot = java.awt.Robot;

Moving the mouse cursor

With Matlab’s root (0) handle, the mouse cursor can be repositioned by

set(0,'PointerLocation',[x y]);

The mouse cursor is placed on the x-th pixel from left edge and the y-th pixel from the bottom edge of the screen.

Alternately, we could use the semi-documented moveptr function [16], which was described here last year.

The same operation can also be done using Robot as follows:

scrSize = get(0,'ScreenSize');
robot.mouseMove(x,scrSize(2)-y);

The extra step must be taken to convert from Matlab to Java screen coordinates.

Depending on the specific case (for example, whether or not we know the absolute or only relative screen coordinates), we may prefer using any of these equivalent mouse-movement alternatives.

Clicking mouse buttons

Unfortunately, we have few alternatives for automating mouse-clicks – Robot is basically our only option. Matlab recognizes 4 different mouse click types as specified in the Figure’s SelectionType property:

  • Normal: Click left mouse button
  • Extend: SHIFT-click left mouse button
  • Alternate: CTRL-click left mouse button
  • Open: Double click left mouse button

To observe the mouse click, open a figure and set its WindowButtonDownFcn callback:

myCallback = @(hObj,event) disp(get(hObj,'SelectionType'));
set(gcf,'WindowButtonDownFcn', myCallback);

To trigger the callback, the figure must have the focus and the mouse cursor must be on the figure. To ensure the callback invocation, “figure(gcf); drawnow;” commands are included in the code examples below. Make sure to place the mouse cursor on the figure before running these codes.

Here’s how the Robot can be used to perform each type of clicking. First, normal click:

figure(gcf); drawnow;
robot.mousePress  (java.awt.event.InputEvent.BUTTON1_MASK);
robot.mouseRelease(java.awt.event.InputEvent.BUTTON1_MASK);

For “open” or double click, repeat it twice:

figure(gcf); drawnow;
robot.mousePress  (java.awt.event.InputEvent.BUTTON1_MASK);
robot.mouseRelease(java.awt.event.InputEvent.BUTTON1_MASK);
robot.mousePress  (java.awt.event.InputEvent.BUTTON1_MASK);
robot.mouseRelease(java.awt.event.InputEvent.BUTTON1_MASK);

For the other two click types, we need to use the keyboard functions. First, “extend” or SHIFT-click:

figure(gcf); drawnow;
robot.keyPress    (java.awt.event.KeyEvent.VK_SHIFT);
robot.mousePress  (java.awt.event.InputEvent.BUTTON1_MASK);
robot.mouseRelease(java.awt.event.InputEvent.BUTTON1_MASK);
robot.keyRelease  (java.awt.event.KeyEvent.VK_SHIFT);

Lastly, “alternate” or CTRL-click:

figure(gcf); drawnow;
robot.keyPress    (java.awt.event.KeyEvent.VK_CONTROL);
robot.mousePress  (java.awt.event.InputEvent.BUTTON1_MASK);
robot.mouseRelease(java.awt.event.InputEvent.BUTTON1_MASK);
robot.keyRelease  (java.awt.event.KeyEvent.VK_CONTROL);

Clicking keyboard keys

Just as we have shown above how to press a modifier key (Ctrl or Alt key) to set the figure’s SelectionType property, keyPress and keyRelease functions can be used with character keys to type text. For example, with the focus on Matlab’s Command Prompt, running the following code executes the ver command:

robot.keyPress    (java.awt.event.KeyEvent.VK_V);
robot.keyRelease  (java.awt.event.KeyEvent.VK_V);
robot.keyPress    (java.awt.event.KeyEvent.VK_E);
robot.keyRelease  (java.awt.event.KeyEvent.VK_E);
robot.keyPress    (java.awt.event.KeyEvent.VK_R);
robot.keyRelease  (java.awt.event.KeyEvent.VK_R);
robot.keyPress    (java.awt.event.KeyEvent.VK_ENTER);
robot.keyRelease  (java.awt.event.KeyEvent.VK_ENTER);

In the previous section, I mentioned that not all KeyEvent constants are supported by java.awt.Robot, and keys must be defined with their unmodified characters. For example, a semicolon (‘;’) can be typed by:

robot.keyPress    (java.awt.event.KeyEvent.VK_SEMICOLON);
robot.keyRelease  (java.awt.event.KeyEvent.VK_SEMICOLON);

However, doing the same for a colon (‘:’ or SHIFT-‘;’) causes an error:

robot.keyPress    (java.awt.event.KeyEvent.VK_COLON);
robot.keyRelease  (java.awt.event.KeyEvent.VK_COLON);
??? Java exception occurred:
java.lang.IllegalArgumentException: Invalid key code
    at sun.awt.windows.WRobotPeer.keyPress(Native Method)
    at java.awt.Robot.keyPress(Unknown Source)

Instead of using the VK_COLON constant, VK_SEMICOLON constant must be used while SHIFT modifier key is pressed:

robot.keyPress    (java.awt.event.KeyEvent.VK_SHIFT);
robot.keyPress    (java.awt.event.KeyEvent.VK_SEMICOLON);
robot.keyRelease  (java.awt.event.KeyEvent.VK_SEMICOLON);
robot.keyRelease  (java.awt.event.KeyEvent.VK_SHIFT);

Although unconfirmed, I suspect VK_SEMICOLON constant would not work with keyboards in which semicolon is mapped to SHIFT-‘,’ (e.g., Spanish or Italian keyboard layouts), while the VK_COLON constant would work with a French keyboard.

Putting it all together: jMouseEmu and inputEmu

To simplify the execution of the mouse and keyboard actions outlined above, I (Kesh Ikuma) have created the jMouseEmu [17] and inputEmu [18] utilities, which are available on the Matlab File Exchange. These interface/wrapper functions will be described in next week’s article.

Do you have any favorite use for Java Robots in your workflow? If so, please tell us all about it in a comment below [19].

Categories: Figure window, Guest bloggers, GUI, Java, Low risk of breaking in future versions


32 Comments (Open | Close)

32 Comments To "GUI automation using a Robot"

#1 Comment By Martin Richard On September 16, 2010 @ 4:18 am

Very interesting. I used this java.awt.Robot class recently to perform GUI testing in MATLAB and, with little effort, it became a very interesting and simple feature to use.
Thanks for posting

#2 Comment By Sabrina Wendt On February 1, 2011 @ 8:22 am

Response to:
“Although unconfirmed, I suspect VK_SEMICOLON constant would not work with keyboards in which semicolon is mapped to SHIFT-’,’ ”

I can confirm that this is true (writting from a danish laptop). At my computer ‘;’ is mapped to ‘,’ and so to generate a semicolon it works to write the following in MATLAB:

robot.keyPress (java.awt.event.KeyEvent.VK_SHIFT);
robot.keyPress (java.awt.event.KeyEvent.VK_COMMA);
robot.keyRelease (java.awt.event.KeyEvent.VK_COMMA);
robot.keyRelease (java.awt.event.KeyEvent.VK_SHIFT);

Great article! Thanks for posting :-)

#3 Comment By Héctor On August 17, 2011 @ 6:24 am

I try this example some time ago from another source. Now I´m reading all your posts to learn new things (great job). When I reach newer posts I will ask you some things I want to do and don´t know how.

#4 Comment By Ankit On January 15, 2012 @ 2:14 am

Hi,
Just like you have given the commands for keyboard and mouse interface is there any command for joystick interface in java. Please help.

#5 Comment By Yair Altman On January 15, 2012 @ 2:23 am

@Ankit – different joysticks interface with the computer differently (for example, via USB or serial port). So there is nothing standard. However, there are numerous external Java joystick interfaces that you can integrate with Matlab. For example, [26] and [27].

#6 Comment By appy On February 29, 2012 @ 11:06 pm

Hi this is very interesting i used it in matlab figure instead of overobj but i have problem i used mouse press
and mouse release in windowmotionFcn in mat lab i.e when i move the mouse at that point mouse button will be pressed programmatically if i move mouse over the toolbar it is pressing the button automatically zoom save are working even to resize or close window will be activated if i just move the mouse over them but i donot want this to happen how to stop.

Thank you.

#7 Comment By timmy On July 6, 2012 @ 8:56 am

Has anyone experienced double clicking using Press/Release/Press/Release not working? Say for opening an image icon on desktop. The icon is correctly selected but the correct double-clicking results (application opening) is not performed. Everything else works, even dragging icons around, just not opening icons via double click. Thanks for the help.

#8 Pingback By ScreenCapture utility | Undocumented Matlab On August 16, 2012 @ 12:21 am

[…] I have discussed the Java Robot in two past articles on this blog, where guest blogger Kesh Ikuma explained how it can be used to simulate mouse and keyboard actions programmatically. […]

#9 Comment By MEENAKSHI.S On January 30, 2013 @ 12:27 am

I want to use the executable file in Matlab for optimisation methods. How can I use the run button of that exe file in matlab command window as a command and not by clicking the run button of exe file so that repeated process of clicking can be avoid in optimisation methods (exe file is visual c++ program-MFC Application).

#10 Comment By Yair Altman On January 30, 2013 @ 4:53 am

@Meenakshi – you would need to write an automation script for this, either using the Java Robot class (as described in the article), or using an external application (e.g., AutoIt) that you would call from Matlab using the system command. If you would like my help writing such an application for you as a consultant, please send me an email (altmany at gmail).

#11 Comment By MEENAKSHI.S On February 12, 2013 @ 3:49 am

thank you sir…i have tried Java Robot class in matlab…it works well..i.e.to press F1 KEY of matlab suddenly help window opened…so really happy to see the response without clicking F1 KEY…it works well…but i want to press F5 KEY of EXECUTABLE FILE IN MATLAB COMMAND WINDOW…What can i do sir.. PLEASE SHARE YOUR IDEAS…Thank you Sir…

#12 Comment By Yair Altman On February 12, 2013 @ 9:21 am

@Meenakshi – I don’t understand your problem. Simply call the relevant function directly, you don’t need F5.

#13 Comment By George On March 14, 2013 @ 10:40 am

thanks a lot for your post,and i have a question

i want to type 1 to 3 with matlab
of course i can write

import java.awt.Robot;
import java.awt.event.*;
robot=Robot();
robot.keyPress(KeyEvent.VK_1);
robot.keyRelease(KeyEvent.VK_1);
robot.keyPress(KeyEvent.VK_2);
robot.keyRelease(KeyEvent.VK_2);
robot.keyPress(KeyEvent.VK_3);
robot.keyRelease(KeyEvent.VK_3);

i wanna ask how to do these with while or for loop

thank you

#14 Comment By Yair Altman On March 14, 2013 @ 4:39 pm

@George – in your specific case you can simply use the numeric equivalents of KeyEvent.VK_*: VK_0=48, VK_1=49 and so on until VK_9=57. You can see the full list using my [28]. Turning these numbers into a loop is trivial. Note that using numbers is far less maintainable than using the enumerations, so add ample documentation to your code.

#15 Comment By Mayank On August 5, 2017 @ 4:01 pm

create an int[] array for all the KeyEvent.Vk_1 to ..last then use the for loop to iterate over the same.

#16 Comment By Muthu Vicky On October 12, 2013 @ 6:23 am

Thanks a lot for this post.. This is really awesome!!

Can you please tell me how to press function keys (i.e.) f6 or f7 or some other function key?

#17 Comment By Lucius Domitius Ahenobarbus On November 19, 2013 @ 2:47 am

Just use:

robot.keyPress(KeyEvent.VK_F6);
robot.keyRelease(KeyEvent.VK_F6);

#18 Comment By Lucius Domitius Ahenobarbus On November 19, 2013 @ 2:49 am

Btw: Thanks a lot – again – this blog was a big help for me!

#19 Comment By hema On December 30, 2013 @ 12:53 am

how to right click on a window??

#20 Comment By Yair Altman On December 31, 2013 @ 4:09 pm

@hema –

jRobot = java.awt.Robot;
jRobot.mouseMove(400,300); % move to absolute x,y pixel position
jEvent = java.awt.event.InputEvent.BUTTON3_MASK;
jRobot.mousePress(jEvent); jRobot.mouseRelease(jEvent);  %=right-click

#21 Comment By puja On May 30, 2014 @ 4:00 am

i am working on dynamic typing project.i want to capture the time when key is pressed and released.can anyone please help me

#22 Comment By Yair Altman On May 30, 2014 @ 7:42 am

@Puja – take a look at the KeyPressedCallback and KeyReleasedCallback of uicontrols, explained here: [29]

#23 Comment By Ujjawal On May 15, 2015 @ 5:44 am

Hi,

Nice post.
Is there a way to press right control(Ctrl) key on windows using Robot or if there is some other solution to press right control key programatically?

Best Regards

#24 Comment By Ajay On June 3, 2015 @ 10:49 am

I am working on a project in which … I am first detecting the color objects from the video and then to those detected colors ..we are assigning mouse functions …like red for mouse pointer movement …green for mouse scroll and one blue for left click …two blue for right click and three blue for double click ….mouse pointer movement …scrolling and left click …upto this we have done …but the right click and double click are the two functions we are facing problem with …we are not being able to fit the one blue object …two blue objects and 3 blue objects in a loop …so it will differentiate between them and excute those functions …so plz help …if u know something regarding this

#25 Comment By Yair Altman On June 3, 2015 @ 11:16 am

@Ajay – as the article explains you can use robot.mousePress and robot.mouseRelease to automate the mouse clicks at the current mouse pointer position.

But ..why ..are ..you ..writing ..like ..this?! – ..it ..is ..so ..difficult ..to ..read! can’t you write text like a normal person ???

#26 Comment By sriram On November 24, 2015 @ 8:56 am

We need to control a different Java application window (launched via JNLP). Need to enter username and password. Is it possible with Matlab.

#27 Comment By Xen On March 23, 2016 @ 11:44 pm

Very helpful indeed. Thank you.
I am simulating the ESCAPE button press in my GUI to cancel drawing using imfreehand tool. After doing this for a while, the GUI becomes slow progressively until a point where it is no longer usable. I suspect that java.awt.Robot might have to do with it. Is there a way to cancel/stop/remove the robot? After creating the robot using

removing it using

does the job appropriately, or there is a more appropriate way? Or maybe something else might be causing the GUI to run slowly?

#28 Comment By Yair Altman On March 24, 2016 @ 11:56 am

Your code snippets did not come through, but in any case I suspect that the problem is not in Robot, which is a very simple class and probably not the source of the issues that you see.

#29 Comment By Apostolos Iliopoulos On March 29, 2016 @ 5:01 pm

Very nice work!!
But i have a problem. I would like to write a code which runs an executable file and then enter to programs window to create a new case. The code that I’m using is this

system('w0315v11-64bit');
robot = java.awt.Robot;
robot.keyPress  (java.awt.event.KeyEvent.VK_ENTER);
robot.keyRelease(java.awt.event.KeyEvent.VK_ENTER);

But instead of pressing enter on the programs window it presses it in the Matlab editor.
Can you please tell me how to overcome this?
Thank you in advance

#30 Comment By Yair Altman On March 30, 2016 @ 10:55 am

@Apostolos – the code as you wrote it just simulates pressing the <Enter> key so the key event will be sent to the currently-active window. If your editor is currently in focus then obviously the key event will be sent to it. If you want to send the key event to another window you must first ensure that the focus is on that window. There is no general way to do this, it is application-dependent.

#31 Comment By NSReddy On September 24, 2016 @ 1:46 pm

Can u help me how to do the ” Alt+S” using Robot Class

#32 Comment By Joshua On November 14, 2017 @ 5:47 am

Hi, is there a way to simulate the keyboard input using the robot when the robot encounters user input request generated? For example,

a = input('Type a word: ', 's');

This is for the purpose of testing the interactive feature of a program. Is it possible?

Thanks!


Article printed from Undocumented Matlab: https://undocumentedmatlab.com

URL to article: https://undocumentedmatlab.com/blog/gui-automation-robot

URLs in this post:

[1] Image: http://undocumentedmatlab.com/feed/

[2] email feed: http://undocumentedmatlab.com/subscribe_email.html

[3] Takeshi (Kesh) Ikuma: http://www.mathworks.com/matlabcentral/fileexchange/authors/36665

[4] Enhanced Input Dialog Box: http://www.mathworks.com/matlabcentral/fileexchange/25862-enhanced-input-dialog-box

[5] java.awt.Robot: http://download.oracle.com/javase/6/docs/api/java/awt/Robot.html

[6] mouseMove: http://download.oracle.com/javase/6/docs/api/java/awt/Robot.html#mouseMove%28int,%20int%29

[7] mousePress: http://download.oracle.com/javase/6/docs/api/java/awt/Robot.html#mousePress%28int%29

[8] mouseRelease: http://download.oracle.com/javase/6/docs/api/java/awt/Robot.html#mouseRelease%28int%29

[9] java.awt.event.InputEvent: http://download.oracle.com/javase/6/docs/api/java/awt/event/InputEvent.html

[10] java.awt.event.KeyEvent: http://download.oracle.com/javase/6/docs/api/java/awt/event/KeyEvent.html

[11] keyPress: http://download.oracle.com/javase/6/docs/api/java/awt/Robot.html#keyPress(int)

[12] keyRelease: http://download.oracle.com/javase/6/docs/api/java/awt/Robot.html#keyRelease(int)

[13] delay: http://download.oracle.com/javase/6/docs/api/java/awt/Robot.html#delay(int)

[14] waitForIdle: http://download.oracle.com/javase/6/docs/api/java/awt/Robot.html#waitForIdle()

[15] ScreenCapture utility: http://www.mathworks.com/matlabcentral/fileexchange/24323-screencapture-get-a-screen-capture-of-a-figure-frame-or-component

[16] moveptr function: https://undocumentedmatlab.com/blog/undocumented-mouse-pointer-functions/#moveptr

[17] jMouseEmu: http://www.mathworks.com/matlabcentral/fileexchange/28357-jMouseEmu

[18] inputEmu: http://www.mathworks.com/matlabcentral/fileexchange/28603-inputEmu

[19] below: https://undocumentedmatlab.com/blog/gui-automation-robot/#respond

[20] GUI automation utilities : https://undocumentedmatlab.com/blog/gui-automation-utilities

[21] Tab panels – uitab and relatives : https://undocumentedmatlab.com/blog/tab-panels-uitab-and-relatives

[22] Matlab-Java interface using a static control : https://undocumentedmatlab.com/blog/matlab-java-interface-using-static-control

[23] ScreenCapture utility : https://undocumentedmatlab.com/blog/screencapture-utility

[24] JGraph in Matlab figures : https://undocumentedmatlab.com/blog/jgraph-in-matlab-figures

[25] Working with non-standard DPI displays : https://undocumentedmatlab.com/blog/working-with-non-standard-dpi-displays

[26] : http://sourceforge.net/projects/javajoystick/

[27] : http://www.hardcode.de/jxinput/

[28] : http://www.mathworks.com/matlabcentral/fileexchange/26947-checkclass

[29] : http://undocumentedmatlab.com/blog/uicontrol-callbacks

Copyright © Yair Altman - Undocumented Matlab. All rights reserved.