GUI automation using a Robot

37 Responses

1. 
   Martin Richard September 16, 2010 at 04:18 Reply

   Very interesting. I used this java.awt.Robot class recently to perform GUI testing in MATLAB and, with little effort, it became a very interesting and simple feature to use.
   Thanks for posting

2. 
   Sabrina Wendt February 1, 2011 at 08:22 Reply

   Response to:
   “Although unconfirmed, I suspect VK_SEMICOLON constant would not work with keyboards in which semicolon is mapped to SHIFT-’,’ ”

   I can confirm that this is true (writting from a danish laptop). At my computer ‘;’ is mapped to ‘,’ and so to generate a semicolon it works to write the following in MATLAB:

   robot.keyPress (java.awt.event.KeyEvent.VK_SHIFT);
   robot.keyPress (java.awt.event.KeyEvent.VK_COMMA);
   robot.keyRelease (java.awt.event.KeyEvent.VK_COMMA);
   robot.keyRelease (java.awt.event.KeyEvent.VK_SHIFT);

   Great article! Thanks for posting 🙂

3. 
   Héctor August 17, 2011 at 06:24 Reply

   I try this example some time ago from another source. Now I´m reading all your posts to learn new things (great job). When I reach newer posts I will ask you some things I want to do and don´t know how.

4. 
   Ankit January 15, 2012 at 02:14 Reply

   Hi,
   Just like you have given the commands for keyboard and mouse interface is there any command for joystick interface in java. Please help.

• 
  Yair Altman January 15, 2012 at 02:23 Reply

  @Ankit – different joysticks interface with the computer differently (for example, via USB or serial port). So there is nothing standard. However, there are numerous external Java joystick interfaces that you can integrate with Matlab. For example, http://sourceforge.net/projects/javajoystick/ and http://www.hardcode.de/jxinput/.

5. 
   appy February 29, 2012 at 23:06 Reply

   Hi this is very interesting i used it in matlab figure instead of overobj but i have problem i used mouse press
   and mouse release in windowmotionFcn in mat lab i.e when i move the mouse at that point mouse button will be pressed programmatically if i move mouse over the toolbar it is pressing the button automatically zoom save are working even to resize or close window will be activated if i just move the mouse over them but i donot want this to happen how to stop.

   Thank you.

6. 
   timmy July 6, 2012 at 08:56 Reply

   Has anyone experienced double clicking using Press/Release/Press/Release not working? Say for opening an image icon on desktop. The icon is correctly selected but the correct double-clicking results (application opening) is not performed. Everything else works, even dragging icons around, just not opening icons via double click. Thanks for the help.

7. ScreenCapture utility | Undocumented Matlab August 16, 2012 at 00:21 Reply

   […] I have discussed the Java Robot in two past articles on this blog, where guest blogger Kesh Ikuma explained how it can be used to simulate mouse and keyboard actions programmatically. […]

8. 
   MEENAKSHI.S January 30, 2013 at 00:27 Reply

   I want to use the executable file in Matlab for optimisation methods. How can I use the run button of that exe file in matlab command window as a command and not by clicking the run button of exe file so that repeated process of clicking can be avoid in optimisation methods (exe file is visual c++ program-MFC Application).

• 
  Yair Altman January 30, 2013 at 04:53 Reply

  @Meenakshi – you would need to write an automation script for this, either using the Java Robot class (as described in the article), or using an external application (e.g., AutoIt) that you would call from Matlab using the system command. If you would like my help writing such an application for you as a consultant, please send me an email (altmany at gmail).

9. 
   MEENAKSHI.S February 12, 2013 at 03:49 Reply

   thank you sir…i have tried Java Robot class in matlab…it works well..i.e.to press F1 KEY of matlab suddenly help window opened…so really happy to see the response without clicking F1 KEY…it works well…but i want to press F5 KEY of EXECUTABLE FILE IN MATLAB COMMAND WINDOW…What can i do sir.. PLEASE SHARE YOUR IDEAS…Thank you Sir…

• 
  Yair Altman February 12, 2013 at 09:21 Reply

  @Meenakshi – I don’t understand your problem. Simply call the relevant function directly, you don’t need F5.

10. 
    George March 14, 2013 at 10:40 Reply

    thanks a lot for your post,and i have a question

    i want to type 1 to 3 with matlab
    of course i can write

    import java.awt.Robot; import java.awt.event.*; robot=Robot(); robot.keyPress(KeyEvent.VK_1); robot.keyRelease(KeyEvent.VK_1); robot.keyPress(KeyEvent.VK_2); robot.keyRelease(KeyEvent.VK_2); robot.keyPress(KeyEvent.VK_3); robot.keyRelease(KeyEvent.VK_3);

    import java.awt.Robot; import java.awt.event.*; robot=Robot(); robot.keyPress(KeyEvent.VK_1); robot.keyRelease(KeyEvent.VK_1); robot.keyPress(KeyEvent.VK_2); robot.keyRelease(KeyEvent.VK_2); robot.keyPress(KeyEvent.VK_3); robot.keyRelease(KeyEvent.VK_3);

    i wanna ask how to do these with while or for loop

    thank you

• 
  Yair Altman March 14, 2013 at 16:39 Reply

  @George – in your specific case you can simply use the numeric equivalents of KeyEvent.VK_*: VK_0=48, VK_1=49 and so on until VK_9=57. You can see the full list using my checkClass utility. Turning these numbers into a loop is trivial. Note that using numbers is far less maintainable than using the enumerations, so add ample documentation to your code.

• 
  Mayank August 5, 2017 at 16:01 Reply

  create an int[] array for all the KeyEvent.Vk_1 to ..last then use the for loop to iterate over the same.

11. 
    Muthu Vicky October 12, 2013 at 06:23 Reply

    Thanks a lot for this post.. This is really awesome!!

    Can you please tell me how to press function keys (i.e.) f6 or f7 or some other function key?

• 
  Lucius Domitius Ahenobarbus November 19, 2013 at 02:47 Reply

  Just use:

  robot.keyPress(KeyEvent.VK_F6); robot.keyRelease(KeyEvent.VK_F6);

  robot.keyPress(KeyEvent.VK_F6); robot.keyRelease(KeyEvent.VK_F6);

• 
  Lucius Domitius Ahenobarbus November 19, 2013 at 02:49

  Btw: Thanks a lot – again – this blog was a big help for me!

12. 
    hema December 30, 2013 at 00:53 Reply

    how to right click on a window??

• 
  Yair Altman December 31, 2013 at 16:09 Reply

  @hema –

  jRobot = java.awt.Robot; jRobot.mouseMove(400,300); % move to absolute x,y pixel position jEvent = java.awt.event.InputEvent.BUTTON3_MASK; jRobot.mousePress(jEvent); jRobot.mouseRelease(jEvent); %=right-click

  jRobot = java.awt.Robot; jRobot.mouseMove(400,300); % move to absolute x,y pixel position jEvent = java.awt.event.InputEvent.BUTTON3_MASK; jRobot.mousePress(jEvent); jRobot.mouseRelease(jEvent); %=right-click

13. 
    puja May 30, 2014 at 04:00 Reply

    i am working on dynamic typing project.i want to capture the time when key is pressed and released.can anyone please help me

• 
  Yair Altman May 30, 2014 at 07:42 Reply

  @Puja – take a look at the KeyPressedCallback and KeyReleasedCallback of uicontrols, explained here: http://undocumentedmatlab.com/blog/uicontrol-callbacks

14. 
    Ujjawal May 15, 2015 at 05:44 Reply

    Hi,

    Nice post.
    Is there a way to press right control(Ctrl) key on windows using Robot or if there is some other solution to press right control key programatically?

    Best Regards

15. 
    Ajay June 3, 2015 at 10:49 Reply

    I am working on a project in which … I am first detecting the color objects from the video and then to those detected colors ..we are assigning mouse functions …like red for mouse pointer movement …green for mouse scroll and one blue for left click …two blue for right click and three blue for double click ….mouse pointer movement …scrolling and left click …upto this we have done …but the right click and double click are the two functions we are facing problem with …we are not being able to fit the one blue object …two blue objects and 3 blue objects in a loop …so it will differentiate between them and excute those functions …so plz help …if u know something regarding this

• 
  Yair Altman June 3, 2015 at 11:16 Reply

  @Ajay – as the article explains you can use robot.mousePress and robot.mouseRelease to automate the mouse clicks at the current mouse pointer position.

  But ..why ..are ..you ..writing ..like ..this?! – ..it ..is ..so ..difficult ..to ..read! can’t you write text like a normal person ???

16. 
    sriram November 24, 2015 at 08:56 Reply

    We need to control a different Java application window (launched via JNLP). Need to enter username and password. Is it possible with Matlab.

17. 
    Xen March 23, 2016 at 23:44 Reply

    Very helpful indeed. Thank you.
    I am simulating the ESCAPE button press in my GUI to cancel drawing using imfreehand tool. After doing this for a while, the GUI becomes slow progressively until a point where it is no longer usable. I suspect that java.awt.Robot might have to do with it. Is there a way to cancel/stop/remove the robot? After creating the robot using

    removing it using

    does the job appropriately, or there is a more appropriate way? Or maybe something else might be causing the GUI to run slowly?

• 
  Yair Altman March 24, 2016 at 11:56 Reply

  Your code snippets did not come through, but in any case I suspect that the problem is not in Robot, which is a very simple class and probably not the source of the issues that you see.

18. 
    Apostolos Iliopoulos March 29, 2016 at 17:01 Reply

    Very nice work!!
    But i have a problem. I would like to write a code which runs an executable file and then enter to programs window to create a new case. The code that I’m using is this

    system('w0315v11-64bit'); robot = java.awt.Robot; robot.keyPress (java.awt.event.KeyEvent.VK_ENTER); robot.keyRelease(java.awt.event.KeyEvent.VK_ENTER);

    system('w0315v11-64bit'); robot = java.awt.Robot; robot.keyPress (java.awt.event.KeyEvent.VK_ENTER); robot.keyRelease(java.awt.event.KeyEvent.VK_ENTER);

    But instead of pressing enter on the programs window it presses it in the Matlab editor.
    Can you please tell me how to overcome this?
    Thank you in advance

• 
  Yair Altman March 30, 2016 at 10:55 Reply

  @Apostolos – the code as you wrote it just simulates pressing the <Enter> key so the key event will be sent to the currently-active window. If your editor is currently in focus then obviously the key event will be sent to it. If you want to send the key event to another window you must first ensure that the focus is on that window. There is no general way to do this, it is application-dependent.

19. 
    NSReddy September 24, 2016 at 13:46 Reply

    Can u help me how to do the ” Alt+S” using Robot Class

20. 
    Joshua November 14, 2017 at 05:47 Reply

    Hi, is there a way to simulate the keyboard input using the robot when the robot encounters user input request generated? For example,

    a = input('Type a word: ', 's');

    a = input('Type a word: ', 's');

    This is for the purpose of testing the interactive feature of a program. Is it possible?

    Thanks!

21. 
    maiky76 January 28, 2022 at 19:34 Reply

    Hi,

    The original post got me up and running in no time. Thank you so much!
    I have been trying to translate what I did in Octave to share the code with others but I can’t get it to work.

    Would anyone be gracious enough to give it a try?

    Here is the code that behaves as I’d like it to:
    Open external program, return to Octave script, Octave makes some keyboard inputs closes the program, then back to Octave to finish the remaining commands of the script

    clear all close all clc   if exist('OCTAVE_VERSION', 'builtin') == 0 % Matlab version import java.awt.*; import java.awt.event.*;   system('"calc.exe"&');   R2D2 = Robot; R2D2.delay(100) R2D2.keyPress(KeyEvent.VK_8) R2D2.keyRelease(KeyEvent.VK_8) R2D2.delay(500) % R2D2.keyPress(KeyEvent.VK_PLUS) % does not work! % R2D2.keyRelease(KeyEvent.VK_PLUS) R2D2.keyPress(KeyEvent.VK_DIVIDE) R2D2.keyRelease(KeyEvent.VK_DIVIDE) R2D2.delay(500) R2D2.keyPress(KeyEvent.VK_2) R2D2.keyRelease(KeyEvent.VK_2) R2D2.delay(500) R2D2.keyPress(KeyEvent.VK_ENTER) R2D2.keyRelease(KeyEvent.VK_ENTER) R2D2.delay(1000) R2D2.keyPress(KeyEvent.VK_ALT) R2D2.keyPress(KeyEvent.VK_F4) R2D2.keyRelease(KeyEvent.VK_ALT) R2D2.keyRelease(KeyEvent.VK_F4) else % Octave Version not working % OBJ = java_set (OBJ, NAME, VAL) % obj = javaObject (classname, arg1, …) % Ret = javaMethod (methodname, obj, arg1, …) % There is no import function so we need to express the full qualifier % https://savannah.gnu.org/task/?12601   system("calc.exe", false, "async");   R2D2 = javaObject("java.awt.Robot"); % works R2D2.delay(5000); % works % R2D2.keyPress(KeyEvent.VK_8) % javaMethod('KeyPress', R2D2, VK_ENTER) % javaMethod("KeyRelease", R2D2, 'VK_8') % javaMethod('KeyPress', 'VK_ENTER') % R2D2.keyRelease(KeyEvent.VK_8)   % R2D2.delay(500); % javaMethod('KeyPress', R2D2, VK_PLUS) % javaMethod('KeyRelease', R2D2, VK_PLUS) % R2D2.delay(500); % javaMethod('KeyPress', R2D2, VK_2) % javaMethod('KeyRelease', R2D2, VK_2) % R2D2.delay(500); % javaMethod('KeyPress', R2D2, VK_ENTER) % javaMethod('KeyRelease', R2D2, VK_ENTER) % 1+2   end

    clear all close all clc if exist('OCTAVE_VERSION', 'builtin') == 0 % Matlab version import java.awt.*; import java.awt.event.*; system('"calc.exe"&amp;'); R2D2 = Robot; R2D2.delay(100) R2D2.keyPress(KeyEvent.VK_8) R2D2.keyRelease(KeyEvent.VK_8) R2D2.delay(500) % R2D2.keyPress(KeyEvent.VK_PLUS) % does not work! % R2D2.keyRelease(KeyEvent.VK_PLUS) R2D2.keyPress(KeyEvent.VK_DIVIDE) R2D2.keyRelease(KeyEvent.VK_DIVIDE) R2D2.delay(500) R2D2.keyPress(KeyEvent.VK_2) R2D2.keyRelease(KeyEvent.VK_2) R2D2.delay(500) R2D2.keyPress(KeyEvent.VK_ENTER) R2D2.keyRelease(KeyEvent.VK_ENTER) R2D2.delay(1000) R2D2.keyPress(KeyEvent.VK_ALT) R2D2.keyPress(KeyEvent.VK_F4) R2D2.keyRelease(KeyEvent.VK_ALT) R2D2.keyRelease(KeyEvent.VK_F4) else % Octave Version not working % OBJ = java_set (OBJ, NAME, VAL) % obj = javaObject (classname, arg1, …) % Ret = javaMethod (methodname, obj, arg1, …) % There is no import function so we need to express the full qualifier % https://savannah.gnu.org/task/?12601 system("calc.exe", false, "async"); R2D2 = javaObject("java.awt.Robot"); % works R2D2.delay(5000); % works % R2D2.keyPress(KeyEvent.VK_8) % javaMethod('KeyPress', R2D2, VK_ENTER) % javaMethod("KeyRelease", R2D2, 'VK_8') % javaMethod('KeyPress', 'VK_ENTER') % R2D2.keyRelease(KeyEvent.VK_8) % R2D2.delay(500); % javaMethod('KeyPress', R2D2, VK_PLUS) % javaMethod('KeyRelease', R2D2, VK_PLUS) % R2D2.delay(500); % javaMethod('KeyPress', R2D2, VK_2) % javaMethod('KeyRelease', R2D2, VK_2) % R2D2.delay(500); % javaMethod('KeyPress', R2D2, VK_ENTER) % javaMethod('KeyRelease', R2D2, VK_ENTER) % 1+2 end

    The reason why I am interested can we found here: Acoustic Horn Design – The Easy Way (Ath4) | Page 459 | diyAudio)
    We are trying to perform some optimization on the design of an DIY acoustic waveguide.
    Any help would be much appreciated!

• 
  maiky76 January 30, 2022 at 03:06 Reply

  I figured it out eventually. To my knowledge this is the only public example: https://octave.discourse.group/t/help-on-java-keyevent/2163/14

  robot = javaObject("java.awt.Robot"); wait1 = robot.delay(500); VKeight = java_get ('java.awt.event.KeyEvent','VK_8'); a1 = javaMethod ('keyPress', robot, VKeight); a2 = javaMethod ('keyRelease', robot, VKeight);

  robot = javaObject("java.awt.Robot"); wait1 = robot.delay(500); VKeight = java_get ('java.awt.event.KeyEvent','VK_8'); a1 = javaMethod ('keyPress', robot, VKeight); a2 = javaMethod ('keyRelease', robot, VKeight);

  etc…

22. 
    Elsa Smith May 18, 2023 at 14:19 Reply

    I’m suspecting that the slow performance of my GUI may be due to the use of java.awt.Robot. Is there a way to cancel/stop/remove the robot after it has been created, or is there an alternative way to simulate an ESCAPE button press to cancel drawing using imfreehand tool without using a Robot that involves a click of some kind?

• 
  Yair Altman May 19, 2023 at 16:45 Reply

  Robot only runs when you tell it to run a command such as keyPress. If you don’t tell it to run a command, it uses no CPU, so there’s no need to remove the Robot object from memory. You just need to ensure that it doesn’t keep running commands continuously, in some kind of loop.

  To simulate an ESCAPE key press, run robot.keyPress(java.awt.event.KeyEvent.VK_ESCAPE);
robot.keyRelease(java.awt.event.KeyEvent.VK_ESCAPE); in your Matlab code.

  Instead of simulating an ESCAPE key-press, you can also simulate a mouse-up event: robot.mouseRelease(java.awt.event.InputEvent.BUTTON1_MASK);

  You can also directly call the Figure’s WindowButtonUpFcn callback function to invoke imfreehand’s exit method without needing to use Robot: hFig.WindowButtonUpFcn(hFig,[])

23. 
    Elsa Smith May 18, 2023 at 14:47 Reply

    I recently used java.awt.Robot to perform GUI testing on MATLAB and found it to be an extremely easy and useful way to control mouse movements.

Leave a Reply

<pre lang="matlab">
a = magic(3);
disp(sum(a))
</pre>

Click here to cancel reply.