figure; surf(peaks); imgData=screencapture(gcf); imshow(imgData);

imageData = screencapture;                     % interactively select screen-capture rectangle
imageData = screencapture(hListbox);           % capture image of a uicontrol
imageData = screencapture(0,  [20,30,40,50]);  % capture a small desktop sub-region
imageData = screencapture(gcf,[20,30,40,50]);  % capture a small figure sub-region
% capture a small sub-region of an axes
imageData = screencapture(gca,[10,20,30,40]);
imshow(imageData);  % display the captured image in a matlab figure
imwrite(imageData,'myImage.png');  % save the captured image to file
% capture a sub-region of an image
img = imread('cameraman.tif');
hImg = imshow(img);
screencapture(hImg,[60,35,140,80]);  % in data units, not pixel units
screencapture(gcf,[],'myFigure.jpg');                   % capture the entire figure into file
screencapture('handle',gcf,'filename','myFigure.jpg');  % same as previous
screencapture('toolbar',gcf);                           % adds a screen-capture button to gcf's toolbar
screencapture('toolbar',[],'file','sc.bmp');            % same, using a default output filename

Purely documented

Over the course of the past few years I have submitted 40 utilities to the Matlab File Exchange, several of which have been selected for POTW. Unfortunately, since most of my utilities employ undocumented Matlab features to some extent (I can’t help myself…), they are ineligible for being selected as POTF, useful and deserving as they may be. In fact, my cprintf utility, which was selected as POTW, was quickly deselected as POTW because of this very issue. MathWorks fears (and I can certainly understand the concern) that highlighting a utility that relies on some undocumented feature as POTW might be considered as an official endorsement of these features.
ScreenCapture is different in this regard: it uses purely documented Matlab functionality to achieve its aims, and apparently still succeeds in providing useful functionality. This does not mean that ScreenCapture uses pure Matlab. In fact, it relies on the Java Robot class‘s functionality of taking a screen-capture of a specified area of the screen. Using the Java Robot class in such a way is an entirely documented Matlab feature. I have discussed the Java Robot in two past articles on this blog, where guest blogger Kesh Ikuma explained (here and here) how it can be used to simulate mouse and keyboard actions programmatically.

Under the hood

ScreenCapture calculates the requested screen-capture rectangle coordinates and then invokes the Java Robot to take the actual bitmap screen-capture. The output is then converted into a Matlab image matrix for output, or stored in an image file, based on the user’s choice of parameters.
Some difficulties that I overcame when programming ScreenCapture:

• The screen position of docked windows cannot be computed reliably. Docked windows therefore need to be automatically temporarily undocked for screen-capture.
• Undocking in Windows 7 with Aero transparency features causes the Robot to take its screen-shot before the window becomes fully opaque. Adding a short delay in undocking solved this issue.
• Images use reversed Y-axis (Y=0 is at the axes top, not bottom). Also, when specifying a sub-region for capture, many users are used to handling images using data units (e.g., for imcrop) rather than ScreenCapture’s standard pixel units. Taking screen-captures of images proved to be a non-trivial challenge indeed.
• Different Matlab objects (controls, axes, figures) have different external borders and internal margins. I had to take these into account in order to achieve tight-fitting image captures of these objects.
• Performance was a problem, and it turned out that the bottleneck was trying to convert from the Java image data format to Matlab’s image data format. A couple of suggestions by Jan Simon and Urs (us) Schwartz significantly improved this performance hotspot. I’ve submitted the relevant code snippets to MathWorks for incorporation in a published technical solution, and I was happy to see that they have indeed incorporated them into that solution.
• Finally, I thought that adding a custom toolbar image would be a nice touch. Since I’m not much of an artist, creating the camera icon programmatically proved to be a bit of a challenge…

Please feel free to download ScreenCapture’s code and check how I chose to program around these issues.

TODO list

Some potentially-useful features have so far eluded me in ScreenCapture’s implementation. Perhaps one day I will find a way to do them:

1. Enable output of the image data, as an image object, to the system clipboard. It is easy to serialize the data and store it as a string in the clipboard, using the built-in clipboard function. But we would not be able to paste this data as an image into an external editor or image-processing utility. I have not yet found an easy way to store the image data as an object, although it should not be very difficult to do (here’s a starter).
2. When interactively selecting a screen-capture region, rbbox‘s starting point needs to be somewhere within the boundaries of a Matlab figure. The box can extend beyond the figure’s borders, but it has to start somewhere within the figure. I would like to be able to use rbbox without this limitation.

Addendum Jan 28. 2013: A new version of ScreenCapture was uploaded to the File Exchange today which appears to solve both of the TODO issues above: The copy-to-clipboard feature relies on Jiro Doke’s imclipboard utility as mentioned by Matt below — simply specify the ‘clipboard’ string as the capture target (rather than a filename); the solution of the rbbox-anywhere feature relies on using a temporary transparent window that spans the entire desktop area, capturing the user’s rbbox clicks anywhere within the desktop area. Interested readers can easily adapt the code to fit multiple monitors (I didn’t bother).

The post ScreenCapture utility appeared first on Undocumented Matlab.

jMouseEmu

The jMouseEmu utility is an interface function to simplify the programmatic execution of mouse actions (movements, clicks and wheel scrolls). jmouseemu features include accepting relative position with regards to a specified Handle Graphics control, supporting the Shift and Control click modifier keys, supporting double clicking and mouse dragging, controlling mouse wheel turns and allowing multiple command execution with interval timing specification.
jmouseemu has two operating modes: single- or multi-command. In this respect, a single jmouseemu command consists of a cursor position and a click option – multiple such commands can be specified to jmouseemu in the multi-command mode.
In the single-command mode, the function takes the cursor position information and click options. The click options can be any of the possible SelectionType property strings, or “none” if no clicking is desired. For example,

jmouseemu([500,250],'normal');

places the mouse cursor at the coordinate (500, 250) with respect to the current figure (in Matlab coordinates) and then clicks the left mouse button. If no figure is currently open, the cursor is placed at (500,250) with respect to the bottom-left corner of the screen.
The reference graphics handle can be added as the first argument. For example, if hEdit is a uicontrol editbox handle, then

jmouseemu(hEdit,[],'open');
jmouseemu(hEdit,[],'normal');

provides a simple way to select the entire text in the edit box: the double-click jmouseemu call selects the first word, then the following single-click jmouseemu extends the selection to the entire text. If the position argument omitted, the cursor is placed on the upper-left hand corner of the object, unless hEdit=0, in which case the cursor stays at the current position.
To turn the mouse wheel 600 notches down (towards you), run:

jmouseemu(-600,'wheel');

Multi-command mode extends the single-command mode by accepting a cell array of commands along with the update interval T in seconds. The command array must have three columns, and its k-th row specifies the mouse command at time (k–1)T second.
Using the multi-command mode, the above example can be executed with a single jmouseemu call, placing the commands in a cell array:

jmouseemu({h,[],'open'; [],[],'normal'},0);  % T=0

Here, if you omit handles (i.e., the first column of command array) for the second or later commands, it defaults to the handle of the previous command.
Multi-command mode enables mouse cursor to follow a predefined path. For example, to make the mouse cursor travel in a full circle:

T = 0.01; % update every 10 milliseconds
x = pi * (0:T:2)';
pos = 100*[cos(x),sin(x)] + 250;
N = numel(x);
cmds = cell(N,3); % empty click option -> no click
cmds{1,1} = gcf; % position w.r.t. current figure
cmds(:,2) = mat2cell(pos,ones(N,1),2);
jmouseemu(cmds,T);

The command interval is realized by calling robot.delay(T*1e3) between commands. This example updates the mouse position every 10 msecs and completes the circle in 2 seconds.
In multi-command mode, there are two additional click options, “drag_on” and “drag_off”, available to enable mouse dragging. Dragging must be initiated and terminated within a single jmouseemu call, and “drag_off” and “none” are the only two acceptable click options during dragging.
Lastly, to pause mouse movement for a moment (longer than the update interval) , a “delay” command may be used:

jmouseemu({...; [],1.0,'delay'; ...}, T);  % pause for 1 second

Using jMouseEmu to trigger multiple Matlab HG callbacks

jmouseemu (or more broadly the Matlab handling of Java Robot commands) has a major drawback: If jmouseemu is used to trigger a Matlab GUI callback during a sequence of mouse maneuver, the callback will not be executed until the entire maneuver is completed. Moreover, since SelectionType property is not logged for every callback event, the incorrect SelectionType property may be reported during callback execution.
For example, add a double click at the beginning of the circular movement:

cmds{:,3} = 'open';
jmouseemu(cmds,T);

Running this code, you can observe the figure’s WindowsButtonDownFcn callback, which has been set to echo SelectionType property from earlier, does not kick in until the circle is completed. Now, introduce “extend” click at the end of the circle:

cmds{end,3} = 'extend';
jmouseemu(cmds,T);

Again, no callback displaying SelectionType echo occurs during the mouse movement. When the callbacks are finally executed, they return in the order – “normal”, “extend”, “open” – instead of expected “normal”, “open”, “extend”. To correct this behavior, jmouseemu must be called twice:

jmouseemu(cmds(1,:),T);
jmouseemu(cmds(2:end,:),T);

The key here is for each jmouseemu call to have only one callback triggering mouse click command and to place it at the end of the command sequence.

inputEmu

The inputEmu utility is similar to jMouseEmu, with extended support for keyboard inputs. The support for relative mouse cursor position with respect to an HG object, however, has been turned off in the current version of inputEmu (it may get turned on in a future version). Position information in inputEmu is always given in pixels with respect to the lower-left corner of the main display screen. Also, the user must note that the input arguments are flipped between jMouseEmu and inputEmu.
In addition to the left mouse click commands from jMouseEmu, inputEmu can emulate the right and middle mouse button clicks. For example, to perform the right click (e.g., to open a context menu):

inputemu({'right_down' []; 'right_up' []}'); % execute right button click

inputEmu’s main feature is keyboard support. There are 4 basic commands (the behavior may be somewhat different on non-windows platforms):

• key_normal – Normal key press
• key_ctrl – CONTROL + key press
• key_alt – ALT + key press
• key_win – WIN + key press

One other obvious key press, SHIFT + key press, is conspicuously missing from this list. This is because the SHIFT key is automatically pressed when a SHIFT modified character (e.g., an upper-case alphabet) is entered. For example, executing

inputemu('key_normal','Hello World'); % emulates typing "Hello World"

directly in Matlab prompt displays “Hello World” in the next command line. The text is first decoded to a sequence of key press and release pairs, including SHIFT key presses and releases as needed, then the sequence of key commands are sent to the java Robot object.
In addition to all the characters on the keyboard, non-character keys can be stroked by using escape characters:

• ‘\SHIFT’
• ‘\CTRL’
• ‘\ALT’
• ‘\WINDOWS’
• ‘\BACKSPACE’
• ‘\TAB’
• ‘\ENTER’
• ‘\ESC’
• ‘\PAGEUP’
• ‘\PAGEDOWN’
• ‘\END’
• ‘\HOME’
• ‘\LEFT’
• ‘\UP’
• ‘\RIGHT’
• ‘\DOWN’
• ‘\INSERT’
• ‘\DELETE’
• ‘\CAPSLOCK’
• ‘\NUMLOCK’
• ‘\SCROLLLOCK’
• ‘\PRINTSCREEN’
• ‘\PA– USE’

Also, use ‘\\’ for the forward slash character and ‘\F1’ – ‘\F12’ for the Function keys. For example, to execute a Matlab “ver” command automatically via command window, run

inputemu('key_normal','ver\ENTER'); % runs the ver command

Lastly, as mentioned in the previous article, java.awt.Robot is sensitive to the keyboard locale, and the inputEmu utility is programmed specifically for the US QWERTY keyboard. To use the inputEmu for another keyboard layout, the program has to be modified accordingly. If anyone’s interested using this utility for different keyboard layout, I’d be glad to provide assistance.
Do you use Java’s Robot class or any of the utilities above in your work? Yair and I would love to hear about your experience in a short comment below.

The post GUI automation utilities appeared first on Undocumented Matlab.

java.awt.Robot class

The online Java documentation describes the purpose of the Robot class as follows:
This class is used to generate native system input events for the purposes of test automation, self-running demos, and other applications where control of the mouse and keyboard is needed.
This class has three main functionalities: mouse control, keyboard control, and screen capture. Here are some of the important member functions:

Mouse control functions

void mouseMove(int x, int y)
This function moves the cursor to the coordinate (x, y) which is defined with respect to the top-left screen corner (in contrast to Matlab’s coordinate origin at the bottom-left corner).
void mousePress(int buttons)
void mouseRelease(int buttons)
This pair of functions performs the button click. Their input argument is an OR’ed combination of java.awt.event.InputEvents:

java.awt.event.InputEvent.BUTTON1_MASK   // left mouse button
java.awt.event.InputEvent.BUTTON2_MASK   // middle mouse button
java.awt.event.InputEvent.BUTTON3_MASK   // right mouse button

Keyboard control functions

Keyboard action is emulated by the following pair of functions. Their keycodes are defined in java.awt.event.KeyEvent:
void keyPress(int keycode)
void keyRelease(int keycode)
NOTE: Although java.awt.event.KeyEvent constants defines most of the US QWERTY keys, not all can actually be used with java.awt.Robot. Specifically, it appears that only the KeyEvent constants for unmodified keys can be used. See the following section for an example.

Utility functions

The robot can be put to sleep for a desired duration (in milliseconds). Also, the calling routine can be blocked until the robot exhausts its command queue.
void delay(int ms)
void waitForIdle()
Readers interested in Robot’s screen-capture capability are invited to take a look at Yair’s ScreenCapture utility on the File Exchange.

Using java.awt.Robot in Matlab

To create the Robot object in Matlab, simply run

robot = java.awt.Robot;

Moving the mouse cursor

With Matlab’s root (0) handle, the mouse cursor can be repositioned by

set(0,'PointerLocation',[x y]);

The mouse cursor is placed on the x-th pixel from left edge and the y-th pixel from the bottom edge of the screen.
Alternately, we could use the semi-documented moveptr function, which was described here last year.
The same operation can also be done using Robot as follows:

scrSize = get(0,'ScreenSize');
robot.mouseMove(x,scrSize(2)-y);

The extra step must be taken to convert from Matlab to Java screen coordinates.
Depending on the specific case (for example, whether or not we know the absolute or only relative screen coordinates), we may prefer using any of these equivalent mouse-movement alternatives.

Clicking mouse buttons

Unfortunately, we have few alternatives for automating mouse-clicks – Robot is basically our only option. Matlab recognizes 4 different mouse click types as specified in the Figure’s SelectionType property:

• Normal: Click left mouse button
• Extend: SHIFT-click left mouse button
• Alternate: CTRL-click left mouse button
• Open: Double click left mouse button

To observe the mouse click, open a figure and set its WindowButtonDownFcn callback:

myCallback = @(hObj,event) disp(get(hObj,'SelectionType'));
set(gcf,'WindowButtonDownFcn', myCallback);

To trigger the callback, the figure must have the focus and the mouse cursor must be on the figure. To ensure the callback invocation, “figure(gcf); drawnow;” commands are included in the code examples below. Make sure to place the mouse cursor on the figure before running these codes.
Here’s how the Robot can be used to perform each type of clicking. First, normal click:

figure(gcf); drawnow;
robot.mousePress  (java.awt.event.InputEvent.BUTTON1_MASK);
robot.mouseRelease(java.awt.event.InputEvent.BUTTON1_MASK);

For “open” or double click, repeat it twice:

figure(gcf); drawnow;
robot.mousePress  (java.awt.event.InputEvent.BUTTON1_MASK);
robot.mouseRelease(java.awt.event.InputEvent.BUTTON1_MASK);
robot.mousePress  (java.awt.event.InputEvent.BUTTON1_MASK);
robot.mouseRelease(java.awt.event.InputEvent.BUTTON1_MASK);

For the other two click types, we need to use the keyboard functions. First, “extend” or SHIFT-click:

figure(gcf); drawnow;
robot.keyPress    (java.awt.event.KeyEvent.VK_SHIFT);
robot.mousePress  (java.awt.event.InputEvent.BUTTON1_MASK);
robot.mouseRelease(java.awt.event.InputEvent.BUTTON1_MASK);
robot.keyRelease  (java.awt.event.KeyEvent.VK_SHIFT);

Lastly, “alternate” or CTRL-click:

figure(gcf); drawnow;
robot.keyPress    (java.awt.event.KeyEvent.VK_CONTROL);
robot.mousePress  (java.awt.event.InputEvent.BUTTON1_MASK);
robot.mouseRelease(java.awt.event.InputEvent.BUTTON1_MASK);
robot.keyRelease  (java.awt.event.KeyEvent.VK_CONTROL);

Clicking keyboard keys

Just as we have shown above how to press a modifier key (Ctrl or Alt key) to set the figure’s SelectionType property, keyPress and keyRelease functions can be used with character keys to type text. For example, with the focus on Matlab’s Command Prompt, running the following code executes the ver command:

robot.keyPress    (java.awt.event.KeyEvent.VK_V);
robot.keyRelease  (java.awt.event.KeyEvent.VK_V);
robot.keyPress    (java.awt.event.KeyEvent.VK_E);
robot.keyRelease  (java.awt.event.KeyEvent.VK_E);
robot.keyPress    (java.awt.event.KeyEvent.VK_R);
robot.keyRelease  (java.awt.event.KeyEvent.VK_R);
robot.keyPress    (java.awt.event.KeyEvent.VK_ENTER);
robot.keyRelease  (java.awt.event.KeyEvent.VK_ENTER);

In the previous section, I mentioned that not all KeyEvent constants are supported by java.awt.Robot, and keys must be defined with their unmodified characters. For example, a semicolon (‘;’) can be typed by:

robot.keyPress    (java.awt.event.KeyEvent.VK_SEMICOLON);
robot.keyRelease  (java.awt.event.KeyEvent.VK_SEMICOLON);

However, doing the same for a colon (‘:’ or SHIFT-‘;’) causes an error:

robot.keyPress    (java.awt.event.KeyEvent.VK_COLON);
robot.keyRelease  (java.awt.event.KeyEvent.VK_COLON);
??? Java exception occurred:
java.lang.IllegalArgumentException: Invalid key code
    at sun.awt.windows.WRobotPeer.keyPress(Native Method)
    at java.awt.Robot.keyPress(Unknown Source)

Instead of using the VK_COLON constant, VK_SEMICOLON constant must be used while SHIFT modifier key is pressed:

robot.keyPress    (java.awt.event.KeyEvent.VK_SHIFT);
robot.keyPress    (java.awt.event.KeyEvent.VK_SEMICOLON);
robot.keyRelease  (java.awt.event.KeyEvent.VK_SEMICOLON);
robot.keyRelease  (java.awt.event.KeyEvent.VK_SHIFT);

Although unconfirmed, I suspect VK_SEMICOLON constant would not work with keyboards in which semicolon is mapped to SHIFT-‘,’ (e.g., Spanish or Italian keyboard layouts), while the VK_COLON constant would work with a French keyboard.

Putting it all together: jMouseEmu and inputEmu

To simplify the execution of the mouse and keyboard actions outlined above, I (Kesh Ikuma) have created the jMouseEmu and inputEmu utilities, which are available on the Matlab File Exchange. These interface/wrapper functions will be described in next week’s article.
Do you have any favorite use for Java Robots in your workflow? If so, please tell us all about it in a comment below.

The post GUI automation using a Robot appeared first on Undocumented Matlab.