ScreenCapture utility

A few days ago, my ScreenCapture utility was selected as Matlab’s Pick of the Week (POTW). POTW selections are normally extremely useful, well-written and instructive utilities, which are both great to use in their own right, as well as a great source of knowledge about Matlab programming features and good practices. I follow the weekly POTW selections closely, and often learn new stuff from these utilities. I take pride in the fact that some of my utilities have been selected for inclusion in this unique set.

ScreenCapture enables Matlab users to take automated (programmatic) as well as interactive screen-captures of any Matlab GUI component or sub-region. This includes figure windows, axes, images, controls and even the Matlab Desktop. If the target handle for the capture is not specified, then ScreenCapture prompts the user to interactively select the capture region using an rbbox limiting box. ScreenCapture also includes a feature that plants a camera icon in the figure toolbar, such that clicking this icon will immediately trigger the interactive region-selection screen-capture.

Whichever manner the capture was made, the user then has the option of sending the output to Matlab (as a 3D RGB image matrix), or to one of the standard image file formats (e.g., JPG or PNG).

ScreenCapture has extensive help and is well-documented and relatively easy to use. For example:

figure; surf(peaks); imgData=screencapture(gcf); imshow(imgData);

ScreenCapture in action

ScreenCapture in action


Some additional usage examples:

imageData = screencapture;                     % interactively select screen-capture rectangle
imageData = screencapture(hListbox);           % capture image of a uicontrol
imageData = screencapture(0,  [20,30,40,50]);  % capture a small desktop sub-region
imageData = screencapture(gcf,[20,30,40,50]);  % capture a small figure sub-region
 
% capture a small sub-region of an axes
imageData = screencapture(gca,[10,20,30,40]);
imshow(imageData);  % display the captured image in a matlab figure
imwrite(imageData,'myImage.png');  % save the captured image to file
 
% capture a sub-region of an image
img = imread('cameraman.tif');
hImg = imshow(img);
screencapture(hImg,[60,35,140,80]);  % in data units, not pixel units
 
screencapture(gcf,[],'myFigure.jpg');                   % capture the entire figure into file
screencapture('handle',gcf,'filename','myFigure.jpg');  % same as previous
screencapture('toolbar',gcf);                           % adds a screen-capture button to gcf's toolbar
screencapture('toolbar',[],'file','sc.bmp');            % same, using a default output filename

Purely documented

Over the course of the past few years I have submitted 40 utilities to the Matlab File Exchange, several of which have been selected for POTW. Unfortunately, since most of my utilities employ undocumented Matlab features to some extent (I can’t help myself…), they are ineligible for being selected as POTF, useful and deserving as they may be. In fact, my cprintf utility, which was selected as POTW, was quickly deselected as POTW because of this very issue. MathWorks fears (and I can certainly understand the concern) that highlighting a utility that relies on some undocumented feature as POTW might be considered as an official endorsement of these features.

ScreenCapture is different in this regard: it uses purely documented Matlab functionality to achieve its aims, and apparently still succeeds in providing useful functionality. This does not mean that ScreenCapture uses pure Matlab. In fact, it relies on the Java Robot class‘s functionality of taking a screen-capture of a specified area of the screen. Using the Java Robot class in such a way is an entirely documented Matlab feature. I have discussed the Java Robot in two past articles on this blog, where guest blogger Kesh Ikuma explained (here and here) how it can be used to simulate mouse and keyboard actions programmatically.

Under the hood

ScreenCapture calculates the requested screen-capture rectangle coordinates and then invokes the Java Robot to take the actual bitmap screen-capture. The output is then converted into a Matlab image matrix for output, or stored in an image file, based on the user’s choice of parameters.

Some difficulties that I overcame when programming ScreenCapture:

  • The screen position of docked windows cannot be computed reliably. Docked windows therefore need to be automatically temporarily undocked for screen-capture.
  • Undocking in Windows 7 with Aero transparency features causes the Robot to take its screen-shot before the window becomes fully opaque. Adding a short delay in undocking solved this issue.
  • Images use reversed Y-axis (Y=0 is at the axes top, not bottom). Also, when specifying a sub-region for capture, many users are used to handling images using data units (e.g., for imcrop) rather than ScreenCapture’s standard pixel units. Taking screen-captures of images proved to be a non-trivial challenge indeed.
  • Different Matlab objects (controls, axes, figures) have different external borders and internal margins. I had to take these into account in order to achieve tight-fitting image captures of these objects.
  • Performance was a problem, and it turned out that the bottleneck was trying to convert from the Java image data format to Matlab’s image data format. A couple of suggestions by Jan Simon and Urs (us) Schwartz significantly improved this performance hotspot. I’ve submitted the relevant code snippets to MathWorks for incorporation in a published technical solution, and I was happy to see that they have indeed incorporated them into that solution.
  • Finally, I thought that adding a custom toolbar image would be a nice touch. Since I’m not much of an artist, creating the camera icon programmatically proved to be a bit of a challenge…

Please feel free to download ScreenCapture’s code and check how I chose to program around these issues.

TODO list

Some potentially-useful features have so far eluded me in ScreenCapture’s implementation. Perhaps one day I will find a way to do them:

  1. Enable output of the image data, as an image object, to the system clipboard. It is easy to serialize the data and store it as a string in the clipboard, using the built-in clipboard function. But we would not be able to paste this data as an image into an external editor or image-processing utility. I have not yet found an easy way to store the image data as an object, although it should not be very difficult to do (here’s a starter).
  2. When interactively selecting a screen-capture region, rbbox‘s starting point needs to be somewhere within the boundaries of a Matlab figure. The box can extend beyond the figure’s borders, but it has to start somewhere within the figure. I would like to be able to use rbbox without this limitation.

Addendum Jan 28. 2013: A new version of ScreenCapture was uploaded to the File Exchange today which appears to solve both of the TODO issues above: The copy-to-clipboard feature relies on Jiro Doke’s imclipboard utility as mentioned by Matt below — simply specify the ‘clipboard’ string as the capture target (rather than a filename); the solution of the rbbox-anywhere feature relies on using a temporary transparent window that spans the entire desktop area, capturing the user’s rbbox clicks anywhere within the desktop area. Interested readers can easily adapt the code to fit multiple monitors (I didn’t bother).

Related posts:

  1. An interesting uitree utility ExploreStruct is a utility that shows how custom uitrees can be integrated in Matlab GUI...
  2. GUI automation using a Robot This article explains how Java's Robot class can be used to programmatically control mouse and keyboard actions...
  3. GUI automation utilities This article explains a couple of Matlab utilities that use Java's Robot class to programmatically control mouse and keyboard actions...
  4. Splash window for deployed applications Deployed (compiled) Matlab applications take a long time to load. I present a splash window that loads immadiately, solving this problem. ...
  5. Figure toolbar components Matlab's toolbars can be customized using a combination of undocumented Matlab and Java hacks. This article describes how to access existing toolbar icons and how to add non-button toolbar components....
  6. uitree This article describes the undocumented Matlab uitree function, which displays data in a GUI tree component...

Categories: GUI, Java, Low risk of breaking in future versions

Tags: , , , ,

Bookmark and SharePrint Print

5 Responses to ScreenCapture utility

  1. wei says:

    it would be nice if tooltip, active menu/submenu, ect can be captured.

    • @wei – capturing tooltips and open menu items is of course impossible in interactive mode, but I’m pretty sure that it works ok in non-interactive (programmatic mode). Meaning, if a timer callback runs ScreenCapture when a menu-item is open or a tooltip is displayed, then I believe that it will indeed be captured.

  2. Matt says:

    Yair,

    In regards to your ‘ToDo #1′ Jiro Doke from Mathworks seems to have a solution
    http://www.mathworks.com/matlabcentral/fileexchange/28708-imclipboard/content/imclipboard.m

    Here’s the ‘paste’ code – the real transform seems to occur in when the ImageSelection object is created.

    % Import necessary Java classes
    import java.awt.Toolkit.*
    import java.awt.image.BufferedImage
    import java.awt.datatransfer.DataFlavor
     
    % Get System Clipboard object (java.awt.Toolkit)
    cb = getDefaultToolkit.getSystemClipboard();
     
    % Get image size
    ht = size(data, 1); 
    wd = size(data, 2);
     
    % Convert to Blue-Green-Red format
    data = data(:, :, [3 2 1]);
     
    % Convert to 3xWxH format
    data = permute(data, [3, 2, 1]);
     
    % Append Alpha data (not used)
    data = cat(1, data, 255*ones(1, wd, ht, 'uint8'));
     
    % Create image buffer
    imBuffer = BufferedImage(wd, ht, BufferedImage.TYPE_INT_RGB);
    imBuffer.setRGB(0, 0, wd, ht, typecast(data(:), 'int32'), 0, wd);
     
    % Create ImageSelection object
    %    % custom java class
    imSelection = ImageSelection(imBuffer);
     
    % Set clipboard content to the image
    cb.setContents(imSelection, []);

    Thanks so much for all your amazing matlab solutions.

    • @Matt – thanks, I missed that one :-)

    • Matt says:

      Yair – On the topic of matlab clipboard interfaces- is there an easy way in these situations to check if the instance of matlab running the paste code has lost ownership of the system clipboard?

      The ClipboardOwner interface has the method lostOwnership, but I’ve had difficulties figuring out how to use that method within the Matlab event handler. It would be sufficient to even be able to check the owner without a callback.
      I currently use the FlavorsChangedCallback as a workaround to try to make automated plot generation more robust – but the ideal would be to know if the clipboard owner changed between the ‘copy’ function, and the ‘paste’ function.

Leave a Reply

Your email address will not be published. Required fields are marked *

*

<pre lang="matlab">
a = magic(3);
sum(a)
</pre>