ScreenCapture utility

Wednesday, August 15th, 2012

A few days ago, my ScreenCapture utility was selected as Matlab’s Pick of the Week (POTW). POTW selections are normally extremely useful, well-written and instructive utilities, which are both great to use in their own right, as well as a great source of knowledge about Matlab programming features and good practices. I follow the weekly POTW selections closely, and often learn new stuff from these utilities. I take pride in the fact that some of my utilities have been selected for inclusion in this unique set.

ScreenCapture enables Matlab users to take automated (programmatic) as well as interactive screen-captures of any Matlab GUI component or sub-region. This includes figure windows, axes, images, controls and even the Matlab Desktop. If the target handle for the capture is not specified, then ScreenCapture prompts the user to interactively select the capture region using an rbbox limiting box. ScreenCapture also includes a feature that plants a camera icon in the figure toolbar, such that clicking this icon will immediately trigger the interactive region-selection screen-capture.

Whichever manner the capture was made, the user then has the option of sending the output to Matlab (as a 3D RGB image matrix), or to one of the standard image file formats (e.g., JPG or PNG).

ScreenCapture has extensive help and is well-documented and relatively easy to use. For example:

figure; surf(peaks); imgData=screencapture(gcf); imshow(imgData);

ScreenCapture in action

ScreenCapture in action


Some additional usage examples:

imageData = screencapture;                     % interactively select screen-capture rectangle
imageData = screencapture(hListbox);           % capture image of a uicontrol
imageData = screencapture(0,  [20,30,40,50]);  % capture a small desktop sub-region
imageData = screencapture(gcf,[20,30,40,50]);  % capture a small figure sub-region
 
% capture a small sub-region of an axes
imageData = screencapture(gca,[10,20,30,40]);
imshow(imageData);  % display the captured image in a matlab figure
imwrite(imageData,'myImage.png');  % save the captured image to file
 
% capture a sub-region of an image
img = imread('cameraman.tif');
hImg = imshow(img);
screencapture(hImg,[60,35,140,80]);  % in data units, not pixel units
 
screencapture(gcf,[],'myFigure.jpg');                   % capture the entire figure into file
screencapture('handle',gcf,'filename','myFigure.jpg');  % same as previous
screencapture('toolbar',gcf);                           % adds a screen-capture button to gcf's toolbar
screencapture('toolbar',[],'file','sc.bmp');            % same, using a default output filename

Purely documented

Over the course of the past few years I have submitted 40 utilities to the Matlab File Exchange, several of which have been selected for POTW. Unfortunately, since most of my utilities employ undocumented Matlab features to some extent (I can’t help myself…), they are ineligible for being selected as POTF, useful and deserving as they may be. In fact, my cprintf utility, which was selected as POTW, was quickly deselected as POTW because of this very issue. MathWorks fears (and I can certainly understand the concern) that highlighting a utility that relies on some undocumented feature as POTW might be considered as an official endorsement of these features.

ScreenCapture is different in this regard: it uses purely documented Matlab functionality to achieve its aims, and apparently still succeeds in providing useful functionality. This does not mean that ScreenCapture uses pure Matlab. In fact, it relies on the Java Robot class‘s functionality of taking a screen-capture of a specified area of the screen. Using the Java Robot class in such a way is an entirely documented Matlab feature. I have discussed the Java Robot in two past articles on this blog, where guest blogger Kesh Ikuma explained (here and here) how it can be used to simulate mouse and keyboard actions programmatically.

Under the hood

ScreenCapture calculates the requested screen-capture rectangle coordinates and then invokes the Java Robot to take the actual bitmap screen-capture. The output is then converted into a Matlab image matrix for output, or stored in an image file, based on the user’s choice of parameters.

Some difficulties that I overcame when programming ScreenCapture:

  • The screen position of docked windows cannot be computed reliably. Docked windows therefore need to be automatically temporarily undocked for screen-capture.
  • Undocking in Windows 7 with Aero transparency features causes the Robot to take its screen-shot before the window becomes fully opaque. Adding a short delay in undocking solved this issue.
  • Images use reversed Y-axis (Y=0 is at the axes top, not bottom). Also, when specifying a sub-region for capture, many users are used to handling images using data units (e.g., for imcrop) rather than ScreenCapture’s standard pixel units. Taking screen-captures of images proved to be a non-trivial challenge indeed.
  • Different Matlab objects (controls, axes, figures) have different external borders and internal margins. I had to take these into account in order to achieve tight-fitting image captures of these objects.
  • Performance was a problem, and it turned out that the bottleneck was trying to convert from the Java image data format to Matlab’s image data format. A couple of suggestions by Jan Simon and Urs (us) Schwartz significantly improved this performance hotspot. I’ve submitted the relevant code snippets to MathWorks for incorporation in a published technical solution, and I was happy to see that they have indeed incorporated them into that solution.
  • Finally, I thought that adding a custom toolbar image would be a nice touch. Since I’m not much of an artist, creating the camera icon programmatically proved to be a bit of a challenge…

Please feel free to download ScreenCapture’s code and check how I chose to program around these issues.

TODO list

Some potentially-useful features have so far eluded me in ScreenCapture’s implementation. Perhaps one day I will find a way to do them:

  1. Enable output of the image data, as an image object, to the system clipboard. It is easy to serialize the data and store it as a string in the clipboard, using the built-in clipboard function. But we would not be able to paste this data as an image into an external editor or image-processing utility. I have not yet found an easy way to store the image data as an object, although it should not be very difficult to do (here’s a starter).
  2. When interactively selecting a screen-capture region, rbbox‘s starting point needs to be somewhere within the boundaries of a Matlab figure. The box can extend beyond the figure’s borders, but it has to start somewhere within the figure. I would like to be able to use rbbox without this limitation.

Addendum Jan 28. 2013: A new version of ScreenCapture was uploaded to the File Exchange today which appears to solve both of the TODO issues above: The copy-to-clipboard feature relies on Jiro Doke’s imclipboard utility as mentioned by Matt below — simply specify the ‘clipboard’ string as the capture target (rather than a filename); the solution of the rbbox-anywhere feature relies on using a temporary transparent window that spans the entire desktop area, capturing the user’s rbbox clicks anywhere within the desktop area. Interested readers can easily adapt the code to fit multiple monitors (I didn’t bother).

Related posts:

  1. GUI automation using a Robot – This article explains how Java's Robot class can be used to programmatically control mouse and keyboard actions...
  2. GUI automation utilities – This article explains a couple of Matlab utilities that use Java's Robot class to programmatically control mouse and keyboard actions...
  3. GUI integrated HTML panel – Simple HTML can be presented in a Java component integrated in Matlab GUI, without requiring the heavy browser control....
  4. Using spinners in Matlab GUI – Spinner controls can easily be added to Matlab GUI. This article explains how. ...
  5. Using JIDE combo-boxes – Matlab includes many custom JIDE combo-box controls that can be used in Matlab GUIs out of the box. ...
  6. Customizing listbox/combobox items – Matlab listboxes can be customized using custom Java cell-renderers. ...
Categories: GUI, Java, Low risk of breaking in future versions

Tags: , , , ,

Bookmark and SharePrint Print

35 Responses to ScreenCapture utility

  1. wei says:
    August 17, 2012 at 8:24 am

    it would be nice if tooltip, active menu/submenu, ect can be captured.

    Reply
    • Yair Altman says:
      August 20, 2012 at 7:50 am

      @wei – capturing tooltips and open menu items is of course impossible in interactive mode, but I’m pretty sure that it works ok in non-interactive (programmatic mode). Meaning, if a timer callback runs ScreenCapture when a menu-item is open or a tooltip is displayed, then I believe that it will indeed be captured.

  2. Matt says:
    August 28, 2012 at 2:18 pm

    Yair,

    In regards to your ‘ToDo #1’ Jiro Doke from Mathworks seems to have a solution
    http://www.mathworks.com/matlabcentral/fileexchange/28708-imclipboard/content/imclipboard.m

    Here’s the ‘paste’ code – the real transform seems to occur in when the ImageSelection object is created.

    % Import necessary Java classes
import java.awt.Toolkit.*
import java.awt.image.BufferedImage
import java.awt.datatransfer.DataFlavor
 
% Get System Clipboard object (java.awt.Toolkit)
cb = getDefaultToolkit.getSystemClipboard();
 
% Get image size
ht = size(data, 1); 
wd = size(data, 2);
 
% Convert to Blue-Green-Red format
data = data(:, :, [3 2 1]);
 
% Convert to 3xWxH format
data = permute(data, [3, 2, 1]);
 
% Append Alpha data (not used)
data = cat(1, data, 255*ones(1, wd, ht, 'uint8'));
 
% Create image buffer
imBuffer = BufferedImage(wd, ht, BufferedImage.TYPE_INT_RGB);
imBuffer.setRGB(0, 0, wd, ht, typecast(data(:), 'int32'), 0, wd);
 
% Create ImageSelection object
%    % custom java class
imSelection = ImageSelection(imBuffer);
 
% Set clipboard content to the image
cb.setContents(imSelection, []);

    Thanks so much for all your amazing matlab solutions.

    Reply
    • Yair Altman says:
      August 28, 2012 at 4:23 pm

      @Matt – thanks, I missed that one :-)

    • Matt says:
      August 29, 2012 at 5:16 pm

      Yair – On the topic of matlab clipboard interfaces- is there an easy way in these situations to check if the instance of matlab running the paste code has lost ownership of the system clipboard?

      The ClipboardOwner interface has the method lostOwnership, but I’ve had difficulties figuring out how to use that method within the Matlab event handler. It would be sufficient to even be able to check the owner without a callback.
      I currently use the FlavorsChangedCallback as a workaround to try to make automated plot generation more robust – but the ideal would be to know if the clipboard owner changed between the ‘copy’ function, and the ‘paste’ function.

  3. manoj kumar says:
    July 30, 2014 at 1:20 pm

    I ran your screencapture.m im my computer and just included the following code to capture the image in the figure and it ran successfully.

    imgData=screencapture(gcf,[790,275,750,600]);

    But when I ran my code in another computer where I didn’t run your screencapture.m, i am facing error as :
    Undefined function ‘screencapture’ for input arguments of
    type ‘double’.

    Error in manual_thresh>done (line 550)

    imgData=screencapture(gcf,[790,275,750,600]);

    Error while evaluating uicontrol Callback

    what is the code that i need to include in my GUI code, so that it can run in any computer with running your screencapture.m file.

    Thanks…

    Reply
    • Yair Altman says:
      August 6, 2014 at 3:11 am

      @Manoj – You need to include the screencapture.m file together with your code, otherwise the other computer will not know where this function is located (duh!).

  4. Hoi Wong says:
    September 18, 2014 at 3:37 pm

    With ScreenCapture utility, is there a way to capture a figure without the borders and toolbar, title, etc?

    This tool is awesome and super useful since TMW never managed to do ‘What You See Is What You Print (or export to EPS,PDF for publishing): fonts and text always gets changed, shifted around and multi-grid legend became an uncontrollable mess once I export/print. Thanks a lot for the great utility!

    Reply
    • Yair Altman says:
      September 18, 2014 at 4:14 pm

      @Hoi – thanks. Please remember to add your feedback/vote to the FEX page.

      You can run ScreenCapture on the figure’s top-level panel handle, assuming you have such a panel in your figure. ScreenCapture will then only take a screenshot of the panel and its contents, nothing outside it.

    • Hoi Wong says:
      September 18, 2014 at 4:34 pm

      Thanks. Just did that.

      For simple figures done by plot(), do they have a top level panel handle? All the child objects are of the type ‘axes’.

    • Yair Altman says:
      September 18, 2014 at 4:36 pm

      Best thing is to have a top-level panel. If you don’t have it, try pointing ScreenCapture at the axes handle, but it may perhaps not be accurate because of the labels/tick-marks/title.

    • Hoi Wong says:
      September 18, 2014 at 5:00 pm

      Thanks for the reply. I have simple figures done by subplot() and plot() commands, so just pointing at one axes handle won’t work. Can you point me to a easy way to retroactively add a top-level panel to natively generated figures?

      The reason I’m using the tool is for PowerPoint presentations, where the resolution doesn’t matter more than what I already seen on the screen because I am not going to zoom in the details. Having to add a uipanel then add the axes under it will be too much logistics.

      I’m using Greenshot for regular screen capture, which I can use the mouse to select a window to capture (with or without toolbars+title bar). Just wondering if there’s a way to do that programmatically in MATLAB.

    • Yair Altman says:
      September 18, 2014 at 5:31 pm

      Of course it’s possible, it’s just that ScreenCapture doesn’t currently have this feature and I have no plans to add it in the near future. You’ll need to add a top-level uipanel. It doesn’t strike me as being such a great hassle.

    • Yair Altman says:
      September 18, 2014 at 5:39 pm

      Alternatively, use ScreenCapture’s ability to specify a pixel position vector (second input arg) and use the figure’s size to determine the position vector’s 3rd and 4th values. Something like [2,2,550,400].

    • Hoi Wong says:
      September 18, 2014 at 6:05 pm

      Thanks. I’ll try to figure out how to retroactively put the plotted figures into an uipanel. I’m not too used to it other than laying out a GUI. That’s why I thought it’s quite a hassle especially when I am trying to capture the plot windows generated by other people’s code. I’ll let you know if I figure out an adapter for ScreenCapture that does the trick :)

  5. Pingback: Another couple of Matlab bugs and workarounds | Undocumented Matlab

  6. Marc Lalancette says:
    September 4, 2015 at 2:55 pm

    Can you clarify the ‘position’ argument? When I supply get(gcf, ‘Position’), the screen capture is not in the right spot. And the help makes it sound like that’s the default… Like Hoi, I wanted to capture the inside of the figure window (which is a gui with 3 axes and other controls, but no “top-level pannel”). Thanks!

    Reply
    • Yair Altman says:
      September 5, 2015 at 10:31 am

      The position argument is relative to the bottom-left corner of the specified handle. For example, a value of [10,20,30,40] means a 30×40-pixel rectangle that starts at an offset of x=10, y=20 from the bottom-left corner of the specified handle.

    • Marc Lalancette says:
      September 9, 2015 at 2:50 pm

      Thanks so much! Thus using [8, 8, FigureWidth, FigureHeight] worked perfectly for me on Windows 7. Such a shame there isn’t any way to properly save a figure in “Matlab-provided” functions.

  7. Krishna says:
    September 8, 2015 at 3:30 am

    Hi Yair, I am using your ScreenCapture code and it is very useful for me. The only problem I am facing is resolution of the image. Is there any way to enhance the same?
    Thanks and best regards, Krishna

    Reply
  8. Klaus says:
    December 1, 2015 at 8:58 am

    Hello Yair,

    great tool, thanks! I’m facing one issue: Stand-alone applications using screencapture crash with the following error: “Undefined function or variable ‘ImageSelection’. Do you have any idea on how to fix this?

    Thank you and best regards

    Klaus

    Reply
    • Yair Altman says:
      December 1, 2015 at 9:03 am

      @Klaus – ImageSelection is a Java class that is part of the ScreenCapture utility’s zip file. The ImageSelection.class file should be added to your deployment project so that it would be included in the distributable, otherwise it will not be there when the program attempts to call it, generating the error that you see.

    • Klaus says:
      December 1, 2015 at 9:40 am

      Hello Yair,

      I simply copied the ‘ImageSelection.class’ and the ‘ImageSelection.java’ files both to the deployment folder and to the folder containing the deployed m-file. The compiled program still crashes with the same error. I googled some other articles on things like “javaaddpath”. Do you know if need to tell Matlab where to find the files? I’m using deploytool to compile the program.

      Thank you and best regards

      Klaus

    • Klaus says:
      December 2, 2015 at 1:20 am

      Hello Yair,

      I just added

      if isdeployed
    folder=pwd;
    folder(folder=='\')='/';
    javaaddpath([folder],'-end')
end

      and now it works. It just gives some warnings, that “testmeas.jar”, “instrument.jar” and “images.jar” are already specified on static java path.

      Many thanks

      Klaus

  9. Rocco says:
    July 14, 2016 at 5:33 pm

    Yair,
    I am new to this community and I have been trying to do exactly what your ScreenCapture utility does, with one important twist. Is there a way to capture a non-matlab figure with the mouse and bring it into the matlab workspace automatically without having to save and import the data? Say I am drawing on a pen-tablet and want to select/capture a region of the drawing and bring it into matlab programmatically as rgb pixels. Is this possible? I have attempted to hire someone on freelancer to do this without luck. If it can be done in Matlab, let’s talk $ for a stand-alone executable. Thanks!

    Reply
    • Yair Altman says:
      July 16, 2016 at 11:35 pm

      @Rocco – the ScreenCapture utility returns an imageData array that you can then use in Matlab, without needing to save and load from file. Read the utility’s documentation carefully for details. Contact me offline (email) if you still need additional consulting.

  10. Kurt says:
    July 27, 2016 at 12:14 am

    When I run this code I get a black box and not the screen capture. Is there something I need or am doing wrong? I have both files in the same directory. Thanks.

    Reply
  11. Rocco says:
    July 27, 2016 at 8:00 pm

    Hi Yair, thanks for the reply. Very useful utility especially when used with frameless (undecorated) figure windows you mention above !

    Reply
  12. Peter Newton says:
    August 9, 2016 at 3:21 pm

    Hi,
    Many thanks for a great screenshot utility. I have a question in that I want to use a variable fname2 that will have a difference file name each time I use it. The problem I am having is I can’t get the export_fig script to take the fname2 variable with the size of the screenshot to work together. I have tried putting the fname2 along with the screenshot variables into one variable, it does run it but includes the screenshot variables in the file name and ignores them. Does anyone have any ideal how I can get this to work, below is an example of what I am trying,

    export_fig (fname3) -m2.5 -c[05 465 250 65] ;

    regards
    Peter

    Reply
  13. Guillaume says:
    October 27, 2016 at 8:56 am

    Hi Yair,

    I successfully use screencapture along with imwrite to save a screenshot of a maximized figure, however I can see some funny results on the bottom and left edges of the saved .png.

    See https://cloud.githubusercontent.com/assets/3498686/19756046/b72e277c-9c65-11e6-901a-a862f3ef5fdc.png

    Code:

    % preserve the color scheme
set(hFig, 'InvertHardcopy', 'off');
 
drawnow;
 
% force figure full screen
try
    warning('off', 'MATLAB:HandleGraphics:ObsoletedProperty:JavaFrame');
    frame_h = get(handle(hFig), 'JavaFrame');
    set(frame_h, 'Maximized', 1);
catch
    disp(['Warning : JavaFrame feature not supported. Figure is going full ' ...
        'screen using normalised units which does not give best results.']);
    oldUnits = get(hFig, 'Units');
    set(hFig, 'Units', 'norm', 'Pos', [0, 0, 1, 1]);
    set(hFig, 'Units', oldUnits);
end
 
% screencapture creates an image of what is really displayed on
% screen.
imwrite(screencapture(hPanel), fileDestination);
    Reply

  14. Pingback: Note: export_fig File Exchange Utility – Nguyen Quan Ba Hong

  15. Sajad Rahmdel says:
    December 26, 2018 at 10:30 am

    Thank you

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *