Undocumented Matlab
  • SERVICES
    • Consulting
    • Development
    • Training
    • Gallery
    • Testimonials
  • PRODUCTS
    • IQML: IQFeed-Matlab connector
    • IB-Matlab: InteractiveBrokers-Matlab connector
    • EODML: EODHistoricalData-Matlab connector
    • Webinars
  • BOOKS
    • Secrets of MATLAB-Java Programming
    • Accelerating MATLAB Performance
    • MATLAB Succinctly
  • ARTICLES
  • ABOUT
    • Policies
  • CONTACT
  • SERVICES
    • Consulting
    • Development
    • Training
    • Gallery
    • Testimonials
  • PRODUCTS
    • IQML: IQFeed-Matlab connector
    • IB-Matlab: InteractiveBrokers-Matlab connector
    • EODML: EODHistoricalData-Matlab connector
    • Webinars
  • BOOKS
    • Secrets of MATLAB-Java Programming
    • Accelerating MATLAB Performance
    • MATLAB Succinctly
  • ARTICLES
  • ABOUT
    • Policies
  • CONTACT

File deletion memory leaks, performance

September 5, 2012 7 Comments

Last week I wrote about Matlab’s built-in pause function, that not only leaks memory but also appears to be less accurate than the equivalent Java function. Today I write about a very similar case. Apparently, using Matlab’s delete function not only leaks memory but is also slower than the equivalent Java function.

Memory leak

The memory leak in delete was (to the best of my knowledge) originally reported in the CSSM newsgroup and on this blog a few weeks ago. The reporter mentioned that after deleting 760K files using delete, he got a Java Heap Space out-of-memory error. The reported solution was to use the Java equivalent, java.io.File(filename).delete(), which does not leak anything.
I was able to recreate the report on my WinXP R2012a system, and discovered what appears to be a memory leak of ~150 bytes per file. This appears to be a very small number, but multiply by 760K (=111MB) and you can understand the problem. Of course, you can always increase the size of the Java heap used by Matlab (here’s how), but this should only be used as a last resort and certainly not when the solution is so simple.

For those interested, here’s the short test harness that I’ve used to test the memory leak:

function perfTest()
    rt = java.lang.Runtime.getRuntime;
    rt.gc();
    java.lang.Thread.sleep(1000);  % wait 1 sec to let the GC time to finish
    orig = rt.freeMemory;  % in bytes
    testSize = 50000;
    for idx = 1 : testSize
        % Create a temp file
        tn = [tempname '.tmp'];
        fid = fopen(tn,'wt');
        fclose(fid);
        % Delete the temp file
        delete(tn);
        %java.io.File(tn).delete();
    end
    rt.gc();
    java.lang.Thread.sleep(1000);  % wait 1 sec to let the GC time to finish
    free = rt.freeMemory;
    totalLeak = orig - free;
    leakPerCall = totalLeak / testSize
end

function perfTest() rt = java.lang.Runtime.getRuntime; rt.gc(); java.lang.Thread.sleep(1000); % wait 1 sec to let the GC time to finish orig = rt.freeMemory; % in bytes testSize = 50000; for idx = 1 : testSize % Create a temp file tn = [tempname '.tmp']; fid = fopen(tn,'wt'); fclose(fid); % Delete the temp file delete(tn); %java.io.File(tn).delete(); end rt.gc(); java.lang.Thread.sleep(1000); % wait 1 sec to let the GC time to finish free = rt.freeMemory; totalLeak = orig - free; leakPerCall = totalLeak / testSize end

I placed it in a function to remove command-prompt-generated fluctuations, but it must still be run several times to smooth the data. The main reason for the changes across runs is the fact that the Java heap is constantly growing and shrinking in a seesaw manner, and explicitly calling the garbage collector as I have done does not guarantee that it actually gets performed immediately or fully. By running a large-enough loop, and rerunning the test several times, the results become consistent due to the law of large numbers.
Running the test above with the delete line commented and the java.io.File line uncommented, shows no discernible memory leak.
To monitor Matlab’s Java heap space size in runtime, see my article from several months ago, or use Elmar Tarajan’s memory-monitor utility from the File Exchange.
Note: there are numerous online resources about Java’s garbage collector. Here’s one interesting article that I have recently come across.

Performance

When running the test function using java.io.File, we notice a significant speedup compared to running using delete. The reason is that (at least on my system, YMMV) delete takes 1.5-2 milliseconds to run while java.io.File only takes 0.4-0.5 ms. Again, this doesn’t seem like much, but multiply by thousands of files and it starts to be appreciable. For our 50K test harness, the difference translates into ~50 seconds, or 40% of the overall time.
Since we’re dealing with file I/O, it is important to run the testing multiple times and within a function (not the Matlab Command Prompt), to get rid of spurious measurement artifacts.
Have you encountered any other Matlab function, where the equivalent in Java is better? If so, please add a comment below.

Related posts:

  1. Matlab-Java memory leaks, performance – Internal fields of Java objects may leak memory - this article explains how to avoid this without sacrificing performance. ...
  2. Profiling Matlab memory usage – mtic and mtoc were a couple of undocumented features that enabled users of past Matlab releases to easily profile memory usage. ...
  3. Internal Matlab memory optimizations – Copy-on-write and in-place data manipulations are very useful Matlab performance improvement techniques. ...
  4. MLintFailureFiles or: Why can't I save my m-file?! – Sometimes Matlab gets into a state where it cannot use a valid m-file. This article explains what can be done. ...
  5. AppDesigner's mlapp file format – MLAPP files created by AppDesigner can be inspected and manipulated outside AppDesigner. ...
  6. Preallocation performance – Preallocation is a standard Matlab speedup technique. Still, it has several undocumented aspects. ...
Java Performance
Print Print
« Previous
Next »
7 Responses
  1. Jan Simon September 7, 2012 at 01:46 Reply

    In text you explain “*pause* takes 1.5-2 milliseconds to run while java.io.File only takes 0.4-0.5 ms”. Do you really mean *pause* or *delete*?

    • Yair Altman September 7, 2012 at 05:33 Reply

      @Jan – thanks, sharp eyes! (now corrected)

  2. Jeremy June 3, 2013 at 14:51 Reply

    Yair- Dug up this post while investigating a similar issue with:

    exist(name,'file')

    In the Matlab call of exist, they are caching the result as to make subsequent calls fast, but if you do large numbers of “exist” calls on files with different names (e.g. with constantly increasing index values in the file name), each one of those results is permanently cached. The result is a constant memory “leak” (well, technically memory is being put to good use, so not a leak in that sense, but there apparently no way to clear that cache and recover the memory). However, using:

    file.io.File(name).isFile

    works like a charm… AS LONG AS you don’t have the need to search the entire Matlab path. That is the ONE thing the Java call won’t do that the exist() call does.

  3. YoGabbaGabba April 17, 2015 at 13:40 Reply

    Do we know if this memory leak still exists in Matlab 2013b or newer versions?

    If so, if I want to override these files for a project, how do I avoid shadow warnings if I wish to keep the files named the same and on my path? Thanks

    • Yair Altman April 18, 2015 at 10:19 Reply

      @Bobby – the test script above should be fairly easy for you to run on your specific system to check this. And as noted, you can always use the java.io.File(tn).delete() workaround.

  4. stefan June 16, 2015 at 02:13 Reply

    i ve got a problem with saving matlab files (.mat). yesterday I overwrote my existing file. no errors occured. today, my file is almost empty (one of 4 folders left).
    is there ANY possibility to recover/restore?

    • Yair Altman June 16, 2015 at 04:42 Reply

      @Stefan – MAT files do not contain history (except if you designed them to contain it), so yyour only option is to check if this file was backed up.

Leave a Reply
HTML tags such as <b> or <i> are accepted.
Wrap code fragments inside <pre lang="matlab"> tags, like this:
<pre lang="matlab">
a = magic(3);
disp(sum(a))
</pre>
I reserve the right to edit/delete comments (read the site policies).
Not all comments will be answered. You can always email me (altmany at gmail) for private consulting.

Click here to cancel reply.

Useful links
  •  Email Yair Altman
  •  Subscribe to new posts (email)
  •  Subscribe to new posts (feed)
  •  Subscribe to new posts (reader)
  •  Subscribe to comments (feed)
 
Accelerating MATLAB Performance book
Recent Posts

Speeding-up builtin Matlab functions – part 3

Improving graphics interactivity

Interesting Matlab puzzle – analysis

Interesting Matlab puzzle

Undocumented plot marker types

Matlab toolstrip – part 9 (popup figures)

Matlab toolstrip – part 8 (galleries)

Matlab toolstrip – part 7 (selection controls)

Matlab toolstrip – part 6 (complex controls)

Matlab toolstrip – part 5 (icons)

Matlab toolstrip – part 4 (control customization)

Reverting axes controls in figure toolbar

Matlab toolstrip – part 3 (basic customization)

Matlab toolstrip – part 2 (ToolGroup App)

Matlab toolstrip – part 1

Categories
  • Desktop (45)
  • Figure window (59)
  • Guest bloggers (65)
  • GUI (165)
  • Handle graphics (84)
  • Hidden property (42)
  • Icons (15)
  • Java (174)
  • Listeners (22)
  • Memory (16)
  • Mex (13)
  • Presumed future risk (394)
    • High risk of breaking in future versions (100)
    • Low risk of breaking in future versions (160)
    • Medium risk of breaking in future versions (136)
  • Public presentation (6)
  • Semi-documented feature (10)
  • Semi-documented function (35)
  • Stock Matlab function (140)
  • Toolbox (10)
  • UI controls (52)
  • Uncategorized (13)
  • Undocumented feature (217)
  • Undocumented function (37)
Tags
ActiveX (6) AppDesigner (9) Callbacks (31) Compiler (10) Desktop (38) Donn Shull (10) Editor (8) Figure (19) FindJObj (27) GUI (141) GUIDE (8) Handle graphics (78) HG2 (34) Hidden property (51) HTML (26) Icons (9) Internal component (39) Java (178) JavaFrame (20) JIDE (19) JMI (8) Listener (17) Malcolm Lidierth (8) MCOS (11) Memory (13) Menubar (9) Mex (14) Optical illusion (11) Performance (78) Profiler (9) Pure Matlab (187) schema (7) schema.class (8) schema.prop (18) Semi-documented feature (6) Semi-documented function (33) Toolbar (14) Toolstrip (13) uicontrol (37) uifigure (8) UIInspect (12) uitools (20) Undocumented feature (187) Undocumented function (37) Undocumented property (20)
Recent Comments
  • Jianfei (31 days 0 hours ago): I have tried the MathWorks CheckBoxList in Matlab 2015b. For whatever the reason, I can’t change the font properties. I can change the font color, but changing font properties...
  • Donato Coppola (36 days 19 hours ago): Hi Yair, I have a problem with the double format. When I run the treeTable function, the numbers in double format cells are displayed with comma as decimal separator. How can...
  • Kim (42 days 9 hours ago): Yair, the following didn’t work for me either: jh.setBorderPainted(false); Regards, Kim
  • Adil (44 days 11 hours ago): Thank you for the blog, it was useful for me. I have a file named App_project.mlapp.zip and when I extract it through Winzip it gives all the files exactly as you described above. I...
  • Mr Ashley Trowell (46 days 21 hours ago): Thank you so much for this analysis. Also, I find it somewhat horrifying. The take away seems to be to use && / || and NOT and / or. Thanks a bunch! ~Ashley
  • Matt (51 days 22 hours ago): Late reply I know, but you can call custom shortcuts with alt-#. Hold down Alt to see what number is assigned to the shortcuts you’ve created. Now if there was a way to create a...
  • James (59 days 17 hours ago): Is there a way to change the location of the window? They pop up at random locations and the user has to chase them down.
  • Robin (60 days 1 hour ago): Hi Yair, This has been a great help. With the removal of com.mathworks sometime in the future (as announced in R2020b), is there an alternative? For me, just getting a preference...
  • Eric Dziekonski (81 days 23 hours ago): This is fantastic! Thanks everyone! To get access to CEF on 2019b, I had to take the advise of Jeffery Devereux and Xiangrui Li. warning off MATLAB:structOnObject % suppress...
  • Yair Altman (82 days 5 hours ago): jPanel1.setParent(jPanel2) will reparent jPanel1 (and similarly any Java component/container) inside jPanel2 (any Java container). While this might work, it seems to me to be an...
  • Eivind (82 days 19 hours ago): Hi, Thanks for a great function! I was trying to put the main panel of my figure into a scrollable panel with the goal of zooming into the panel using shift+scroll and scrolling up...
  • Yair Altman (97 days 6 hours ago): There is no direct way to replicate the CW in a GUI window, you’ll need to display the text in the GUI using your program logic. Here is one possible implementation that...
  • Sunki Reddy Gunugu (99 days 3 hours ago): hi 1.i would like to know how the error in command window of matlab is captured and displayed in the GUI with the same color in CW 2. Is their any way to replicate the...
  • Matthias Brenneis (101 days 22 hours ago): Hi, thank you so much for the inspiring work and comments! My issue is: As soon as the panel gets an axes, the panel is no longer transparent: f =...
  • Collin (121 days 19 hours ago): Yair, In 2020a Mathworks added this semi-useful class matlab.ui.internal.JavaMigrati onTools Which contains static methods to call javacomponent, actxcontrol or to get an...
Contact us
Undocumented Matlab © 2009 - Yair Altman
Scroll to top