Comments on: Improving save performance

By: Yair Altman

Yair Altman — Wed, 20 May 2015 15:18:24 +0000

In reply to CaptainObv. Not directly, but you could try using the low-level HDF5 functions for this.

By: CaptainObv

CaptainObv — Wed, 20 May 2015 15:16:11 +0000

Is it possible to save matfiles without creating the initial variable at the beginning?

I am continually adding output data to my matfile block by block (I.e continually expanding its dimensions). I want to speed this up and I considered preallocating the space within the matfile (as you do with regular arrays before a loop).

However my initial variable will be quite large (hence why I am block processing), and I don’t want to risk the possibility of a memory error, so ideally I’d like To create the Matfile without Saving the variable to matlab memory as you did with your A and B variables.

Is there a way?

By: jkoether

jkoether — Thu, 22 Jan 2015 18:35:43 +0000

For me, the biggest advantage of savefast is that by saving uncompressed, a specific portion of an array can be loaded MUCH faster when using the matfile function. There seems to be some decompression overhead that is a function of the array size regardless of how little data you are asking for. For example, with a 250Mb file just using the save function, it take a minimum of 2 seconds to extract data, even if I am only loading 10k elements. If the file is created by savefast it only takes .02 seconds to load 10k elements. This will be a huge improvement for my software.

By: Hugo

Hugo — Mon, 30 Jun 2014 05:47:01 +0000

Hello,
I’ve just tried to save a figure (around 2e4 double data points) with -v7.3 in R2014a, and it resulted in 100Mb file size instead of 1.4Mb with default -v7. A new bug maybe?

By: Roger

Roger — Wed, 16 Apr 2014 15:27:22 +0000

What about the load function? Inside a function, I found that using assignin(‘base’, variables) is faster than using evalin(‘base’, load .mat containing the variables). I thought loading the variables from a .mat file would be faster than using the assignin command. Would really like to see the same kind of topic but on the load command.

By: Explicit multi-threading in Matlab – part 1 | Undocumented Matlab

Explicit multi-threading in Matlab – part 1 | Undocumented Matlab — Wed, 19 Feb 2014 20:20:30 +0000

[…] to be too slow for our specific needs. We could perhaps improve it a bit with some fancy tricks for save or fwrite. But let’s take a different approach today, using multi-threading:Using Java […]

By: Serializing/deserializing Matlab data | Undocumented Matlab

Serializing/deserializing Matlab data | Undocumented Matlab — Wed, 22 Jan 2014 19:58:05 +0000

[…] Last year I wrote an article on improving the performance of the save function. The article discussed various ways by which we can store Matlab data on disk. […]

By: How to store large datasets? | Matlabtips.com

How to store large datasets? | Matlabtips.com — Mon, 23 Dec 2013 07:04:53 +0000

[…] you need to optimizing read/write speed when dealing with MAT/HDF5 files, I encourage you to read Yair’s excellent post on that […]

By: Robin

Robin — Mon, 17 Jun 2013 16:41:43 +0000

Are you sure Matlab doesn’t compress structures/cells saved in v7.3?

I have exactly the opposite problem that it is not able to disable compression, which in v7.3 provides almost no space benefit for my data at the cost of extremely long save and load times.

When I save structures/cell arrays compression is still used. I think compression kicks in for any numeric object which is greater than 10000 bytes (so if you have large structure or cell contents I think they will still be compressed).

The serialization trick is really interesting, thanks!

By: How to store large datasets? | Matlabtips.com

How to store large datasets? | Matlabtips.com — Wed, 15 May 2013 12:02:42 +0000

[…] If you need to optimizing read/write speed when dealing with MAT/HDF5 files, I encourage you to read Yair’s excellent post on that topic. […]