- Undocumented Matlab - https://undocumentedmatlab.com -

Faster csvwrite/dlmwrite

Posted By Yair Altman On October 3, 2017 | 5 Comments

Matlab’s builtin functions for exporting (saving) data to output files are quite sub-optimal (as in slowwwwww…). I wrote a few posts about this in the past (how to improve fwrite performance [1], and save performance [2]). Today I extend the series by showing how we can improve the performance of delimited text output, for example comma-separated (CSV) or tab-separated (TSV/TXT) files.
The basic problem is that Matlab’s dlmwrite function, which can either be used directly, or via the csvwrite function which calls it internally, is extremely inefficient: It processes each input data value separately, in a non-vectorized loop. In the general (completely non-vectorized) case, each data value is separately converted into a string, and is separately sent to disk (using fprintf). In the specific case of real data values with simple delimiters and formatting, row values are vectorized, but in any case the rows are processed in a non-vectorized loop: A newline character is separately exported at the end of each row, using a separate fprintf call, and this has the effect of flushing the I/O to disk each and every row separately, which is of course disastrous for performance. The output file is indeed originally opened in buffered mode (as I explained in my fprintf performance [1] post), but this only helps for outputs done within the row – the newline output at the end of each row forces an I/O flush regardless of how the file was opened. In general, when you read the short source-code of dlmwrite.m you’ll get the distinct feeling that it was written for correctness and maintainability, and some focus on performance (e.g., the vectorization edge-case). But much more could be done for performance it would seem.
This is where Alex Nazarovsky comes to the rescue.

Alex was so bothered by the slow performance of csvwrite and dlmwrite that he created a C++ (MEX) version that runs about enormously faster (30 times faster on my system). He explains the idea [3] in his blog, and posted it as an open-source utility (mex-writematrix [4]) on GitHub.
Usage of Alex’s utility is very easy:

mex_WriteMatrix(filename, dataMatrix, textFormat, delimiter, writeMode);

where the input arguments are:

  • filename – full path name for file to export
  • dataMatrix – matrix of numeric values to be exported
  • textFormat – format of output text (sprintf format), e.g. '%10.6f'
  • delimiter – delimiter, for example ',' or ';' or char(9) (=tab)
  • writeMode – 'w+' for rewriting file; 'a+' for appending (note the lowercase: uppercase will crash Matlab!)

Here is a sample run on my system, writing a simple CSV file containing 1K-by-1K data values (1M elements, ~12MB text files):

>> data = rand(1000, 1000);  % 1M data values, 8MB in memory, ~12MB on disk
>> tic, dlmwrite('temp1.csv', data, 'delimiter',',', 'precision','%10.10f'); toc
Elapsed time is 28.724937 seconds.
>> tic, mex_WriteMatrix('temp2.csv', data, '%10.10f', ',', 'w+'); toc   % 30 times faster!
Elapsed time is 0.957256 seconds.

Alex’s mex_WriteMatrix function is faster even in the edge case of simple formatting where dlmwrite uses vectorized mode (in that case, the file is exported in ~1.2 secs by dlmwrite and ~0.9 secs by mex_WriteMatrix, on my system).

Trapping Ctrl-C interrupts

Alex’s mex_WriteMatrix code includes another undocumented trick that could help anyone else who uses a long-running MEX function, namely the ability to stop the MEX execution using Ctrl-C. Using Ctrl-C is normally ignored in MEX code, but Wotao Yin showed [5] how we can use the undocumented utIsInterruptPending() MEX function to monitor for user interrupts using Ctrl-C. For easy reference, here is a copy of Wotao Yin’s usage example (read his webpage [5] for additional details):

/* A demo of Ctrl-C detection in mex-file by Wotao Yin. Jan 29, 2010. */
#include "mex.h"
#if defined (_WIN32)
#elif defined (__linux__)
#ifdef __cplusplus
    extern "C" bool utIsInterruptPending();
    extern bool utIsInterruptPending();
void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[]) {
    int count = 0;
    while(1) {
        #if defined(_WIN32)
            Sleep(1000);        /* Sleep one second */
        #elif defined(__linux__)
            usleep(1000*1000);  /* Sleep one second */
        mexPrintf("Count = %d\n", count++);  /* print count and increase it by 1 */
        mexEvalString("drawnow;");           /* flush screen output */
        if (utIsInterruptPending()) {        /* check for a Ctrl-C event */
            mexPrintf("Ctrl-C Detected. END\n\n");
        if (count == 10) {
            mexPrintf("Count Reached 10. END\n\n");

Matlab performance webinars

I am offering a couple of webinars about various ways to improve Matlab’s run-time performance:

Both the webinar videos and their corresponding slide-decks are available for download. The webinars content is based on onsite training courses that I presented at multiple client locations (details [12]).
Additional Matlab performance tips can be found under the Performance tag [16] in this blog, as well as in my book “Accelerating MATLAB Performance [17]“.
 Email me [18] if you would like additional information on the webinars, or an onsite training course, or about my consulting.

Categories: Low risk of breaking in future versions, Mex, Undocumented feature

5 Comments (Open | Close)

5 Comments To "Faster csvwrite/dlmwrite"

#1 Comment By AndrewV On October 3, 2017 @ 20:04

Any resources out there for faster csvread (or any other text parsing) functions? I believe there were a few on file exchange but I had no luck with them, either from minimal speed improvements or functions that just do not work.

#2 Comment By Yair Altman On October 3, 2017 @ 21:09

@Andrew – my advise is to read the entire CSV file into memory (a long string) using fread and then parse the string, for example using textscan. This works well with files that are not huge; with huge files you may want to read them in chunks.

#3 Comment By Michal Kvasnicka On October 4, 2017 @ 12:07

@Yair – thanks for very interesting post!

Any examples how to read/write HUGE text files (CSV) by fread+textscan in chunks?

#4 Comment By Saket On October 5, 2017 @ 10:52

@Andrew, @Yair – Try out the Mex Version of CSV Reader from Stanislaw Adaszewski: [25]

You might need to modify the C code as per your need and create a .mex file from the c-code. I have been using it for last 2 Years. This is quick – very quick..!

#5 Comment By Yair Altman On October 11, 2017 @ 02:51

@Saket – thanks for sharing, useful indeed

Article printed from Undocumented Matlab: https://undocumentedmatlab.com

URL to article: https://undocumentedmatlab.com/articles/faster-csvwrite-dlmwrite

URLs in this post:

[1] fwrite performance: http://undocumentedmatlab.com/blog/improving-fwrite-performance

[2] save performance: http://undocumentedmatlab.com/blog/improving-save-performance

[3] explains the idea: http://nazarovsky.ru/2014/08/22/fast-csv-values-export-to-file-mex-based-dlmwrite-replacement-for-matlab

[4] mex-writematrix: https://github.com/nazarovsky/mex-writematrix

[5] Wotao Yin showed: http://www.caam.rice.edu/~wy1/links/mex_ctrl_c_trick

[6] Matlab performance tuning part 1: http://undocumentedmatlab.com/courses/Matlab_Performance_Tuning_1_Webinar.pdf

[7] buy: https://www.paypal.com/cgi-bin/webscr?cmd=_xclick¤cy_code=USD&business=octahedron.ltd@gmail.com&quantity=1&amount=295&item_name=Matlab+performance+1+webinar

[8] Matlab performance tuning part 2: http://undocumentedmatlab.com/courses/Matlab_Performance_Tuning_2_Webinar.pdf

[9] buy: https://www.paypal.com/cgi-bin/webscr?cmd=_xclick¤cy_code=USD&business=octahedron.ltd@gmail.com&quantity=1&amount=295&item_name=Matlab+performance+2+webinar

[10] buy: https://www.paypal.com/cgi-bin/webscr?cmd=_xclick¤cy_code=USD&business=octahedron.ltd@gmail.com&quantity=1&amount=495&item_name=Matlab+performance+webinars

[11] time in your local timezone: https://www.google.com/search?q=9am+edt+to+my+time

[12] details: http://undocumentedmatlab.com/training#onsite

[13] buy: https://www.paypal.com/cgi-bin/webscr?cmd=_xclick&currency_code=USD&business=octahedron.ltd@gmail.com&quantity=1&amount=195&item_name=Matlab+performance+1+webinar

[14] buy: https://www.paypal.com/cgi-bin/webscr?cmd=_xclick&currency_code=USD&business=octahedron.ltd@gmail.com&quantity=1&amount=195&item_name=Matlab+performance+2+webinar

[15] buy: https://www.paypal.com/cgi-bin/webscr?cmd=_xclick&currency_code=USD&business=octahedron.ltd@gmail.com&quantity=1&amount=345&item_name=Matlab+performance+webinars

[16] Performance tag: http://undocumentedmatlab.com/blog/tag/performance

[17] Accelerating MATLAB Performance: http://undocumentedmatlab.com/books/matlab-performance

[18] Image: mailto: altmany @gmail.com?subject=Matlab webinars&body=Hi Yair, &cc=;&bcc=

[19] Faster findjobj : https://undocumentedmatlab.com/articles/faster-findjobj

[20] MEX ctrl-c interrupt : https://undocumentedmatlab.com/articles/mex-ctrl-c-interrupt

[21] Matrix processing performance : https://undocumentedmatlab.com/articles/matrix-processing-performance

[22] Explicit multi-threading in Matlab part 3 : https://undocumentedmatlab.com/articles/explicit-multi-threading-in-matlab-part3

[23] Preallocation performance : https://undocumentedmatlab.com/articles/preallocation-performance

[24] datestr performance : https://undocumentedmatlab.com/articles/datestr-performance

[25] : http://adared.ch/efficient-csv-reader-for-matlab

Copyright © Yair Altman - Undocumented Matlab. All rights reserved.