Comments on: Speeding up Matlab-JDBC SQL queries

By: Yair Altman

Yair Altman — Fri, 23 Feb 2024 14:02:41 +0000

In reply to Peter Raahauge. @Peter - I'm happy to hear that you found my work useful. Feel free to use my algorithm and code for any non-commercial research usage. I just ask that you add references to undocumentedmatlab.com and my book (Accelerating Matlab Performance, CRC Press, ISBN 9781482211290) in your published work.

By: Peter Raahauge

Peter Raahauge — Tue, 13 Feb 2024 11:01:50 +0000

Hi Yair,
I have a copyright question, and a comment to your statement above: “the speed remains nearly unaffected when you pre-allocate data properly.”
One of my colleges are downloading data from Wharton Research Data Service (WRDS) for his research in Matlab.
WRDS are providing a guide for Matlab-users, and it basically includes an algorithm like the first one presented above for getting the data into Matlab’s memory.
The SQL-tables he’s downloading typically have 3 million rows but very few variables.
Initially, he was complaining about the WRDS-solution, as a query like the one above took 2-3 hours. However, once he got the data variable pre-allocated appropriately, the query was done in 5 minutes. That’s actually a factor 30(!)
With your Java-code, we can get the time down even more with something similar to the factor 15 you mention above.
This leads me to the copyright question: How should I deal with the copyright? We are only using your algorithm for research purposes and nothing commercial.

By: Jeremy

Jeremy — Mon, 23 Sep 2019 16:57:38 +0000

Yair,

Thank you for your work, most likely buying the book soon. But in the meantime I’m hitting a snag on setting up the JDBC connection on the fly. Can you provide a bit more explanation and or hints on the steps required? I can use database toolbox to do the setup and it works just fine, but I would rather be able to hardcode it so that my application/code is portable.

By: Paul

Paul — Mon, 08 May 2017 23:19:17 +0000

Hey there,

just wanted to say thank you for this page. I found a lot of valuable information here how to handle result sets from java in matlab.

Best Regards
Paul

By: Yair Altman

Yair Altman — Tue, 22 Nov 2016 06:43:30 +0000

In reply to virns. Try to close the ResultSet and stmt (as I've shown in the post) after you've used them, in order to clear the cache between subsequent JDBC calls, otherwise they might remain in memory and cause an eventual heap-memory error.

By: Yair Altman

Yair Altman — Mon, 21 Nov 2016 16:57:25 +0000

In reply to aviolov. I do not believe that JDBC has such an easy-to-use wrapper, unfortunately.

By: Oleg

Oleg — Mon, 21 Nov 2016 16:46:37 +0000

I faced this issue with the Matlab Database Toolbox and tackled it by: * setting the 'DataReturnFormat' to 'numeric'; * querying results by column and converting the numeric stream into its column format

By: virns

virns — Mon, 21 Nov 2016 10:51:32 +0000

Hi
Please help me get rid of “jave heap memory error” in standalone executable (tried both windows and linux versions). I am creating a SQL connection (using JDBC) outside the infinity while loop and then checking with “isopen or isconnection” inside the loop. I read in some forum that the SQL JDBC driver caches the queries which could be the cause. I’ve also tried deleting unused variables in the loop but it didn’t help.

Thanks

By: aviolov

aviolov — Mon, 21 Nov 2016 08:24:32 +0000

Cool,

We do a similar thing using C# and DataTable

import System.Data.DataTable
import System.Data.SqlClient.*
sqlConn = SqlConnection(...);
sqlCmd = SqlCommand(sql_query_string, sqlConn);
sqlAdaptor = SqlDataAdapter(sqlCmd);
dataTable = DataTable();
sqlConn.Open();
sqlAdaptor.Fill(dataTable);
T = dataTable2matlabTable(dataTable); % This does the same column based-fill as in your code.
...

So I’m wondering is it really necessary to make the specific class (JDBC_Fetch.java) to iterate the rows of a ResultSet or does Java not have an equivalent to C#’s DataTable that kind of does that for you, i.e. a data-struct that returns columns instead of rows, roughly speaking, and you can already work on its columns instead of calling rs.next() in an extra for loop? Or maybe the equivalent does not give the necessary speed-up?

best,