- Undocumented Matlab - https://undocumentedmatlab.com -

Expanding urlread capabilities

Posted By Yair Altman On March 21, 2012 | 17 Comments

I would like to welcome guest blogger Jim Hokanson [1]. Today, Jim will explain some of the limitations that at one time or another many of us have encountered with Matlab’s built-in urlread function. More importantly, Jim has spent a lot of time in creating an expanded-capabilities Matlab function, which he explains below. Note that while urlread‘s internals are undocumented, the changes outlined below rely on pretty standard Java and HTTP, and should therefore be quite safe to use on multiple Matlab versions.

Abstract

I recently tried to implement the Mendeley [2] API but quickly found that urlread was not going to be sufficient for my needs. The first indication of this was my inability to send the proper authorization information for POST requests. It became even more obvious with the need to perform DELETE and PUT methods, since urlread only supports GET and POST. My implementation of urlread, which I refer to as urlread2, addresses these and a couple of other issues and can be found on the Matlab File Exchange [3]. Other developers have tackled urlread‘s shortcomings (this example [4] added timeout support, and this example [5] added support for binary file upload) – today’s article will focus on my solution, but others are obviously possible.

Introduction

HTTP [6] is the underlying computer networking protocol that enables us to read webpages on the Internet. It consists of a request made by the user to an Internet server (typically located via URL [7]), and a response from that server. Importantly, the request and response consist of three main parts: a resource line (for requests) or status line (for responses), followed by headers, and optionally a message body.
Matlab’s built-in urlread function enables Matlab users to easily read the server’s response text into a Matlab string:

text = urlread('http://www.google.com');

This is done internally using Java code that connects to the specified URL and reads the information sent by the URL’s server (more on this [8]).
urlread accepts optional additional inputs specifying the request type (‘get’ or ‘post’) and parameter values for the request.
Unfortunately, urlread has the following limitations:

  1. It does not allow specification of request headers
  2. It makes assumptions as to the request headers needed based on the input method
  3. It does not expose the response headers and status line
  4. It assumes the response body contains text, and not a binary payload
  5. It does not enable uploading binary contents to the server
  6. It does not enable specifying a timeout in case the server is not responding

urlread2

The urlread2 function addresses all of these problems. The overall design decision for this function was to make it more general, requiring more work up front to use in some cases, but more flexibility.
For reference, the following is the calling format for urlread2 (which is reminiscent of urlread‘s):

urlread2(url,*method,*body,*headersIn, varargin)

The * indicate optional inputs that must be spatially maintained.

  • url – (string), url to request
  • method – (string, default GET) HTTP request method
  • body – (string, default ”), body of the request
  • headersIn – (structure, default []), see the following section
  • varargin – extra properties that need to be specified via property/pair values

Addressing Problem 1 – Request header

urlread internally uses a Java object called urlConnection that is generally an instance of the class sun.net.www.protocol.http.HttpURLConnection [9]. The method setRequestProperty() can be used to set headers for the request. This method has two inputs, the header name and the value of that header. A simple example of this can be seen below:

urlConnection.setRequestProperty('Content-Type','application/x-www-form-urlencoded');

Here ‘Content-Type’ is the header name and the second input is the value of that property. My function requires passing in nearly all headers as a structure array, with fields for the name and value. The preceding header would be created using a helper function http_createHeader.m:

header = http_createHeader('Content-Type','application/x-www-form-urlencoded');

Multiple headers can be passed in to the function by concatenating header structures into a structure array.

Addressing Problem 2 – Request parameters

When making a POST request, parameters are generally specified in the message body using the following format:
[property]=[value]&[property]=[value]
The properties and values are also encoded in a particular way, generally termed urlencoded [10] (encoding and decoding can be done using Matlab’s built-in urlencode and urldecode functions). For GET requests this string is appended to the url with the “?” symbol. Since urlencoding methods can vary, and in the spirit of reducing assumptions, I use separate functions to generate these strings outside of urlread2, and then pass the result in either as the url (for GET) or as the body input (for POST). As an example, I might search the Mathworks website [11] using the upper right search bar on its site for “undocumented matlab” under file exchange (hmmm… pretty cute stuff there!). Doing this performs a GET request with the following property/value pairs:

params = {'search_submit','fileexchange', 'term','undocumented matlab', 'query','undocumented matlab'};

These property/value pairs are somewhat obvious from looking at the URL, but could also be determined by using programs such as Fiddler [12], Firebug [13], or HttpWatch [14].
After urlencoding and concatenating, we would form the following string:
search_submit=fileexchange&term=undocumented+matlab&query=undocumented+matlab
This functionality is normally accomplished internally in urlread, but I use a function http_paramsToString to produce that result. That function also returns the required header for POST requests. The following is an example of both GET and POST requests:

[queryString,header] = http_paramsToString(params,1);
% For GET:
url = [url '?' queryString];
urlread2(url)
% For POST:
urlread2(url,'POST',queryString,header)

Addressing Problem 3 – Response header

According to the HTTP protocol, each server response starts with a simple header that indicates a numeric response status [15]. The following Matlab code provides access to the status line using the urlConnection object:

status = struct('value',urlConnection.getResponseCode(), 'msg',char(urlConnection.getResponseMessage))
status =
    value: 200
      msg: 'OK'

urlConnection‘s getHeaderField() and getHeaderFieldKey() methods enable reading the specific parts of the response header:

headerValue = char(urlConnection.getHeaderField(headerIndex));
headerName  = char(urlConnection.getHeaderFieldKey(headerIndex));

headerIndex starts at 0 and increases by 1 until both headerValue and headerName return empty.
It is important to note that header keys (names) can be repeated for different values. Sometimes this is desired, such as if there are multiple cookies being sent to the user. To generically handle this case, two header structures are returned. In both cases the header names are the field names in the structure, after replacing hyphens with underscores [16]. In one case, allHeaders, the values are cell arrays of strings containing all values presented with the particular key. The other structure, firstHeaders, contains only the first instance of the header as a string to avoid needing to dereference a cell array.

Addressing Problem 4 – Response body

urlread assumes text output. This is fine for most webpages, which use HTML and are therefore text-based. However, urlread fails when trying to download any non-text resource such as an image, a ZIP file, or a PDF document. I have added a flag in urlread2 called CAST_OUTPUT, which defaults to true, i.e. text response, just as urlread assumes. Using varargin, this flag can be set to false ({‘CAST_OUTPUT’,false}) to indicate a binary response.

Summary

urlread2‘s functionality has been expanded to also address other limitations of urlread: It enables binary inputs, better character-set handling of the output, redirection following, and read timeouts.
The modifications described above provide direct access to the key components of the HTTP request and response messages. Its more generic nature lets urlread2 focus on HTTP transmission, and leaves request formation and response interpretation up to the user. I think ultimately this approach is better than providing one-off modifications of the original urlread function to suit a particular need. urlread2 and supporting files can be found [3] on the Matlab File Exchange.

Categories: Guest bloggers, Java, Low risk of breaking in future versions, Stock Matlab function


17 Comments (Open | Close)

17 Comments To "Expanding urlread capabilities"

#1 Comment By Andrew Walsh On May 4, 2012 @ 02:10

Thanks for this!
I am trying to use urlread to scrape a website for stats

eg.
[23]

which works with a browser but not with urlread which returns

I figure I can use urlread2 to do the refresh for me somehow but as I am fairly inexperienced with scraping I dont know exactly how to do this.
I imagine it would be fairly easy.
Can you give me a tip please?

#2 Comment By Richard On May 20, 2012 @ 17:13

I found it much easier to create a urlreadbin.m file based on urlread with a single line change.
Edit urlread.m in the iofun directory using “open urlread.m” and change the following:

%output = native2unicode(typecast(byteArrayOutputStream.toByteArray','uint8'),'UTF-8');
output = typecast(byteArrayOutputStream.toByteArray','uint8'); %052012 oglraz

Save as urlreadbin.m in the iofun directory.

This eliminates the bothersome unicode conversion.

This is only for reads but grabs jpg files without an issue.

#3 Comment By Alejo On July 14, 2013 @ 16:45

Hi, i’m trying to read an image using urlread2
Images is on [24] how can i do that? Or how can i do this with imread?
Thanks!

#4 Comment By cglopez On August 23, 2013 @ 20:20

How can I used urlread2 with basic authentication.
Thanks

#5 Comment By Yingyun On June 6, 2017 @ 22:39

Can I ask if you solve the problem by using urlread2 to do a authentication? thank you

#6 Comment By Raphael On September 27, 2013 @ 10:33

Hello, is the timeout property working ?

I have matlab r2011b but cannot make it work.
my query: urlread2(‘http://www.google.com’, ‘GET’, ”, [], ‘READ_TIMEOUT’, 100);

I tried 1, 100, 1000 as I wasn’t sure of the unit, but all of them time out in ~20seconds.

#7 Comment By Raphael On September 27, 2013 @ 10:49

adding a value for setConnectTimeout fixed it

#8 Comment By Raphael On September 27, 2013 @ 11:47

is there a way to make the function work for local files/machines ?

I am trying to find a way to check the connection to a local server faster than doing exist(path, ‘dir’). this function has a 20s time out

#9 Comment By ibrahim On December 26, 2013 @ 00:43

I’m trying to post some data to local web server. I could not get any result. could you help me to identify the problem in my code.

clear
clc
uname='ibrahim';
email='test@tes.com';
reemail= email;
pswd='1234';
confirm= pswd;
counter=1;
submit='submit';

while (counter < 2)
    username=[uname,num2str(counter)];
    password=pswd;
    params = {username,password,email,reemail,confirm,'submit'};
    s=urlread('http://localhost/new_sdas/php_code/register.php','POST',params)
    counter=counter+1;
end

#10 Comment By Yair Altman On December 26, 2013 @ 01:42

@ibrahim – this is not a general-purpose blog but one that is devoted to undocumented and advanced aspects of Matlab. With regular Matlab questions you will find more luck at the [25].

I would try checking if the web-page is accessible (and accepts POST queries) using a regular browser, before trying in Matlab’s urlread. It’s a localhost server so maybe you forgot to turn it on. Also, maybe it works via cookies, which urlread does not support.

#11 Comment By J Syn On June 30, 2014 @ 03:03

I was using urlread2 for the first time yesterday and encountered a very strange error. The script was very simple – containing just one line calling the urlread2 function. The script froze during execution, and Matlab produced errors relating to the “rmdir” command. When I checked the folder I was working in, all files and folders except the currently running script file had been deleted. Can you offer any explanation as to why this might have happened? Please see the Matlab output below:

“Operation terminated by user during urlread2 (line 199)
In TestDatabaseRead (line 3)
str =
urlread2(‘http://localhost:13595/I/GetLocalData’);
Error using rmdir”

#12 Comment By J Syn On June 30, 2014 @ 07:22

Just an update to my previous post – I have tried running the same code on another machine and had no issues at all. I have no idea what happened, but it must have been my own fault somehow. The GET functionality is so much faster using this than the urlread method supplied with Matlab. Jim was also fantastic, replying to my e-mail query right away.

#13 Comment By Tjarko On July 3, 2014 @ 11:05

Hi, I’m trying to use the urlread2 function for sending PUT requests. I’m new to webservices for that I need a little help.

I want to do this

PUT http://[DEVICE_IP]:7979/rest/devices/battery/C23 HTTP/1.1
Host: [IP]
Content-type: text/html
Content-length: 4
1000

What do I have to do in Matlab to perform this action?

#14 Comment By Joel On August 4, 2014 @ 06:07

Hello,

I am having performance issues with urlread and wonder whether urlread2 would be faster. A simple 35000 character webpage

A=urlread(‘http://www.mathworks.com’);

takes about 700ms to read and this seems to be rather matlab/hardware/internet connection independent as other people have gotten similar results. I posted my original question here:

[26]

So I am wondering if there are any ways to speed up urlread? Or is urlread2 significantly faster? Unfortunately I could not get urlread2 working in my old Matlab 7. It gave errors as explained in the linked question.

I feel that performance should be much better. A simple Autohotkey command UrlDownloadToFile does this much faster but it is painful to make different programs operate together.

Any help is greatly appreciated. Thank you for any help in advance!

#15 Pingback By HTTP Requests with Matlab | Tasos Sangiotis On May 11, 2015 @ 05:06

[…] you want more advanced features checkout urlread2 by Jim Hokanson. It is extending urlread and makes it very easy todo that stuff with […]

#16 Comment By JosephAgnes On June 17, 2015 @ 07:16

Hi, how can I create a persistent http session in MATLAB? I first do a login to the website homepage which requires password to open it. And then in order to manipulate other pages I could change the URLs at my choice but staying logged in! Thanks and please comment.

#17 Pingback By HTTP Requests with Matlab | Tasos Writes On June 21, 2015 @ 05:55

[…] if you want more advanced features checkout urlread2 by Jim Hokanson. It is extending urlread and makes it very easy to do that stuff with Matlab. […]


Article printed from Undocumented Matlab: https://undocumentedmatlab.com

URL to article: https://undocumentedmatlab.com/articles/expanding-urlreads-capabilities

URLs in this post:

[1] Jim Hokanson: http://sites.google.com/site/jimhokanson/

[2] Mendeley: http://dev.mendeley.com

[3] found on the Matlab File Exchange: http://www.mathworks.com/matlabcentral/fileexchange/35693-urlread2

[4] this example: http://www.mathworks.com/matlabcentral/fileexchange/8474-rewrites-of-urlread-and-urlwrite

[5] this example: http://www.mathworks.com/matlabcentral/fileexchange/27189-urlreadpost-url-post-method-with-binary-file-uploading

[6] HTTP: http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol

[7] URL: http://en.wikipedia.org/wiki/Uniform_resource_locator

[8] more on this: http://docs.oracle.com/javase/tutorial/networking/urls/index.html

[9] sun.net.www.protocol.http.HttpURLConnection: http://docs.oracle.com/javase/1.5.0/docs/api/java/net/HttpURLConnection.html

[10] urlencoded: http://en.wikipedia.org/wiki/Percent-encoding

[11] search the Mathworks website: http://www.mathworks.com/matlabcentral/fileexchange/?search_submit=fileexchange&term=undocumented+matlab&query=undocumented+matlab

[12] Fiddler: http://www.fiddler2.com/fiddler2/

[13] Firebug: http://getfirebug.com/

[14] HttpWatch: http://www.httpwatch.com/

[15] response status: http://en.wikipedia.org/wiki/List_of_HTTP_status_codes

[16] replacing hyphens with underscores: http://www.mathworks.com/help/techdoc/ref/genvarname.html

[17] Multi-line uitable column headers : https://undocumentedmatlab.com/articles/multi-line-uitable-column-headers

[18] Setting status-bar components : https://undocumentedmatlab.com/articles/setting-status-bar-components

[19] Setting status-bar text : https://undocumentedmatlab.com/articles/setting-status-bar-text

[20] Matlab and the Event Dispatch Thread (EDT) : https://undocumentedmatlab.com/articles/matlab-and-the-event-dispatch-thread-edt

[21] Matlab-Java interface using a static control : https://undocumentedmatlab.com/articles/matlab-java-interface-using-static-control

[22] Using Groovy in Matlab : https://undocumentedmatlab.com/articles/using-groovy-in-matlab

[23] : http://www.footywire.com/afl/footy/ft_match_statistics?mid=5343

[24] : http://192.168.1.20/snapshot.cgi

[25] : http://www.mathworks.com/matlabcentral/answers/

[26] : http://www.mathworks.nl/matlabcentral/answers/144600-how-can-i-make-urlread-faster-problem-using-urlread2

Copyright © Yair Altman - Undocumented Matlab. All rights reserved.