6 Querying historical and intra-day data

Historical data can be retrieved from IB,49 subject to your account subscription rights, and IB’s lengthy list of pacing violation limitations.50 Note that these are IB server limitations, not IB-Matlab limitations. As of Nov 2015, these limitations include:

Historical data is limited by default to 2000 results (data bars). You may have access to more results depending on your IB subscription level. If you request more results than your limit, the entire request is dropped.

Historical data is limited by default to the past year. If you purchase additional concurrent real-time market data-lines from IB you can access up to 5 years of history. If you request data older than your limit, the entire request is dropped.

Historical data requests that use a small (<1 min) bar size can only go back 6 months. If older data is requested, the entire request is dropped.

Requesting identical historical data requests within 15 seconds is prohibited. IB-Matlab will automatically return the previous results in such a case.

Requesting 6+ historical data requests having small bar-size and same contract, exchange, tick type within 2 seconds is prohibited; the request will be dropped.

Requesting 60+ historical data requests having small bar-size of any type within 10-minutes is prohibited; the entire request will be dropped.

IB supports only some combinations of Duration and BarSize (details below).

Historic data must be queried from the same computer (IP) as the trading TWS.

You must have an IB subscription for data from the requested exchange

Also note that historical data retrieval is subject to the same pre-conditions as for retrieving the current live market data (see §5.1). If any of these limitations is not met, then an error message will be displayed and no data will be returned.

Subject to these limitations, retrieving information in IB-Matlab is quite simple. For example, to return the 1-hour bars from the past day, within the regular trading hours:

>> data = IBMatlab('action','history', 'symbol','IBM', ...
'barSize','1 hour', 'useRTH',1)
data =
dateNum: [1x7 double]
dateTime: {1x7 cell}
open: [161.08 160.95 161.66 161.17 161.57 161.75 162.07]
high: [161.35 161.65 161.70 161.60 161.98 162.09 162.34]
low: [160.86 160.89 161.00 161.13 161.53 161.61 161.89]
close: [160.93 161.65 161.18 161.60 161.74 162.07 162.29]
volume: [5384 6332 4580 2963 4728 4465 10173]
count: [2776 4387 2990 1921 2949 2981 6187]
WAP: [161.07 161.25 161.35 161.31 161.79 161.92 162.14]
hasGaps: [0 0 0 0 0 0 0]

As can be seen, the returned data object is a Matlab struct whose fields are:

dateNum – a numeric array of date/time values in Matlab’s numeric format51

dateTime – a cell-array of date strings, or a numeric array of date values in IB format (see the FormatDate parameter, explained below). Intra-day bars use local timezone; daily bars use exchange timezone.

open – the bar’s opening price

high – the high price during the time covered by the bar

low – the low price during the time covered by the bar

close – the bar’s closing price

volume – the trading volume during the time covered by the bar

count – number of trades that occurred during the time covered by the bar Note: only valid when WhatToShow='Trades' (see below)

WAP – the weighted average price during the time covered by the bar

hasGaps – whether or not there are gaps (unreported bars) in the data

The fields are Matlab data arrays (numeric arrays for the data and cell-arrays for the timestamps). To access any specific field, use the standard Matlab notation:

>> data.dateTime
ans =
'20110225 16:30:00' '20110225 17:00:00' '20110225 18:00:00'
'20110225 19:00:00' '20110225 20:00:00' '20110225 21:00:00'
'20110225 22:00:00'

>> lastOpen = data.open(end); % =162.07 in this specific case

As a slightly more complicated example of using these parameters, the following query fetches historical 5-minute bars for the past week, including bars outside regular trading hours:

>> data = IBMatlab('action','history', 'symbol','IBM', ...
'barSize','5 mins', 'useRTH',0, ...
'DurationValue',1, 'DurationUnits','W');

If your query fails one of IB’s list of limitations on historical data requests (see the top of this section), an error message will be reported by IB and no data will be returned. For example:

[API.msg2] Historical Market Data Service error message: Trading TWS session is connected from a different IP address {527921821, 162}

or:

[API.msg2] Historical Market Data Service error message: No market data permissions for NYSE STK {527921824, 162}

The following parameters affect historical data retrieval:52

As noted earlier, IB only supports certain combinations of BarSize and Duration, as detailed in the table below. Note that a 1-sec duration is called “1 secs”, not “1 sec”, while “1 min”, “1 hour”, “1 day”, “1 week” and “1 month” use the singular form.

Other Duration values (that are not specified in the table) are sometimes, but not always, accepted by IB. For example, 60D (=60 days) is accepted, but 61D is not. In such cases, you can always find a valid alternative (3M instead of 61D, for example).

IB-Matlab does not prevent users from entering invalid Durations and BarSizes – it is up to you to verify that your specified parameters are accepted by IB. If they are not, then IB will report an error message in the Matlab command window:

[API.msg2] Error validating request:-'qd' : cause - Historical data bar size setting is invalid. Legal ones are: 1 secs, 5 secs, 10 secs,…

It is possible to specify a BarSize value larger than the duration. For example, on July 14, if we specify a duration of 3 weeks, and BarSize='1w', we’d get the results for all Fridays (=end of trading week) in the past 21 days. If we set BarSize='1m', we’d get two results: for June 30 (=end of last trading month) and July 14.

IB’s historical data mechanism enables retrieving data as recent as a minute ago, or as old as a year (or more, if you purchase this option from IB). Some software vendors differentiate between intra-day and historical information, but as far as IB and IB-Matlab are concerned, this is merely a semantic difference and there is no actual difference. Subject to the available options in the Duration-vs.-BarSize table above, we can select any date/time window that we wish.

In some cases, users may be tempted to use the historical data mechanism to retrieve real-time data. This is relatively easy to set-up. For example, implement an endless Matlab loop that sleeps for 60 seconds, requests the latest historical data for the past minute and then goes to sleep again (advanced Matlab users would improve this by implementing a recurring timer object that wakes up every minute). In such cases, the user should consider using the streaming quotes or realtime bars mechanisms, rather than historical quotes. Streaming data is the subject of the following section.

Note that some securities and exchanges do not support certain historical parameter combinations. For example, FOREX (currency) historical data requests on the IDEALPRO exchange do not support WhatToShow='Trades', only 'Midpoint'. IB displays a very cryptic error message in such cases, and we are only left with the option of guessing what parameter value to modify, or ask IB’s customer support. IB-Matlab automatically tries to re-query using Midpoint if a Trades query fails, but this should not be relied-upon, because the failed query might be due to other causes.

Refer to IB’s documentation56 for the latest information on the allowed parameter values for historical data requests. Here is a table listing the allowed WhatToShow values for various SecTypes, valid as of 7/2018:57

In addition to these WhatToShow values, the Yield_* values are only reported for Indices, not for any other security type.

When no data is returned by IB, IB-Matlab will automatically try to resend the historical data request using WhatToShow='Trades' (for SecType='IND'), or WhatToShow='Midpoint' (for any other SecType), if these are different from the WhatToShow in the original request:

>> data = IBMatlab('action','history', 'symbol','EUR', ...
'localSymbol','EUR.USD', 'secType','cash', ...
'exchange','idealpro', 'barSize','1 hour')

[API.msg2] Historical Market Data Service error message: No historical market data for EUR/CASH@FXSUBPIP Last 3600 {786819168, 162}

No data returned from IB - retrying with WhatToShow='Midpoint'...

data =
dateNum: [1×18 double]
dateTime: {1×18 cell}
...

Also note that some exchanges return the requested historical data, but do not provide all of the historical data fields. For example, with FOREX on IDEALPRO, the volume, count and WAP fields are not returned, and appear as arrays of -1 when returned to the user in the data struct:

>> data = IBMatlab('action','history', 'symbol','EUR', ...
'localSymbol','EUR.USD', 'secType','cash', ...
'exchange','idealpro', 'barSize','1 day', ...
'DurationValue',3, 'WhatToShow','midpoint')
data =
dateNum: [734909 734910 734913]
dateTime: {'20120210' '20120211' '20120214'}
open: [1.32605 1.3286 1.32095]
high: [1.3321 1.329075 1.328425]
low: [1.321625 1.315575 1.320275]
close: [1.328575 1.319825 1.325975]
volume: [-1 -1 -1]
count: [-1 -1 -1]
WAP: [-1 -1 -1]
hasGaps: [0 0 0]

In this example, historical daily (BarSize = '1 day') data from the past 3 days was requested on 2012-02-13 (Monday). Data was received for 2012-02-09 (Thursday), 2012-02-10 (Friday) and 2012-02-13 (Monday). Data was not received for 2012-02-11 and 2012-02-12 because the specified security was not traded during the weekend.

Another oddity is that the dates were reported with an offset of 1 (2012-02-10 instead of 2012-02-09 etc.). The reason is that the information is collected on a daily basis and reported as of the first second after midnight, i.e., on the following date. This is indeed confusing, so if you rely on the reported historical data dates in your analysis, then you should take this into consideration. This 1-day offset only occurs when UseRTH=0 (which is the default value): if you set UseRTH=1, then the correct dates will be reported, since regular trading hours end within the same day, not at midnight.

One user has reported that in some cases IB returns empty data for historical index (SecType='Ind') queries. Restarting TWS/Gateway and re-querying when the exchange is active appears to solve this problem.

Note that if IncludeExpired is set to 1 (or true), the historic data on expired contracts is limited to the last year of the contract’s life, and is initially only supported by IB for expired futures contracts (IBMatlab imposes no limitation, but IB may indeed).