7 Streaming data

Streaming data is a near-real-time mechanism, where IB sends ongoing information to IB-Matlab about quote ticks (bids and asks) and aggregated real-time bars.

7.1 Streaming quotes

The streaming quotes mechanism has two distinct parts:

Request IB to start sending the stream of quotes for a specified security. This is done by using Action='query' and QuotesNumber with a positive >1 value. The request’s ID (a scalar integer) is returned.

Later, whenever you wish to read the latest quote(s), simply use Action='query' and QuotesNumber= -1 (minus one). This will return the latest information (a data struct), without stopping the background streaming.

For example, let’s request 100 streaming quotes for EUR.USD:

>> reqId = IBMatlab('action','query', 'symbol','EUR', ...
'localSymbol','EUR.USD', 'currency','USD', ...
'secType','cash', 'exchange','idealpro', ...
'QuotesNumber',100)
reqId =
147898050

This causes IB to start sending quotes to IB-Matlab in the background, up to the specified QuotesNumber, without affecting normal Matlab processing. This means that you can continue to work with Matlab, process/display information etc.

QuotesNumber can be any number higher than 1 for streaming to work (a value of 1 is the standard market-query request described in §5.1). To collect streaming quotes endlessly, simply set QuotesNumber to the value inf. Note that in Matlab, inf is a number not a string so do not enclose it in quotes ('inf') when submitting requests.

Also note that the request to start streaming quotes returns the request ID, not data.

The quotes are collected into an internal data buffer in IB-Matlab. A different buffer is maintained for each contract (or rather, combination of LocalSymbol, SecType and Expiry). The buffer size can be controlled using the QuotesBufferSize parameter, which has a default value of 1. This means that by default only the latest streaming quote of each type (bid/ask) is stored, along with high/low/close data.

If you set a higher value for QuotesBufferSize,58 then up to the specified number of latest bid quotes will be stored (note: only bid quotes are counted here):

>> reqId = IBMatlab('action','query', 'symbol','GOOG', ...
'QuotesNumber',100, 'QuotesBufferSize',500)

Note that using a large QuotesBufferSize increases memory usage, which could have an adverse effect if you use a very large buffer size (many thousands) and/or streaming for a large number of different securities.59

Subsequent requests to retrieve the latest accumulated quotes buffer data, without stopping the background streaming, should use QuotesNumber = -1 (minus one). These requests return a Matlab data struct similar to this:

>> dataStruct = IBMatlab('action','query', ...
'localSymbol','EUR.USD', ...
'QuotesNumber',-1)
dataStruct =
reqId: 147898050
symbol: 'EUR'
localSymbol: 'EUR.USD'
isActive: 1
quotesReceived: 6
quotesToReceive: 10
quotesBufferSize: 1
genericTickList: ''
data: [1x1 struct]
contract: [1x1 struct]

Streaming quotes are stored using a combination of the LocalSymbol, SecType, and Expiry date values that were specified in the initial request for the streaming quotes.

In most cases (as above), we only need to specify the Symbol/LocalSymbol and the QuotesNumber in the subsequent requests.60 Specifying the other parameters is not normally necessary, since IB-Matlab already knows them from the initial streaming request. We only need to specify the SecType and possibly Expiry when there is a potential conflict between distinct streaming quotes (e.g., streaming quotes of both the underlying asset and some Futures index of it). This is useful and easy to use, but also means that you cannot have two simultaneous streams for the same combination of LocalSymbol, SecType and Expiry, even if using different other parameters.

In the returned dataStruct, we can see the following fields:

reqId – this is the request ID (scalar integer) for the original streaming request, the same ID that was returned by IBMatlab in our initial request.

symbol, localSymbol – the security whose data is being streamed.

isActive – indicates whether quotes are currently streamed for this security. When QuotesNumber bid quotes have been received, this flag is set to false (0).

quotesReceived – number of streaming bid quotes received for this security.

quotesToReceive – maximal number of streaming bid quotes requested for the security (=QuotesNumber). When quotesReceived >= quotesToReceive, streaming is stopped and isActive is set to false (0).61

quotesBufferSize – size of the data buffer (=QuotesBufferSize).

genericTickList – GenericTickList requested in the initial request (see below)

contract – a sub-struct that holds the contract’s definition. Note: in IBMatlab v2.18 and older this was a Java object not a Matlab struct.

Data – this is a sub-struct that holds the actual buffered quotes data.

To get the actual quotes data, simply read the data field of this dataStruct:

>> dataStruct.data
ans =
dataTimestamp: 734892.764653854
high: 1.3061
highTimestamp: 734892.762143183
low: 1.29545
lowTimestamp: 734892.762143183
close: 1.30155
closeTimestamp: 734892.762143183
bidPrice: 1.2986
bidPriceTimestamp: 734892.764653854
bidSize: 1000000
bidSizeTimestamp: 734892.764653854
askPrice: 1.29865
askPriceTimestamp: 734892.764499421
askSize: 18533000
askSizeTimestamp: 734892.764653854

Note that each data item has an associated timestamp, because different data items are sent separately and independently from IB server. You can convert the timestamps into human-readable string by using Matlab’s datestr function, as follows:

>> datestr(dataStruct.data.dataTimestamp)
ans =
24-Jan-2012 23:56:32

The dataTimestamp field currently holds the same data as bidPriceTimestamp. Future versions may possibly indicate the latest timestamp of any quote, not necessarily a bid.

If instead of using QuotesBufferSize=1 (which is the default value), we had used QuotesBufferSize=3, then we would see not the latest quote but the latest 3 quotes:

>> reqId = IBMatlab('action','query', 'symbol','EUR', ...
'localSymbol','EUR.USD', 'currency','USD', ...
'secType','cash', 'exchange','idealpro', ...
'QuotesNumber',10, 'QuotesBufferSize',3);

% now run the following command at any time to get the latest 3 quotes:

>> dataStruct = IBMatlab('action','query', ...
'localSymbol','EUR.USD', ...
'QuotesNumber',-1);
>> dataStruct.data
ans =
dataTimestamp: [734892.99760235 734892.99760235 734892.997756076]
high: 1.3061
highTimestamp: 734892.99740162
low: 1.29545
lowTimestamp: 734892.99740162
bidPrice: [1.30355 1.3035 1.30345]
bidPriceTimestamp: [734892.99760235 734892.99760235 734892.997756076]
bidSize: [2000000 4000000 4000000]
bidSizeTimestamp: [734892.997756076 734892.997756076 734892.997756076]
askPrice: [1.30355 1.3036 1.30355]
askPriceTimestamp: [734892.997667824 734892.997667824 734892.997756076]
askSize: [3153000 2153000 4153000]
askSizeTimestamp: [734892.997756076 734892.997756076 734892.997756076]
close: 1.30155
closeTimestamp: 734892.997407037

Note that the high, low and close fields are only sent once by the IB server, as we would expect. Only the bid and ask information is sent as a continuous stream of data from IB. Also note how each of the quote values has an associated timestamp.

To stop collecting streaming quotes for a symbol, resend the query request with QuotesNumber=0. The request will return the dataStruct with the latest data accumulated up to that time. To stop streaming quotes for all symbols, stop them one by one, or disconnect from IB (for example using IBMatlab('disconnect'), see §13).

The ReconnectEvery parameter can be used to bypass occasional problems with high-frequency streams. In such cases, it has been reported that after several thousand quotes, IB stops sending streaming quotes data, without any reported error message.62 The ReconnectEvery numeric parameter (default=5000) controls the number of quotes (total of all streaming securities) before IB-Matlab automatically reconnects to IB and re-subscribes to the streaming quotes. You can specify any positive numeric value, or inf to accept streaming quotes without any automated reconnection.

Here is a simulated timeline that illustrates the use of streaming data in IBMatlab:

Additional market data about a security can be retrieved using IB’s Generic Tick List mechanism,63 which is accessed via the GenericTickList parameter and requires a streaming (non-snapshot) data query. GenericTickList values should be set to a string (default='' =empty) that accepts comma-separated integers such as '100,101' or '236'.64 Note that the value should be a string ('236'), not a number (236).

IBMatlab('action','query', 'symbol','GOOG', ...
'QuotesNumber',2, ... % 2=streaming (not snapshot)
'GenericTickList','236'); % Note: '236', not 236

Multiple comma-separated tick types can be specified, e.g. '233,236,258'.

One of the useful tick-types is 236, which returns information about whether or not the specified security is shortable. Only some securities and exchanges support this feature (mainly US stocks). When the feature is supported, an additional shortable field is returned with basic information about the security’s shortability.65

Note: according to IB,66 “Generic Tick Tags cannot be specified if you elect to use the snapshot market data subscription”. Using GenericTickList with non-streaming (snapshot) queries is a common bug, where generic ticks information is not returned.

To ensure a streaming (not snapshot) query, set QuotesNumber=2 (or anything >1), then wait a bit (to let the streamed data time to arrive), then collect the streamed data using QuotesNumber=-1. Put this in a loop to ensure robustness in case the streamed data has not yet arrived by the time that you try to collect it using QuotesNumber=-1.

Here is a usage example to fetch an option’s open interest information (tick type 101):

% send a streaming quotes query with GenerickTickList='101'
queryParams = {'action','query', 'symbol','IBM', 'secType','OPT',...};
IBMatlab(queryParams{:}, 'QuotesNumber',2, 'GenericTickList','101');

% wait up to 6 x 0.5 = 3 secs for data to arrive (timeout mechanism)
attempt = 1;
while attempt <= 6
pause(0.5); % wait a bit to let the streamed data time to arrive
data = IBMatlab(queryParams{:}, 'QuotesNumber',-1); %collect data
try data = data.data; catch, end
try callOI = data.OptionCallOpenInterest; catch, callOI = -1; end
try putOI = data.OptionPutOpenInterest; catch, putOI = -1; end
allOIs = [callOI,putOI];
openInterest = max(allOIs);
% wait for a valid value (or two 0 values)
if openInterest > 0 || all(allOIs==0)
data.openInterest = openInterest;
break
end
attempt = attempt + 1;
end

% stop streaming this symbol in case it has not stopped by now
try IBMatlab(queryParams{:}, 'QuotesNumber',0); catch, end

Here is a summary of the IBMatlab parameters that directly affect streaming quotes:

Notes:

IB does not send ‘flat’ ticks (quotes where price does not change). Also, IB streaming data is NOT tick-by-tick, but rather snapshots of the market (every 5ms for Forex, 10ms for Options, and 250ms for all other security types).

By default, IB limits the streaming to 100 concurrent requests (contracts). Users can purchase additional 100-contract blocks (“Quote Booster”) from IB.

IB’s messages rate limitation (50/sec, see §3.1) does not directly affect streaming quotes, only messages sent to the IB server. There is no known IB limitation on streamed messages rate. However, a practical limitation is ~50-100 quotes/sec due to your client computer processing time.

Streaming data retrieval is subject to the same pre-conditions as for retrieving the current live market data (see §5.1).

7.2 Realtime bars

The realtime bars mechanism is similar to streaming quotes in the sense that it enables the user to receive information about a security every several seconds, until the specified QuotesNumber of bars have been received. The mechanism is also similar to historical data in the sense that the bars information is aggregated. Each bar contains the OHLCV information just as for historical data bars (see §6 for details).

Similarly to streaming quotes, the realtime bars mechanism has two distinct parts:

Request IB to start sending the stream of bars for a specified security. This is done by using Action='realtime_bars' and QuotesNumber with a positive (>0) value. If QuotesNumber=1 (the default value), then the data for the single bar is returned immediately; Otherwise, only the request ID is returned.

Later, whenever you wish to read the latest bar(s) data, simply use Action='realtime_bars' and QuotesNumber= -1 (minus one). This will return the latest information without stopping the background streaming.

Like streaming quotes, the streamed bars are stored based on a unique combination of their LocalSymbol, SecType and Expiry. As with streaming quotes, there is the ability to automatically reconnect to IB after every specified number of received bars.

Note that IB currently limits the realtime bars to 5-second bars only.68 Also, only some combinations of securities, exchanges and data types (the WhatToShow parameter) are supported. If you have doubts about whether a specific combination is supported, ask IB customer service (the limitation is on the IB server, not IBMatlab).

Users can process realtime bars in one of two ways:

use a Matlab timer to query the latest accumulated bars data (via QuotesNumber = -1), or:

use the CallbackRealtimeBar parameter to set a dedicated Matlab callback function that will be invoked whenever a new bar is received (every 5 secs).69

Here is a simple example of using realtime bars for a single (snapshot) bar (QuotesNumber = 1), representing the previous 5 seconds:

>> data = IBMatlab('action','realtime', 'symbol','IBM')
data =
dateNum: 735551.017997685
dateTime: {'13-Nov-2013 00:25:55'}
open: 183
high: 183
low: 183
close: 183
volume: 0
WAP: 183
count: 0

And here is a slightly more complex example, with QuotesNumber=3. The data struct that is returned in this case is correspondingly more complex:

>> reqId = IBMatlab('action','realtime', 'symbol','AMZN', ...
'QuotesNumber',3, 'QuotesBufferSize',10)
reqId =
345327051

(now wait 15 seconds or more for the 3 bars to be received)

>> dataStruct = IBMatlab('action','realtime', 'symbol','AMZN',
'QuotesNumber',-1)
dataStruct =
reqId: 345327051
symbol: 'AMZN'
localSymbol: ''
isActive: 0
quotesReceived: 3
quotesToReceive: 3
quotesBufferSize: 10
whatToShow: 'TRADES'
useRTH: 0
data: [1x1 struct]
contract: [1x1 struct]

>> dataStruct.data
ans =
dateNum: [735551.008912037 735551.008969907 735551.009027778]
dateTime: {1x3 cell}
open: [349.97 349.97 349.97]
high: [349.97 349.97 349.97]
low: [349.97 349.97 349.97]
close: [349.97 349.97 349.97]
volume: [0 0 0]
WAP: [349.97 349.97 349.97]
count: [0 0 0]

>> dataStruct.data.dateTime
ans =
'13-Nov-2013 00:12:50' '13-Nov-2013 00:12:55' '13-Nov-2013 00:13:00'

You may sometimes see warning messages of the following form:

[API.msg2] Can't find EID with tickerId:345313582 {345313582, 300}

These messages can safely be ignored. They represent harmless requests by IBMatlab to IB, to cancel realtime bar requests that were already cancelled on the IB server.

Realtime bar requests are subject to both historical data pacing limitations (see §6 for details) and streaming data pacing limitations (§7.1). You may be able to loosen the limitations by purchasing additional data slots from IB. Discuss your alternatives with IB customer service, if you encounter pacing violation messages:

[API.msg2] Invalid Real-time Query: Historical data request pacing violation {8314, 420}

As with streaming quotes (§7.1), streaming realtime bars can be stopped by resending the query request with QuotesNumber=0 or by disconnecting from IB (§13).

Here is a summary of the IBMatlab parameters that directly affect realtime bars:

7.3 Streaming market depth (Level II) data

The streaming market depth mechanism71 is also similar to streaming quotes in the sense that it enables the user to receive Level II information about a security every several seconds, until the specified QuotesNumber have been received. In fact, the only difference between streaming market depth data and streaming quotes data is that for market depth, the NumberOfRows parameter is set to an integer value between 2-5 (i.e., 2, 3, 4, or 5), and a slightly-different returned dataStruct:

>> reqId = IBMatlab('action','query', 'symbol','EUR', ...
'localSymbol','EUR.USD', 'currency','USD', ...
'secType','cash', 'exchange','idealpro', ...
'NumberOfRows',3, 'QuotesNumber',1000)
reqId =
464879608

>> dataStruct = IBMatlab('action','query', 'localSymbol','EUR.USD', ...
'NumberOfRows',3, 'QuotesNumber',-1)
dataStruct =
reqId: 464879608
reqTime: '16-Dec-2014 14:46:47'
lastEventDateNum: 735949.615954282
lastEventDateTime: '16-Dec-2014 14:46:57'
symbol: 'EUR'
localSymbol: 'EUR.USD'
isActive: 1
quotesReceived: 362
quotesToReceive: 1000
bid: [1x3 struct]
ask: [1x3 struct]
contract: [1x1 struct]

>> dataStruct.bid(1)
ans =
price: 1.2546
size: 6560000
marketMaker: ''
dateNum: 735949.615954271
dateTime: '16-Dec-2014 14:46:57'

Streaming market-depth update events can be captured and processed using the CallbackUpdateMktDepth and CallbackUpdateMktDepthL2 parameters. A usage example of a continuously-updating order-book GUI is provided in §11.5.

Note: market-depth quotes are sent from the IB server at a much higher rate than streaming quotes. For EUR.USD at a specific date-time, there were 2-3 streaming quotes per second, compared to 30-50 market-depth updates per second.

As with streaming quotes (§7.1), streaming market-depth updates can be stopped by resending the query request with QuotesNumber=0 or disconnecting from IB (§13).