6 Streaming data

Streaming data is a near-real-time mechanism, where IQFeed sends ongoing asynchronous update messages to IQML of tick (quote and trade) and news events.

These events are accumulated in IQML memory buffers. They can either be queried asynchronously (via ad-hoc queries that “peek” at the latest accumulated data without disturbing the ongoing streaming, as shown in §6.1-§6.4), or handled synchronously (immediately as each event is received, using callbacks (§10) or alerts (§11)).

Depending on your IQFeed subscription, streaming may be delayed by 10+ minutes compared to a real-time feed (a real-time data subscription is needed for live data).97

Request IQFeed to start sending a stream of quotes for a specified security. This is done by using the 'quotes' action and setting a NumOfEvents parameter to a positive >1 value.

At any later time(s), you can access the accumulated quotes using either of the following alternatives:

Use 'quotes' action and NumOfEvents of -1 (minus one). This will return the latest streamed data, without stopping the background streaming.

If you set the AssignTo variable in the original request, you can directly access the streamed data in the specified base workspace variable.

For example, to request 100 streaming quotes for a continuous VIX future contract:

IQFeed will start sending quotes to IQML in the background, up to the specified NumOfEvents, without affecting normal Matlab processing. You can continue to work in Matlab, process/display information etc., while quotes accumulate in the background.

Quotes will only stream in the background in non-blocking mode. If you assign the IQML command results to a variable, the request is treated as blocking and IQML will wait for all the events to accumulate (or Timeout to occur), as described in §4.1:

NumOfEvents can be any number >1 for streaming (a value of 1 is the standard snapshot query in §4.1). To collect streaming quotes endlessly, set NumOfEvents to inf. Note that in Matlab, inf is a number (not a string), so do not enclose it in quotes ('inf').

The quotes are collected into an internal data buffer in IQML. A different buffer is maintained for each symbol. The buffer size can be controlled using the MaxItems parameter, which has a default value of 1. This means that by default only the latest streaming quote of each type (bid/ask) is stored, along with high/low/close data.

If you set a higher value for MaxItems,98 then up to the specified number of latest quotes will be stored. For example, to store the latest 5 quotes:

Note: MaxItems increases memory usage, multiplied by the number of streamed symbols.99

Subsequent requests to retrieve the latest accumulated quotes buffer data, without stopping the background streaming, should use NumOfEvents < 0 (e.g. -1). These requests return a Matlab data struct similar to the following:

>> data = IQML('quotes', 'Symbol','@VX#', 'NumOfEvents',-1)
data =
Symbol: '@VX#'
Command: 'w@VX#'
isActive: 1
EventsToProcess: 100
EventsProcessed: 57
LatestEventDatenum: 737128.637260451
LatestEventTimestamp: '20180309 15:17:39'
DataType: 'quotes and trades'
ProcessType: 'stream'
AssignTo: ''
errorMsg: ''
BufferSize: 5
Buffer: [5×1 struct]
LatestData: [1×1 struct]

isActive – logical flag indicating whether quotes are currently streamed for this security. When NumOfEvents ticks are received, this flag is set to false (0).

EventsToProcess – total number of streaming ticks requested for the security (using the NumOfEvents parameter).

EventsProcessed – number of streaming ticks received for this security. When EventsProcessed >= EventsToProcess, streaming quotes are turned off and isActive is set to false (0). Note that it is possible that EventsProcessed > EventsToProcess, since it takes a while for the streaming cancellation request to reach IQFeed, and during this time a few additional ticks may have arrived.

LatestEventDatenum – Matlab numeric datenum representation of the LatestEventTimestamp (local timezone).

LatestEventTimestamp – timestamp (string format) when this quote event was received by IQML (local timezone).

AssignTo – the name of the assigned variable in the base Matlab workspace (=AssignTo parameter, see below). In normal cases, this field is empty ('').

errorMsg – contains the error message in case streaming cannot be processed (for example: 'Symbol not found'). In normal cases, this field is empty ('').

Here is a simulated timeline that illustrates the use of streaming data in IQML:

Different quotes are sent independently from IQFeed server with a unique timestamp. Note: data.LatestEventDatenum and data.LatestEventTimestamp are specified in local time-zone. In contrast, data.LatestData.Most_Recent_Trade_Time and data.Buffer.Most_Recent_Trade_Time use the server time-zone, typically US Eastern.

To get the quotes data, simply read the fields of the returned data struct, for example:100

>> data.LatestData
ans =
Symbol: '@VX#'
Most_Recent_Trade: 17.08
Most_Recent_Trade_Size: []
Most_Recent_Trade_Time: '08:06:20.716000'
Most_Recent_Trade_Market_Center: 32
Total_Volume: 4507
Bid: 17.05
Bid_Size: 63
Ask: 17.1
Ask_Size: 244
Open: 17.2
High: 17.35
Low: 17
Close: 17.23
Message_Contents: 'Cbasohlcv'
Message_Description: 'Last qualified trade; A bid update
occurred; An ask update occurred; A
settlement occurred; An open declaration
occurred; A high declaration occurred; A
low declaration occurred; A close decla-
ration occurred; A volume update occurred'
Most_Recent_Trade_Conditions: '4D'
Trade_Conditions_Description: 'Implied'
Most_Recent_Market_Name: 'CBOE Futures Exchange (CFE)'

Note that data.LatestData is typically the same as data.Buffer(end), regardless of the values of MaxItems or NumOfEvents.101

Each streaming security asset can have a different BufferSize, by specifying a different MaxItems value in the command (large for heavily-traded assets, small for others).

Once the data is retrieved, you can direct IQML to clear (empty) the internal Buffer, by setting ClearBuffer to true or 1. The latest buffer will be returned, and the internal Buffer (but no other field) will be immediately emptied, awaiting new streaming quotes:102

To stop collecting streaming quotes, simply resend a request with NumOfEvents=0:

IQFeed reports 16 standard data fields by default. If you have the Professional (or trial) IQML license, you can customize the returned data fields by requesting up to 50+ additional fields, removing standard fields, and setting the order of the reported fields. This can be done using the Fields parameter, as explained in §4.1. For example:

Note: setting the Fields parameter only affects streaming data from then on – it does not affect previously-received data. Therefore, you should set Fields parameter in the initial streaming command, not ongoing data-fetch commands (NumOfEvents<0).

When DataType is 'q' or 'quotes', whenever any of the requested data fields (either the standard 16 fields, or a customized set) gets updated (not necessarily to a different value), a new tick (update/quote) message is sent/streamed. Adding data fields means a corresponding increase in tick messages. It is not possible in IQFeed to request data fields without the corresponding update messages for these fields (or vice versa). The only exception to this rule is setting DataType to 't' or 'trades': in this case only trade updates (containing all the requested fields) will be streamed, but no field updates.

In summary, the fewer data fields that are requested, the faster the run-time processing, and the lower the corresponding tick message rate, thus enabling a larger number of usable quotes to be streamed and processed by your Matlab program each second.

You can specify multiple symbols for streaming at the same time, in a single IQML command, by specifying a colon-delimited or cell-array list of symbols. For example:

>> data = IQML('quotes', 'symbol','IBM:GOOG:AAPL', 'numOfEvents',-1);
data =
1×3 struct array with fields:
Symbol
Command
isActive
EventsToProcess
EventsProcessed
LatestEventDatenum
LatestEventTimestamp
DataType
ProcessType
BufferSize
Buffer
LatestData

>> data(1).LatestData
ans =
struct with fields:
Symbol: 'IBM'
Most_Recent_Trade: 142.48
Most_Recent_Trade_Size: 41149
Most_Recent_Trade_Time: '17:33:40.531781'
Most_Recent_Trade_Market_Center: 19
...

To get the latest data for all streamed symbols, omit the Symbol parameter (or set it to empty ['']) in the IQML command. Note: this will return both active and non-active streams:

>> data = IQML('quotes', 'numOfEvents',-1); % no symbol: return ALL streams
data =
1×5 struct array with fields:
Symbol
Command
...

Similarly, to cancel all active streams in a single command, omit Symbol (or set it to ''):103

IQFeed typically allows streaming up to 500 symbols. This limit can be increased by paying DTN for increased data subscriptions. In any case, the actual maximal number of concurrently-streaming symbols is limited by performance considerations (see §3.6).

Note: Streaming quote messages are automatically processed by IQML in the background, reducing the CPU time that is left available to process other IQML queries (e.g., a blocking historical data query) or Matlab analysis functions. It is therefore advisable to stop streaming IQFeed data when not needed, even if only temporarily.

In cases of high load (such as market open/close), numerous streaming messages (quote ticks, news items etc.) may flood the system, causing Matlab to appear frozen for long minutes, until the events backlog is fully processed. For such cases, use IQML’s built-in overflow protection mechanism with OverflowMode and MaxEventsBacklog parameters. By default, OverflowMode is 0 (no protection); when OverflowMode is set to 1, then in cases of extreme load, if more than MaxEventsBacklog messages (default=100) have still not yet been processed, then any additional messages will be immediately dropped (ignored) and not processed. This allows Matlab time to process the backlog, before additional new messages are accepted for processing. The downside is that new (more recent) messages are ignored, whereas the older messages in the backlog are processed. Alternatively, set OverflowMode to 2 to ignore the older messages in the backlog and only process the newer messages. This OverflowMode=2 alternative is a bit slower than the OverflowMode=1 alternative, so the messages processing rate of OverflowMode=1 will unusally be higher. In summary:

OverflowMode=0 does not protect from messages overflow; all messages will be processed. Matlab may appear frozen when flooded with IQFeed events.

OverflowMode=1 will drop new messages when the backlog is too large; this is the fastest overflow handling alternative, having minimal CPU overhead.

OverflowMode=2 will drop old messages when the backlog is too large; this is slower than OverflowMode=1, but more responsive to new messages.

OverflowMode is only relevant for streaming data. Overflow protection is automatically disabled (OverflowMode set to 0) for any other (blocking) query. For example, if you request blocking market data (NumOfEvents>0) while quotes are streaming, overflow handling will be disabled from then onward.

OverflowMode and MaxEventsBacklog are specific to each of the major IQFeed data query ports: Level1, Level2, and Lookup. When you specify these parameters in a query, they are automatically assigned to the appropriate data port and only affect the messages arriving via that port. For example, you can set different overflow handling parameters for Level1 and Level2 data. Similarly, a blocking historic data query (using the Lookup port) does not disable overflow handling for streaming quotes (using the Level1 port).

Note that during non-trading hours, there is no streaming data (naturally). To test the streaming-data mechanism during non-trading hours, try to use the dummy symbol TST$Y, for which IQFeed sends a continuous 24/7 stream of pre-recorded data.104

When debugging streaming, it is sometimes useful to assign the streamed data to a base Matlab workspace variable, which can be monitored live using Matlab’s Variable Explorer or directly checked/used within a Matlab script, without a need to call IQML('quotes',…,'numOfEvents',-1) to constantly refetch the latest data. This data mirroring can be done by setting the AssignTo parameter to any valid Matlab variable name or assignment target, for example: 'myData' or 'streamed.VX' or 'data{3}'.

>> myData
myData =
struct with fields:
Symbol: '@VX#'
Command: 'w@VX#'
isActive: 1
EventsToProcess: Inf
EventsProcessed: 129
LatestEventDatenum: 737902.831878183
LatestEventTimestamp: '20200421 19:57:54'
DataType: 'quotes and trades'
ProcessType: 'stream'
AssignTo: 'myData'
errorMsg: ''
BufferSize: 100
Buffer: [100×1 struct]
LatestData: [1×1 struct]

Regional quotes are Bid and Ask prices delivered from various regional markets (exchanges). The streaming regional market update mechanism has two parts, just like streaming ticks (§6.1):

Request IQFeed to start sending a stream of regional updates. This is done by using the 'regional' action and setting a NumOfEvents parameter to a positive >1 value. You must specify the Symbol(s) for which regional updates will stream.

At any later time(s), you can access the accumulated regional updates using either of the following alternatives:

Use 'regional' action and NumOfEvents of -1 (minus one). This will return the latest streamed data, without stopping the background streaming.

If you set the AssignTo variable in the original request, you can directly access the streamed data in the specified base workspace variable.

This causes IQFeed to start sending regional updates to IQML in the background, up to the specified NumOfEvents, without affecting normal Matlab processing. You can continue to work with Matlab, process and display information etc., while the regional updates accumulate in the background.

Regional updates will only stream in the background in non-blocking mode. If you assign the IQML command results to a variable, the request is treated as blocking and IQML will wait for all data to accumulate (or Timeout to occur), as described in §7.2:

NumOfEvents can be any number higher than 1 for streaming to work (a value of 1 is the standard snapshot regional-update request described in §7.2). To collect streaming regional updates endlessly, set NumOfEvents to the value inf. Note that in Matlab, inf is a number (not a string), so do not enclose it in quotes ('inf').

The regional updates are collected into an internal data buffer in IQML. A different buffer is maintained for each symbol. The buffer size can be controlled using the MaxItems parameter, which has a default value of 1106. This means that by default only the latest streaming regional update that affect the specified symbols will be stored in the buffer and become accessible for later processing.

If you set a higher value for MaxItems, then up to the specified number of latest regional update items will be stored. For example, to store the latest 5 updates:

Note that using a large MaxItems increases memory usage. This could have an adverse effect if you set a very large buffer size (many thousands) and/or streaming of a large number of different securities.107

Subsequent requests to retrieve the latest accumulated regional updates buffer data, without stopping the background streaming, should use NumOfEvents = -1 (minus one). These requests return a Matlab data struct similar to the following:

>> data = IQML('regional', 'Symbol','FB', 'NumOfEvents',-1)
data =
Symbol: 'FB'
Command: 'S,REGON,FB'
isActive: 1
EventsToProcess: 100
EventsProcessed: 83
LatestEventDatenum: 737146.784037153
LatestEventTimestamp: '20180327 18:49:00'
DataType: 'regional'
ProcessType: 'stream'
AssignTo: ''
errorMsg: ''
BufferSize: 5
Buffer: [5×1 struct]
LatestData: [1×1 struct]

isActive – a logical flag indicating whether regional updates are currently being streamed for this security. When NumOfEvents ticks have been received, this flag is set to false (0).

EventsToProcess – total number of streaming regional updates requested (using the NumOfEvents parameter).

EventsProcessed – number of streaming regional updates received. When EventsProcessed >= EventsToProcess, streaming updates are turned off and isActive is set to false (0).
Note that it is possible that EventsProcessed > EventsToProcess, since it takes a while for the streaming cancellation request to reach IQFeed and during this time a few additional update messages may have arrived.

LatestEventDatenum – Matlab numeric datenum representation of the LatestEventTimestamp (local timezone).

LatestEventTimestamp – timestamp (string format) when this regional update event was received by IQML (local timezone).

AssignTo – the name of the assigned variable in the base Matlab workspace (=AssignTo parameter, see below). In normal cases, this field is empty ('').

errorMsg – contains the error message in case streaming cannot be processed (for example: 'Symbol not found'). In normal cases, this field is empty ('').

To get the regional updates data, simply read the fields of the returned data struct:108

>> data.LatestData
ans =
RegionalBid: 155.34
RegionalBidSize: 100
RegionalBidTime: '12:29:45'
RegionalAsk: 155.55
RegionalAskSize: 200
RegionalAskTime: '12:29:45'
FractionDisplayCode: 14
DecimalPrecision: 4
FractionDisplayDescription: 'Four decimal places'
MarketCenter: 11
MarketCenterDescription: 'NYSE Archipelago (NYSE_ARCA)'

Each update has an associated timestamp, since different regional updates are sent separately and independently from IQFeed server.

Note: data.LatestEventDatenum and data.LatestEventTimestamp are specified in the local time-zone; in contrast, data.LatestData.RegionalBidTime and .RegionalAskTime are specified in the server’s time-zone (typically US Eastern time zone).

Note that data.LatestData is typically the same as data.Buffer(end), regardless of the values of MaxItems or NumOfEvents.109

Each streaming security asset can have a different BufferSize, by specifying a different MaxItems value in the streaming command. This can be used for specifying a larger MaxItems for heavily-traded assets vs. lightly-traded ones.

To stop collecting regional updates, simply resend a request with NumOfEvents=0:

You can specify multiple symbols for streaming at the same time, in a single IQML command, by specifying a colon-delimited or cell-array list of symbols. For example:

As with streaming quotes (§6.1), to get the latest data for all streamed symbols, omit the Symbol parameter or set it to empty ['']. This returns all streams (both active/not):

>> data = IQML('regional', 'numOfEvents',-1); % no symbol: get ALL streams
data =
1×5 struct array with fields:
Symbol
Command
isActive
EventsToProcess
...

Similarly, to cancel all active streams in a single command, omit Symbol (or set it to ''):111

When debugging streaming, it is sometimes useful to assign the streamed data to a base Matlab workspace variable, which can be monitored live using Matlab’s Variable Explorer or directly checked/used within a Matlab script, without a need to call IQML('regional',…,'numOfEvents',-1) to constantly refetch the latest data. This data mirroring can be done by setting the AssignTo parameter to any valid Matlab variable name or assignment target, for example: 'myData' or 'streamed.FB' or 'data{3}'.

>> myData
myData =
Symbol: 'FB'
Command: 'S,REGON,FB'
isActive: 1
EventsToProcess: 100
EventsProcessed: 83
...

Here is a summary of the IQML parameters that affect streaming regional updates:

The streaming interval bars feature has two parts, just like streaming ticks (§6.1):

Request IQFeed to start sending a stream of interval bars for a specified security. This is done by using the 'intervalbars' action and setting a NumOfEvents parameter to a positive >1 value.

At any later time(s), you can access the accumulated interval bars using either of the following alternatives:

Use 'intervalbars' action and NumOfEvents of -1 (minus one). This will return the latest streamed data, without stopping the background streaming.

If you set the AssignTo variable in the original request, you can directly access the streamed data in the specified base workspace variable.

For example, request 80 streaming interval bars of a continuous VIX future contract:

This causes IQFeed to start sending interval bars to IQML in the background, up to the specified NumOfEvents, without affecting normal Matlab processing. This means you can continue to work with Matlab, process data, display information etc.

IQML('intervalbars', 'Symbol','@VX#', 'NumOfEvents',80); % streaming, non-blocking

NumOfEvents can be any number higher than 1 for streaming to work. To collect streaming quotes endlessly, set NumOfEvents to the value inf. Note that in Matlab, inf is a number (not a string), so do not enclose it in quotes ('inf').

If you set a higher value for MaxItems,113 then up to the specified number of latest quotes will be stored, subject to IQFeed server limitations:114

Note that using a large MaxItems increases memory usage. This may have an adverse effect if you use a large buffer (many thousands) and/or stream multiple Symbols.115

Subsequent requests to retrieve the latest accumulated interval bars buffer data, without stopping the background streaming, should use NumOfEvents = -1 (minus one). These requests return a Matlab data struct similar to the following:

>> data = IQML('intervalbars', 'Symbol','@VX#', 'NumOfEvents',-1)
data =
Symbol: '@VX#'
Command: 'BW,@VX#,60,,1,3,,,B'
isActive: 0
EventsToProcess: 80
EventsProcessed: 80
LatestEventDatenum: 737902.504044143
LatestEventTimestamp: '20200421 12:05:49.414'
DataType: 'intervalbars'
ProcessType: 'stream'
AssignTo: ''
errorMsg: ''
MaxDaysToProcess: 1
BufferSize: 3
Buffer: [3×1 struct]
LatestData: [1×1 struct]

isActive – logical flag indicating whether interval bars are currently streamed for the security. Once NumOfEvents bars are received this flag is set to false (0).

EventsToProcess – total number of streaming interval bars requested for the security (using the NumOfEvents parameter).

EventsProcessed – number of streaming interval bars received for this security. When EventsProcessed>=EventsToProcess, streaming is turned off and isActive is set to false (0). Note: it is possible that EventsProcessed > EventsToProcess, since it takes a while for the streaming cancellation request to reach IQFeed, and during this time a few additional bars may have arrived.

LatestEventDatenum – Matlab numeric datenum representation of the LatestEventTimestamp (local timezone).

LatestEventTimestamp – timestamp (string format) when this bar event was received by IQML (local timezone).

AssignTo – the name of the assigned variable in the base Matlab workspace (=AssignTo parameter, see below). In normal cases, this field is empty ('').

errorMsg – contains the error message in case streaming cannot be processed (for example: 'Symbol not found'). In normal cases, this field is empty ('').

To retrieve the interval bars data, read the fields of the returned data struct:116

>> data.LatestData
ans =
Symbol: '@VX#'
BarType: 'Complete interval bar from history'
Timestamp: '2020-04-21 04:50:00'
Open: 38.95
High: 38.95
Low: 38.85
Close: 38.9
CumulativeVolume: 6381
IntervalVolume: 18
NumberOfTrades: 0
BarTypeCode: 'BH'
EventDatenum: 737902.504044143
EventTimestamp: '20200421 12:05:49.414'

Note that data.LatestData is typically the same as data.Buffer(end), regardless of the values of MaxItems or NumOfEvents.117

IQFeed sends interval bars asynchronously, with a Timestamp that identifies each bar (server timezone, typically US Eastern timezone). Note that data.LatestEventDatenum= data.LatestData.EventDatenum and data.LatestEventTimestamp=data.LatestData.EventTimestamp are specified in the local timezone (your computer’s time). Also note that LatestEventTimestamp and EventTimestamp have a different format than Timestamp.

The data.LatestData.NumberOfTrades field indicates the number of trades that occurred within this bar (i.e., not cumulative). This field is only relevant when IntervalType is 'ticks'/'trades'. In all other cases, the field will be empty ([]) or 0.

The IntervalType (default: 'secs') and IntervalSize (default: 60) parameters should typically be specified together. Note that IntervalSize must be a positive integer value (i.e. its value cannot be 4.5 or 0). If IntervalType is 'ticks'/'trades', IntervalSize must be 2 or higher. If IntervalType is 'volume', IntervalSize must be 100 or higher. If IntervalType is 'secs', IntervalSize must be any integer between 1-300 (5 minutes), or any multiple of 60 (1 minute) between 300-3600 (1 hour), or 7200 (2 hours).118

The data.LatestData.BarType and BarTypeCode fields indicate if this is a historic bar (BH), a complete bar from the live (real-time) stream (BC), or an update bar (BU).

Unlike other streaming types, intervalbar queries also fetch historic bars data, starting from the date/time that is set by the BeginDateTime parameter. This is similar to (and subject to the same limitations as) historic interval data (§5.4), but with no specified end point. BeginDateTime’s default value is 00:00:00 today (server time), so we almost always get historic (BH) bars before live streaming (BC) bars. IQML will immediately return the historic bars, and new realtime streaming bars as they become available. If we run the query at mid-day, we may get hundreds of historic bars before the first live streaming bar. Depending on the specified BeginDateTime and NumOfEvents, we may receive only historic bars without any live streaming bars. Historic and streaming data bars can be distinguished based on their BarType field.

Note: the initial set of historic bars also includes the latest (incomplete) bar. For example, if we query at 21:34, we get the complete historic (BH) intervalbars for 21:20, 21:25, 21:30 and also the current (incomplete) bar for 21:35. A minute later, at exactly 21:35:00, IQFeed sends the complete BC bar for 21:35. This BC bar is added at the end of the buffer, so we will see two 21:35 bars in the buffer.

By default, IQFeed only sends historic BH bars followed by streaming BC bars. If the MaxUpdateDuration>0, IQFeed also sends BU bar update messages when the specified number of seconds have passed since the last trade or BC bar message. For example, if IntervalSize=60 and MaxUpdateDuration=15, the following scenario may occur:119

9:30:03 - A trade occurs - no message is sent by IQFeed (15-sec timer is reset to 0)
9:30:06 - A trade occurs - no message is sent by IQFeed (15-sec timer is reset to 0)
9:30:21 - No trade or BC message for 15 seconds, so IQFeed sends an update (BU) message
9:30:57 - A trade occurs - no message is sent by IQFeed (15-sec timer is reset to 0)
9:31:12 - No trade or BC message for 15 seconds, so IQFeed sends a new update (BU) message
9:31:15 - A trade occurs - this is the first trade outside the 9:30 minute, so IQFeed sends a bar complete (BC) message for the 9:30 bar. (15-sec timer is reset to 0)
9:31:29 - A trade occurs - no message is sent by IQFeed (15-sec timer is reset to 0)
9:31:40 - A trade occurs - no message is sent by IQFeed (15-sec timer is reset to 0)
9:31:53 - A trade occurs - no message is sent by IQFeed (15-sec timer is reset to 0)
9:32:03 - A trade occurs - IQFeed sends a bar complete (BC) message for the 9:31 bar; only 10 seconds have passed since the last trade, so no BU message is sent by IQFeed before the BC.

Note: IQFeed only sends BC bar complete messages when the first trade outside the bar occurs. This may occur a long time after the bar has ended. Use the MaxUpdateDuration parameter to receive more timely BU bar update messages.

BU update bars are appended to the buffer like other bar messages, so we may see bars with the same Timestamp. These bars do not necessarily have the same data, since they were sampled at different times (per their respective EventDatenum, EventTimestamp). The most updated data is always the last bar for each unique Timestamp. For example:

>> IQML('intervalbars', 'Symbol','@VX#', 'NumOfEvents',inf, 'MaxItems',9, ...
'IntervalSize',60, 'IntervalType','secs', 'MaxUpdateDuration',15);

>> data = IQML('intervalbars', 'Symbol','@VX#', 'NumOfEvents',-1);
>> struct2table(data.Buffer)
ans =
9×12 table
Symbol BarType Timestamp Open …
_______ ____________________________________ _____________________ _____ …
'@VX#' 'Complete interval bar from history' '2020-04-21 03:51:00' 39.45 …
'@VX#' 'Complete interval bar from history' '2020-04-21 03:52:00' 39.45 …
'@VX#' 'Complete interval bar from history' '2020-04-21 03:53:00' 39.35 …
'@VX#' 'Complete interval bar from stream' '2020-04-21 03:53:00' 39.35 …
'@VX#' 'Updated interval bar' '2020-04-21 03:54:00' 39.45 …
'@VX#' 'Complete interval bar from stream' '2020-04-21 03:54:00' 39.45 …
'@VX#' 'Updated interval bar' '2020-04-21 03:55:00' 39.45 …
'@VX#' 'Updated interval bar' '2020-04-21 03:55:00' 39.45 …
'@VX#' 'Complete interval bar from stream' '2020-04-21 03:55:00' 39.45 …

data = IQML('intervalbars', 'symbol','IBM', 'NumOfEvents',-1, ...
'ClearBuffer',true);

You can specify multiple symbols for streaming at the same time, in a single IQML command, by specifying a colon-delimited or cell-array list of symbols. For example:

As with streaming quotes (§6.1), to get the latest data for all streamed symbols, omit the Symbol parameter or set it to empty ['']. This returns all streams (both active/not). In the following example, we get a struct array of size 5, one struct for each symbol for which that we have requested intervalbars during this Matlab session:

>> data = IQML('intervalbars', 'numOfEvents',-1); % no symbol: get ALL streams
data =
1×5 struct array with fields:
Symbol
Command
isActive
...

Similarly, to cancel all active streams in a single command, omit Symbol (or set it to ''):

When debugging streaming, it is sometimes useful to assign the streamed data to a base Matlab workspace variable, which can be monitored live using Matlab’s Variable Explorer or directly checked/used within a Matlab script, without a need to call IQML('intervalbars',…,'numOfEvents',-1) to constantly refetch the latest data. This data mirroring can be done by setting the AssignTo parameter to any valid Matlab variable name or assignment target, for example: 'myData' or 'streamed.VX' or 'data{3}'.

>> myData
myData =
struct with fields:
Symbol: '@VX#'
Command: 'BW,@VX#,60,20200421 000000,7,100,,,B,s,,5'
isActive: 1
EventsToProcess: Inf
EventsProcessed: 129
LatestEventDatenum: 737902.831878183
LatestEventTimestamp: '20200421 19:57:54'
DataType: 'intervalbars'
ProcessType: 'stream'
AssignTo: 'myData'
errorMsg: ''
MaxDaysToProcess: 7
BufferSize: 100
Buffer: [100×1 struct]
LatestData: [1×1 struct]

Parameter	Data type	Default	Description
Symbol or Symbols121	colon or comma-delimited string, or cell-array of strings	(none)	Limits the request to the specified symbol(s). Examples: '@VX#' 'IBM:AAPL:GOOG' 'IBM,AAPL,GOOG' {'IBM', 'AAPL', 'GOOG'} This parameter must be set to valid symbol name(s) when NumOfEvents>0
NumOfEvents	integer	Max Items	One of: inf – continuous endless streaming interval bars for specified symbol(s) N>1 – stream only N interval bars 1 – get only a single interval bar 0 – stop streaming interval bars -1 – return latest interval bars data while continuing to stream new bars
MaxItems	integer	100	Returns up to the specified number of bars (if available). This is the max Buffer size.
MaxDays	integer	1	Max number of trading days to retrieve
IntervalType	string	'secs'	Sets the type of interval size. One of the following values: 's' or 'secs' – time [seconds] (default) 'v' or 'volume' – traded volume 't' or 'ticks' – number of ticks
IntervalSize	integer	60	Size of bars in IntervalType units. Must be ≥1 for secs, ≥2 for ticks, ≥100 for volume.
BeginFilterTime	string	'00:00:00'	Only return bars that begin after this time of day (US Eastern time-zone). Format: 'hhmm', 'hh:mm', 'hhmmss' or 'hh:mm:ss'.
EndFilterTime	string	'23:59:59'	Only return bars that end before this time of day (US Eastern time-zone). Format: 'hhmm', 'hh:mm', 'hhmmss' or 'hh:mm:ss'.
BeginDateTime	integer or string or datetime object	'' (empty string) meaning today at 00:00:00	Only return bars that begin after this date/time (US Eastern time-zone). Format: Matlab datenum, or 'yyyymmdd hhmmss', or 'yyyy-mm-dd hh:mm:ss' etc. Note: there is no corresponding EndDateTime parameter for streaming intervalbars (only for historic bars: §5.4).
MaxUpdateDuration	integer	0	Max number of seconds to wait after a trade before receiving a bar update message
Timeout	number	5.0	Max number of seconds to wait (0-9000) for data in blocking mode (0 means infinite)
ClearBuffer	logical (true/false)	false	If true or 1, the internal cyclic quotes buffer is cleared after data is returned to the caller
AssignTo	string	'' (empty string)	Contains the assignment target (typically a variable name) in the base Matlab workspace, useful for stream debugging.
OverflowMode	integer	[]	If set, overflow handling for the query’s IQFeed data port is updated as follows: 0 – no overflow handling (all messages will be processed) 1 – drop new messages when backlog is too large 2 – drop old messages (slower)
MaxEventsBacklog	integer	100	Size of messages backlog, per IQFeed data port, above which overflow handling is done (if OverflowMode is 1 or 2).

The streaming market depth mechanism also has two distinct parts, just like streaming level 1 quotes (§6.1):

Request IQFeed to start sending a stream of market depth quotes for a specified security. This is done by using the 'marketdepth' action.

At any later time(s), you can access the current market depth data using either of the following alternatives:

Use 'marketdepth' action and NumOfEvents of -1 (minus one). This will return the latest streamed data, without stopping the background streaming.

If you set the AssignTo variable in the original request, you can directly access the streamed data in the specified base workspace variable.

For example, let’s request market depth quotes for a continuous E-mini contract:

This causes IQFeed to start sending market depth updates to IQML in the background, up to the specified NumOfEvents, without affecting normal Matlab processing. This means you can continue to work with Matlab, process data, display information etc.

Note that each incoming quote message updates the data for a single market depth row. The market depth row cannot be specified nor predicted by the user, and the order of messages is unrelated to the market depth row, for example, an update for row #3 can follow an update of row #5.

Market depth data will only stream in the background in non-blocking mode. If you assign the IQML command results to a variable, the request is treated as blocking and IQML will wait for all the events to accumulate (or Timeout to occur), as described in §4.1:

IQML('marketdepth', 'Symbol','@ES#', 'NumOfEvents',600); % streaming, non-blocking

NumOfEvents is an optional input parameter and can be any number >1 for streaming. To collect market depth data endlessly, set NumOfEvents to the value inf. Note that in Matlab, inf is a number (not a string), so do not enclose it in quotes ('inf').

The quotes are collected into an internal data structure in IQML. A different structure is maintained for each symbol.

>> data = IQML('marketdepth', 'Symbol','@ES#', 'NumOfEvents',-1)
data =
Symbol: '@ES#'
Command: 'w@ES#'
EventsToProcess: 600
EventsProcessed: 437
LatestEventDatenum: 737195.518211377
LatestEventTimestamp: '20180515 12:26:13'
AssignTo: ''
errorMsg: ''
IncludeEmptyQuotes: 0
LatestData: [1×10 struct]

EventsToProcess – total number of streaming interval bars requested for the security (using the NumOfEvents parameter).

EventsProcessed – number of streaming market depth data quotes received for this security. When EventsProcessed >= EventsToProcess, streaming market depth data for this security is turned off.

LatestEventDatenum – Matlab numeric datenum representation of the LatestEventTimestamp (local timezone).

LatestEventTimestamp – timestamp (string format) when this market depth quote event was received by IQML (local timezone).

AssignTo – the name of the assigned variable in the base Matlab workspace (=AssignTo parameter, see below). In normal cases, this field is empty ('').

errorMsg – contains the error message in case streaming cannot be processed (for example: 'Symbol not found'). In normal cases, this field is empty ('').

IncludeEmptyQuotes – value of the specified IncludeEmptyQuotes parameter (default value: false). If true or 1, then empty quotes (having no valid Bid or Ask) will be reported and be counted as a valid “event”; otherwise they will not.

To retrieve the market depth data at the nth market depth row, simply read the fields of the LatestData at the nth location, for example:

>> data.LatestData(4)
ans =
Symbol: '@ES#'
ID: 'MD04'
Bid: 2725.5
Ask: 2727.25
BidSize: 65
AskSize: 148
BidTime: '05:25:59.761191'
Date: '2018-05-15'
AskTime: '05:25:59.760278'
BidInfoValid: 1
AskInfoValid: 1
Condition: 52
Condition_Description: 'regular'
ID_Description: 'Order book row #4'

BidInfoValid and AskInfoValid values are logical (true/false) values, which appear as 1 or 0, respectively, in the struct display above. The ID field indicates the corresponding order-book row (for futures) or market-maker name (for equities).

Different market depth quotes are sent independently from the IQFeed server with a unique timestamp, in a non-ordered manner. Note that data.LatestEventDatenum and data.LatestEventTimestamp are specified in the local time-zone.

Note: unlike streaming quotes (§6.1), regional updates (§6.2), and interval bars (§6.3), the streaming market depth mechanism does not store an internal buffer of quote updates, so there is no Buffer field and previous updates are not stored. Only the latest snapshot of the order book (in the LatestData field) is updated.

To stop collecting market depth quotes for a security, simply send the request again, this time with NumOfEvents=0.

You can specify multiple symbols for streaming at the same time, in a single IQML command, by specifying a colon-delimited or cell-array list of symbols. For example:

As with the blocking request (§4.4), you’ll receive an error message when requesting market depth info from an exchange for which you have no Level 2 data subscription:

As with streaming quotes (§6.1), to get the latest data for all streamed symbols, omit the Symbol parameter or set it to empty ['']. This returns all streams (both active/not):

>> data = IQML('marketdepth', 'numOfEvents',-1); % no symbol: get ALL streams
data =
1×5 struct array with fields:
Symbol
Command
isActive
EventsToProcess
...

Similarly, to cancel all active streams in a single command, omit Symbol (or set it to ''):

When debugging streaming, it is sometimes useful to assign the streamed data to a base Matlab workspace variable, which can be monitored live using Matlab’s Variable Explorer or directly checked/used within a Matlab script, without a need to call IQML('marketdepth',…,'numOfEvents',-1) to constantly refetch the latest data. This data mirroring can be done by setting the AssignTo parameter to any valid Matlab variable name or assignment target, for example: 'myData' or 'streamed.VX' or 'data{3}'.

>> myData
myData =
struct with fields:
Symbol: '@ES#'
Command: 'w@ES#'
EventsToProcess: 600
EventsProcessed: 437
LatestEventDatenum: 737195.518211377
LatestEventTimestamp: '20180515 12:26:13'
AssignTo: 'myData'
errorMsg: ''
IncludeEmptyQuotes: 0
LatestData: [1×10 struct]

The following parameters affect market depth data queries (see §4.4 for details):

Parameter	Data type	Default	Description
Symbol or Symbols 122	colon or comma-delimited string, or cell-array of strings	(none)	Limits the request to the specified symbol(s). Examples: '@ES#' 'IBM:AAPL:GOOG' 'IBM,AAPL,GOOG' {'IBM', 'AAPL', 'GOOG'} This parameter must be set to valid symbol name(s) when NumOfEvents>0
NumOfEvents	integer	10	One of: inf – continuous endless streaming Level 2 data for specified symbol(s) N>1 – only process N incoming quotes 1 – get only a single quote 0 – stop streaming market depth data -1 – return the latest Level 2 data while continuing to stream new data updates
MaxLevels	integer	5	Max number of data levels (rows) to return
IncludeEmptyQuotes	logical (true/false)	false	If set to true or 1, empty Level 2 quotes (with neither a valid Bid nor valid Ask) will also be returned. By default (false), they will not be.
Detailed	logical (true/false)	false	If set to true or 1, each price level’s components will be reported: trade order IDs for futures; market makers for equities. This parameter is ignored in IQFeed client/protocol 6.1 or older.
ClearBuffer	logical (true/false)	false	If true or 1, the internal cyclic quotes buffer is cleared after the data is returned to the caller.
AssignTo	string	'' (empty string)	Contains the assignment target (typically a variable name) in the base Matlab workspace, useful for stream debugging.

Note: Market Depth (Level 2) data is only available in the Professional IQML license.

97 See §9.3 for a programmatic method to determine whether your exchange subscription is delayed or real-time.

98 MaxItems is a numeric parameter like NumOfEvents, so don’t enclose the parameter value within string quotes (‘’)

99 Quotes use up to ~16KB of Matlab memory (depending on the selected Fields). So, if MaxItems=1000, each streamed symbol would need up to 1000*16KB = 16MB of Matlab memory when its buffer becomes full (which could take a while). Streaming 100 symbols will require up to ~1.6GB. Such large memory usage might significantly degrade overall performance.

100 The textual description fields depend on the MsgParsingLevel parameter having a value of 2 or higher (see §3.2 and §8)

101 When NumOfEvents events have been received, IQFeed is instructed to stop streaming updates, but some update messages may already be on their way from IQFeed before streaming actually stops. These extra update messages are not accumulated in the Buffer, but the latest of these messages will be reflected in LatestData field.

102 During the buffer clear operation some streaming data may be lost, so it is advised not to clear too often.

103 Note that cancelling all active streams cancels streaming regional updates (§6.2) in addition to streaming quotes.

104 IQFeed’s streaming functionality of TST$Y is currently broken. To get notified when DTN reports that the functionality is fixed, follow this forum thread: http://forums.dtn.com/index.cfm?page=topic&topicID=4286

105 In IQML, the Symbol and Symbols parameters are synonymous – you can use either of them, in any capitalization

106 Note that MaxItems is a numeric parameter like NumOfEvents, so don’t enclose the parameter value within string quotes (‘’)

107 Each regional update item uses 2KB of Matlab memory. During trading hours, there could be dozens of updates per second for highly liquid symbols (i.e., 500MB or more per hour, if all updates are saved). Limiting MaxItems to some finite value ensures that the memory usage and performance impact remain low.

108 The textual Description fields depend on the MsgParsingLevel parameter having a value of 2 or higher (see §3.2 and §8)

109 When NumOfEvents events have been received, IQFeed is instructed to stop streaming updates, but one or more update messages may already be on their way from IQFeed before streaming actually stops. These extra update messages are not accumulated in the Buffer, but the latest of these messages will be reflected in LatestData field.

110 During and around the time of the buffer clear, some streaming data may be lost, so it is advised not to clear to often…

111 Note that cancelling all active streams cancels streaming quotes (§6.1) in addition to streaming regional updates.

112 In IQML, the Symbol and Symbols parameters are synonymous – you can use either of them, in any capitalization

113 MaxItems is a numeric parameter like NumOfEvents, so don’t enclose the parameter value within string quotes (‘’)

114 The number of reported bars may possibly be limited by the IQFeed server, depending on your data subscriptions and exchange.

115 Interval bars use ~2KB of Matlab memory. So, if MaxItems=1000, each streamed symbol would need 1000*2KB = 2MB of Matlab memory when its buffer becomes full (which could take a while). Streaming 100 symbols will require up to ~200MB.

116 See §4.3 for an explanation of the fields in this data struct

117 When NumOfEvents events have been received, IQFeed is instructed to stop streaming updates, but some update messages may already be on their way from IQFeed before streaming actually stops. These extra update messages are not accumulated in the Buffer, but the latest of these messages will be reflected in LatestData field.

118 Note that IQFeed’s limitations on live 'secs' interval bars are stricter than the limitations on historical interval bars (§5.4): http://forums.dtn.com/index.cfm?page=topic&topicID=5529

119 http://forums.iqfeed.net/index.cfm?page=topic&topicID=4341. IQFeed’s term “update interval” was renamed MaxUpdateDuration in IQML in order to avoid confusion with the interval bars. Note: the described scenario depends on IQFeed’s implementation of the “update interval” mechanism – this implementation is not in IQML’s control.

120 During and around the time of the buffer clear, some streaming data may be lost, so it is advised not to clear to often…

121 In IQML, the Symbol and Symbols parameters are synonymous – you can use either of them, in any capitalization

122 In IQML, the Symbol and Symbols parameters are synonymous – you can use either of them, in any capitalization

Time	Events so far	User command	Description
9:50:00	0	IQML('quotes', 'Symbol','IBM', 'MaxItems',100,'NumOfEvents',100)	Streaming data for IBM starts. Up to 100 events to accumulate.
9:50:10	23	data = IQML('quotes', 'symbol','IBM', 'NumOfEvents',-1)	Return the 23 accumulated quotes; background streaming continues.
9:50:20	42	data = IQML('quotes', 'symbol','IBM', 'NumOfEvents',-1)	Return the 42 accumulated quotes; background streaming continues.
9:50:30	57	IQML('quotes', 'Symbol','IBM', 'MaxItems',80, 'NumOfEvents',80)	Reduce max # of events 100à80. Only 57 events accumulated until now, so streaming continues.
9:50:40	65	data = IQML('quotes', 'symbol','IBM', 'NumOfEvents',-1)	Return the 65 accumulated quotes; background streaming continues.
9:50:50	72	IQML('quotes', 'Symbol','IBM', 'NumOfEvents',0)	Reduce max # of events 80à0. 72 events accumulated until now, so streaming stops immediately.