10 Attaching user callbacks to IQFeed messages

10.1 Processing IQFeed messages in IQML

IQFeed uses an asynchronous event-based mechanism for sending information to clients. This means that we do not simply send a request to IQFeed and wait for the answer. Instead, we send a request, and when IQFeed is ready it will send us one or more (or zero) messages in response. Each of these messages evoke an event that carry data (the message content and the originating IQFeed channel/port-name). By analyzing the event data we (hopefully) receive the answer that we were waiting for.

Matlab has built-in support for asynchronous events, called callbacks in Matlab jargon.168 Whereas Matlab callbacks are normally used in conjunction with Graphical User Interfaces (GUI), they are also used with IQML, which automatically converts all the IQFeed events into a Matlab callback invocation.

The callback that processes incoming IQFeed messages is constantly being “fired” (i.e., invoked) by asynchronous messages from IQFeed, ranging from client stats and time messages (once per second, for each of IQFeed’s 3 channels/ports), to system messages (e.g. connection losses and reconnections), to error messages and responses to market queries. Some of the events are triggered by user actions (market or portfolio queries, for example), while others are triggered by IQFeed (e.g., client stats once a second).

In addition to the regular IQML callback that processes all incoming IQFeed message events, you can assign your own custom Matlab function that will be triggered whenever a certain IQFeed message arrives. In all cases, the parameter controlling this callback in IQML is called ProcessFunc.

There are two types of callbacks that you can use in IQML:

Generic callback – this is a catch-all callback function that is triggered upon any IQFeed message event. Within this callback, you would need to write some code to distinguish between the different event types in order to process the events’ data. A skeleton for this is given below.

Specific callback – this is a callback function that is only triggered when the specific event type is received from IQFeed. Since the event type is known, you can process its event data more easily than in the generic callback case. You can specify a different specific callback for each of the event types that you wish to process, as well as a default callback that will be used for any other event that was not assigned a specific callback.

When you specify any callback function to IQML, the command/action does not need to be related to the callback. For example:

data = IQML('time', 'ProcessFunc',@IQML_Callback);

where IQML_Callback() is a Matlab function created by you that accepts two input arguments, which are automatically populated in run-time:

iqObject – this is currently an empty array. Future versions of IQML may place an actual object in this argument.

eventData – a Matlab struct that contains the event’s data in separate fields. This struct includes the following fields:

Timestamp – local time in Matlab numeric datenum format.

MessagePort – the name of the IQFeed port that sent the message: 'Level1', 'Level2', 'Lookup' or 'Admin'.

MessageType – the event type, which corresponds to the custom fields that can be set in the ProcessFunc parameter for specific callbacks.

MessageHeader – the first part of the message text string, that identified the message type. This is typically used to set the MessageType field.

MessageString – the message text string as received from IQFeed.

MessageParts – processed parts of MessageString, as a cell-array.

An example of defining a Matlab callback function is:

function IQML_Callback(iqObject, eventData)

% do callback processing here using the info in eventData
end

You can pass external data to your callback functions using the callback cell-array format.169 For example, to pass two extra data values:

callbackDefinition = {@IQML_Callback, 123, 'abc'};
IQML('time', 'ProcessFunc',callbackDefinition);

function IQML_Callback(iqObject,eventData,extra1,extra2)

% do callback processing here using the info in eventData, extra1, extra2
end

Here are examples of the eventData for two different IQFeed messages – a timestamp message (sent from IQFeed once every second on the Level1 and Level2 ports), and a connection stats message (sent from IQFeed once a second on the Admin port):

Timestamp: 737128.675475417
MessagePort: 'Level1'
MessageType: 'Time'
MessageHeader: 'T'
MessageString: 'T,20180309 09:12:39'
MessageParts: {'T' '20180309 09:12:39'}

Timestamp: 737128.675479248
MessagePort: 'Admin'
MessageType: 'System'
MessageHeader: 'S'
MessageString: 'S,STATS,66.112.148.225,60002,1300,0,4,0,0,0,Mar 09
3:10PM,Mar 09 09:12AM,Connected,5.2.7.0,464720-
1,86.43,0.04,0.02,759.37,0.20,0.20'
MessageParts: {1×20 cell}

All IQFeed messages typically begin with a single character followed by ‘,’, which we call the MessageHeader, that identifies the MessageType. In the examples above, the MessageHeader of the first message is 'T' (indicating a Time message), and in the second message it is 'S' (indicating a System message).170

All the callbacks examples so far have set a generic callback that is used for all incoming IQFeed messages. As noted above, you can also set specific callbacks for specific messages. For example:

% Alternative #1: using the struct() function:
>> callbacks = struct('Time','disp TIME!', ...
'System',@(h,e)disp(e.MessageString));

% Alternative #2: using direct field assignments:
>> callbacks.Time = 'disp TIME!';
>> callbacks.System = @(h,e)disp(e.MessageString);

>> IQML('time','processFunc',callbacks);

TIME!
TIME!
S,STATS,66.112.156.228,60002,1300,0,4,0,0,1,Mar 11 12:36PM,Mar 11 07:14AM,Connected,5.2.7.0,464720-1,51.51,0.04,0.02,516.30,0.23,0.23
TIME!
TIME!
S,STATS,66.112.156.228,60002,1300,0,4,0,0,1,Mar 11 12:36PM,Mar 11 07:14AM,Connected,5.2.7.0,464720-1,51.54,0.04,0.02,516.48,0.23,0.23
TIME!

In this example, we have set two separate custom callbacks for two different IQFeed messages: the periodic timestamp messages and the periodic system update messages.

In addition to specific callbacks for specific message types, you can also set a “Default” callback that will be invoked for each incoming IQFeed message that does not have a specific callback assigned to it.

The following message types can be trapped, corresponding to the eventData’s MessageType field (e.MessageType):

You can set individual callbacks to any of these MessageType values, by using the MessageType value as a field-name in the ProcessFunc parameter. For example, to process quote-update (tick) messages in a dedicated callback function:

>> callbacks.Quote_update = @IQML_Quote_Update_Callback;

>> IQML('time','ProcessFunc',callbacks);

Here is a more elaborate example, were we set different callbacks for different message types, along with a default callback for all other message types:

% Alternative #1: using the struct() function:
>> callbacks = struct('Time','disp TIME!', ...
'System',[], ... % ignore System messages
'Quote_update',@IQML_Quote_Update_Callback, ...
'Default',@IQML_Default_Callback);

% Alternative #2: using direct field assignments:
>> callbacks.Time = 'disp TIME!';
>> callbacks.System = []; % ignore System messages
>> callbacks.Quote_update = @IQML_Quote_Update_Callback;
>> callbacks.Default = @IQML_Default_Callback);

>> IQML('time','processFunc',callbacks);

When you specify any callback function to IQML, you only need to set it once, in any IQML command. Unlike most IQML parameters, which are not remembered across IQML commands and need to be re-specified, callbacks do not need to be re-specified. They are remembered from the moment they are first set, until such time as Matlab exits or the callback parameter is changed.

Note that it is not an error to re-specify the callbacks in each IQML command, it is simply useless and makes the code less readable.

If an error occurs during the evaluation of your specified callback function, an error message will be displayed in the Matlab console. In blocking mode (data=IQML(…)) a Matlab exception will then be thrown, which can be trapped in the calling code using a try-catch block (see §3.4 item 1); in non-blocking mode no exception will be thrown:

>> IQML('quotes','symbol','IBM','processFunc',struct('Quote_Summary',@myFunc))
20200330 11:02:55.510 error in user-defined callback (myFunc) for IQFeed Quote_Summary message: Invalid argument in myFunc line 123

To reset all callbacks (i.e., remove any callback invocation), simply set the ProcessFunc parameter value to [] (empty square brackets):

IQML('time', 'ProcessFunc',[]);

You can also set individual message callbacks to an empty value, in order to ignore just these specific messages but not the other messages:

>> callbacks.Time = 'disp TIME!';
>> callbacks.System = []; % ignore System messages
>> callbacks.Default = @IQML_Default_Callback);

>> IQML('time','ProcessFunc',callbacks);

Matlab callbacks are invoked even if you issue a custom IQFeed command (see §9.4). This can be very useful: you can send a request to IQFeed, and then process the results in a Matlab callback function. However, note that in such a case, it is possible that the returned message will contain a MessageHeader that will not be a recognized. Such messages will be assigned a MessageType of 'Other'.

10.2 Run-time performance implications

It is very important to ensure that any callback function that you assign in IQML completes in the fastest possible time. This is important for programming in general, but it is especially important for IQML callbacks, which are invoked (executed) every time that a new message arrives from IQFeed, numerous times each second.

As explained in §3.6, IQML’s standard callback processing has an overhead of 1-2 milliseconds per IQFeed message. This means that without any user-specified callbacks, and without any other Matlab or other code running, IQML can typically process up to 500-1000 IQFeed messages per second.

When you add your own user-defined callbacks, their processing time is added to IQML’s. For example, if your callback takes an average of just 3 msecs to process (which is quite fast), the total average message processing time will be 4-5 msecs. This will lower IQML’s effective processing rate from 500-1000 messages/sec to just 200-250 messages/sec. The more callbacks and alerts that you define, and the longer each of them takes to process, the lower IQML’s message processing rate will be.

The following specific tips may assist you to reduce the callback performance impact:

Ensure that you have enough physical memory to avoid memory swapping to disk. This is probably the single most important tip to improve performance

Avoid setting user callbacks and alerts, or at least disable them when not needed.

Avoid setting a Default callback or a general ProcessFunc, but rather specific callbacks only for the messages that you need (e.g. for News or Regional).

Limit the streaming data to just those events and symbols that are of interest to you. For example, if you are only interested in the GOOG symbol, and set a Quote_update callback, this callback will also be processed for streaming quotes for other symbols, so it’s better to stop streaming those other symbols.

Minimize disk access: disk I/O is much slower than memory access. Save data to memory and flush it to disk at the end of the trading day, or once in a while (e.g. every 5 minutes), but not in each callback.

If you need to access disk files, use SSD (solid-state disk) rather than a spinning hard-disk.

If you need to load data from disk, do it once and preserve the data in memory using Matlab persistent or global variables, to be reused in callback calls.

Instead of re-computing values that are based on static data in each callback call, compute once and cache results in Matlab persistent or global variables.

Use Matlab’s built-in Profiler tool171 to check your callback code for run-time performance hotspots that can be optimized to run faster.

Read the textbook “Accelerating MATLAB Performance”,172 authored by IQML’s creator (see §15.2), for numerous tips on improving Matlab run-time.

10.3 Usage example – using callbacks to parse options/futures chains

In this example, we request IQFeed to send the list of symbols in an options/futures chain, then parse the incoming results to retrieve the symbols in the chain (see §8.2).

We first send the relevant command to IQFeed using IQML’s custom command functionality (§9.4), specifying a custom callback function for the 'Chain' MessageType:173

% Equity options chain for GOOG:
processFunc.Chain = @IQML_Chain_Callback;
>> IQML('command', 'String','CEO,GOOG,p,,1', 'PortName','lookup', ...
'debug',1, 'ProcessFunc',processFunc)

=> 20180405 13:13:00.063 (lookup) CEO,GOOG,p,,1

<= 20180405 13:13:00.574 (lookup) :,GOOG1806P1000,GOOG1806P1002.5,GOOG1806P1
005,GOOG1806P1007.5,GOOG1806P1010,GOOG1806P1012.5,GOOG1806P1015,GOOG1806P1017.5,GOOG1806P1020,GOOG1806P1022.5,GOOG1806P1025,GOOG1806P1027.5,GOOG1806P1030,GOOG1806P1032.5,GOOG1806P1035,GOOG1806P1037.5,GOOG1806P1040,GOOG1806P1042.5,GOOG1806P1045,GOOG1806P1047.5,GOOG1806P1050,…,

<= 20180405 13:13:00.578 (lookup) !ENDMSG!

% Future options chain for C:
>> IQML('command', 'String','CFO,C,p,,9,1', 'PortName','lookup', ...
'debug',1, 'ProcessFunc',processFunc)

=> 20180405 13:31:48.677 (lookup) CFO,C,p,,9,1

<= 20180405 13:31:49.149 (lookup) :,CH19P2000,CH19P2100,CH19P2200,CH19P2300,CH19
P2400,CH19P2500,CH19P2600,CH19P2700,CH19P2800,CH19P2900,CH19P3000,CH19P3100,CH19P3200,CH19P3300,CH19P3400,CH19P3500,CH19P3600,CH19P3700,CH19P3800,CH19P3900,CH19P4000,CH19P4100,CH19P4200,CH19P4300,CH19P4400,CH19P4500,CH19P4600,CH19P4700,CH19P4800,CH19P4900,CH19P5000,CH19P5100,CH19P5200,CH19P5300,CH19P5400,CH19P5500,CH19P5600,CH19P5700,CH19P5800,CH19P5900,CH19P6000,CH19P6100,CH19P6200,CH19P6300,CH19P6400

<= 20180405 13:31:49.158 (lookup) !ENDMSG!

The custom callback function may look something like this:

function IQML_Chain_Callback(iqObject,eventData)

symbols = eventData.MessageParts(2:end); %discard the ':' message header

% now do something useful with the reported symbols...
end

10.4 Usage example – using callbacks for realtime quotes GUI updates

In this example, we wish to update a real-time ticker window with the latest streaming quotes data. The idea is to create a minimalistic window and update its title with the symbol name and latest trade price, whenever a new tick arrives.

The code relies on the format of IQFeed’s Quote_update (Q) message, which by default is a 16-element cell array: {Symbol, Most_Recent_Trade, Most_Recent_Trade_Size, Most_Recent_Trade_Time, Most_Recent_Trade_Market_Center, Total_Volume, Bid, Bid_Size, Ask, Ask_Size, Open, High, Low, Close, Message_Contents, Most_Recent_Trade_Conditions}:

>> processFunc = struct('Quote_Update', @Quote_Update_Callback);
>> IQML('quotes', 'symbol','@VX#', 'numofevents',100, ...
'ProcessFunc',processFunc, 'debug',1)

=> 20180411 12:03:40.131 (Level1) w@VX#
<= 20180411 12:03:40.391 (Level1) F,@VX#,20,,,28.05,12.85,,,,,,,,,,,,,,,,,,CBOE …
<= 20180411 12:03:40.409 (Level1) P,@VX#,20.61,,04:52:29.711000,32,5668,20.60,50,
20.65,87,20.20,20.70,20.15,20.18,Cbasohlcv,4D
<= 20180411 12:03:44.887 (Level1) Q,@VX#,20.61,,04:52:29.711000,32,5668,20.60,50,
20.65,86,20.20,20.70,20.15,20.18,a,4D

In our case, we are only interested in the 1st (Symbol) and 2nd (Most_Recent_Trade) elements of the 'Q' update messages:

eventData =
Timestamp: 737161.502602859
MessagePort: 'Level1'
MessageType: 'Quote_Buffer'
MessageHeader: 'Q'
MessageString: 'Q,@VX#,20.61,,04:52:29.711000,32,5668,20.60,50,20.65,86,
20.20,20.70,20.15,20.18,a,4D'
MessageParts: {'@VX#' 20.61 [] '04:52:29.711000' 32 5668 20.6 50
20.65 86 20.2 20.7 20.15 20.18 'a' '4D'}

The corresponding callback function will be:

function Quote_Update_Callback(iqObject, eventData)

% Symbol is 1st data element of IQFeed 'Q' messages

symbol = eventData.MessageParts{1};

% Last trade price is 2nd data element of the IQFeed 'Q' messages
latestTrade = eventData.MessageParts{2};

% Get the handle for this symbol's ticker window
hFig = findall(0, 'Tag',symbol, '-depth',1);
if isempty(hFig)
% Ticker window not found, so create it
hFig = figure('Tag',symbol, 'Position',[300,300,250,1], ...
'Resize','off', 'NumberTitle','off', ...
'Menu','none', 'Toolbar','none',);
end

% Update the ticker window's title
hFig.Name = sprintf('%s: %.2f', symbol, latestTrade);

end

And the resulting ticker window will look like this:

As noted in §6.1, tick events may be sent at a very high rate from the IQFeed server. So instead of updating the GUI with each tick, you may want to use a periodic Matlab timer having a Period of 0.5 [secs], that will invoke a timer callback, which will call IQML(…,'NumOfEvents',-1) to fetch the latest data and update the GUI.

10.5 Usage example – using callbacks for realtime order-book GUI updates

In this example, we wish to update a real-time GUI display of the order-book (at least the top few rows of the book).

Note: Market Depth (Level 2) data is only available in the Professional IQML license.

As noted in §6.4, market-depth events may be sent at a very high rate from the IQFeed server, and so it is not feasible or useful to update the Matlab GUI for each update. Instead, we update the GUI with the latest data at a steady rate of 2 Hz (twice a second). This can be achieved in two different ways: one alternative is to set-up a periodic timer that will run our GUI-update callback every 0.5 secs, which will call IQML(…,'NumOfEvents',-1) to fetch the latest data and update the GUI.

Another alternative, shown here below (also downloadable174), is to attach a user callback function to Level 2 market-depth messages, updating an internal data struct, but only updating the GUI if 0.5 secs or more have passed since the last GUI update:

% IQML_MktDepth - sample Market-Depth usage function
function IQML_MktDepth(symbol)

% Initialize data
numRows = 10;
depthData = cell(numRows,8);
lastUpdateTime = -1;
GUI_refresh_period = 0.5 * 1/24/60/60; % =0.5 secs

% Prepare the GUI
hFig = figure('Name','IQML market-depth example', ...
'NumberTitle','off','CloseReq',@figClosedCallback,...
'Menubar','none', 'Toolbar','none', ...
'Resize','off', 'Pos',[100,200,660,260]);
color = get(hFig,'Color');
headers = {'Ask valid','Ask time','Ask size','Ask price', ...
'Bid price','Bid size','Bid time','Bid valid'};
formats = {'logical','char','numeric','long', ...
'long','numeric','char','logical'};
hTable = uitable('Parent',hFig, 'Pos',[10,40,635,203], ...
'Data',depthData, 'ColumnName',headers, ...
'ColumnFormat',formats, ...
'ColumnWidth',{60,100,80,80,80,80,100,60});
hButton = uicontrol('Parent',hFig, 'Pos',[50,10,60,20], ...
'String','Start', 'Callback',@buttonCallback);
hLabel1 = uicontrol('Parent',hFig, 'Pos',[120,10,100,17], ...
'Style','text', 'String','Last updated:', ...
'Horizontal','right', 'Background',color);
hLabelTime = uicontrol('Parent',hFig, 'Pos',[225,10,100,17], ...
'Style','text', 'String','(not yet)', ...
'Horizontal','left', 'Background',color);

% Send the market-depth request to IQFeed using IQML
contractParams = {'symbol',symbol}; % symbol='@ES#'
callbacks = struct('Market_depth',@mktDepthCallbackFcn);
IQML('marketdepth', contractParams{:}, 'processFunc',callbacks);

% Figure close callback function - stop market-depth streaming
function figClosedCallback(hFig, ~)
% Delete figure window and stop any pending data streaming
delete(hFig);
IQML('marketdepth', contractParams{:}, 'numofevents',0);
end % figClosedCallback

% Start/stop button callback function
function buttonCallback(hButton, ~)
currentString = get(hButton,'String');
if strcmp(currentString,'Start')
set(hButton,'String','Stop');
else
set(hButton,'String','Start');
end
end % buttonCallback

% Callback functions to handle IQFeed Market Depth update events
function mktDepthCallbackFcn(~, eventData)

% Ensure that it's the correct MktDepth event
allMsgParts = strsplit(eventData.MessageString,',');
allMsgParts(strcmpi(allMsgParts,'T')) = {true};
allMsgParts(strcmpi(allMsgParts,'F')) = {false};

if strcmp(eventData.MessagePort,'Level2') && ...
strcmp(allMsgParts{2},symbol)

% These are the field names of the IQFeed messages
inputParams = {'Intro','Symbol','MMID',...
'Bid','Ask','BidSize','AskSize',...
'BidTime','Date','ConditionCode',...
'AskTime','BidInfoValid',...
'AskInfoValid','EndOfMsgGroup'};

% Get the updated data row
% Note: Java indices start at 0, Matlab starts at 1
mmid = allMsgParts{strcmpi(inputParams,'MMID')};
row = sscanf(mmid,'%*c%*c%d');

% Get the size & price data fields from the event's data
bidValid = allMsgParts{strcmpi(inputParams,'BidInfoValid')};
askValid = allMsgParts{strcmpi(inputParams,'AskInfoValid')};
bidTime = allMsgParts{strcmpi(inputParams,'BidTime')};
askTime = allMsgParts{strcmpi(inputParams,'AskTime')};
bidSize = allMsgParts{strcmpi(inputParams,'BidSize')};
askSize = allMsgParts{strcmpi(inputParams,'AskSize')};
bidPrice = allMsgParts{strcmpi(inputParams,'Bid')};
askPrice = allMsgParts{strcmpi(inputParams,'Ask')};
thisRowsData = {askValid,askTime,askSize,askPrice,...
bidPrice,bidSize,bidTime,bidValid};
depthData(row,:) = thisRowsData;

% Update the GUI if more than 0.5 secs have passed and
% the <Stop> button was not pressed
if ~isvalid(hButton), return, end
isStopped = strcmp(get(hButton,'String'),'Start');
if now - lastUpdateTime > GUI_refresh_period && ~isStopped
set(hTable,'Data',depthData);
set(hLabelTime,'String',datestr(now,'HH:MM:SS'));
lastUpdateTime = now;
end
end

end % mktDepthCallbackFcn
end % IQML_MktDepth