3 Using IQML
3.1 General usage
IQML uses the IQFeed client7 to connect to the IQFeed server. If an active IQFeed client is not detected, IQML will automatically attempt to start the IQFeed client and to connect to it. Note that this may not work for some IQFeed client installations. You can always start the IQFeed client manually, before running IQML. In any case, if an IQFeed connection is unsuccessful, IQML will error.
IQML’s Matlab wrapper function is called IQML, contained within the IQML.p file. Its accompanying IQML.m file provides basic usage documentation using standard Matlab syntax, e.g.:
>> help('IQML')
>> help IQML % equivalent alternative
>> doc IQML
The IQML function accepts a variable number of input parameters, and returns data in a single output argument, with an optional errorMsg output. The general syntax is:
>> [data, errorMsg] = IQML(action, parameters);
where:
data is the output value. If this output value is requested, then Matlab processing will block data until the result is available; if the output data is not requested then the Matlab processing will proceed immediately (non-blocking) – the IQFeed data will stream asynchronously (see below).
errorMsg is the latest error message that was reported (if any); see §3.5.
action is a string that denotes the requested query type (mandatory input).
parameters can be specified, depending on the requested action. There are several ways to specify parameters, which are described below.
For example:
>> data = IQML('time'); %'time' action (blocking), 0 parameters
>> IQML('quotes', 'Symbol','IBM'); %streaming 'quotes' action, 1 parameter
>> IQML('command', 'String',command, 'PortName','Admin'); %2 parameters
Note that when an output data is requested, IQML treats the request as blocking (synchronous), meaning that Matlab processing will wait for IQFeed’s data (or a timeout) before proceeding with the next Matlab command. For example:
>> t = IQML('time'); % blocking until data is available
When an output data is not requested, IQML treats the request as streaming (non-blocking, a-synchronous) and Matlab processing will proceed immediately. This non-blocking mode is typically useful for sending IQFeed requests (for example, to start streaming trades/ticks), without waiting for a response from IQFeed. The streamed data is accumulated by IQML in the background, and can later be retrieved using the mechanism that is discussed in §6. Examples of such non-blocking commands:
>> IQML('quotes', 'Symbol','IBM'); %start non-blocking IBM quotes stream
>> IQML('command', 'String',command); %asynchronous/non-blocking command
Here are the action values recognized by IQML, in the Standard and Professional licenses; trial licenses have the full functionality of the Professional license:
Action | Description | User Guide Section(s) | Standard | Pro & trial |
'version' | Display product version information | Yes | Yes | |
'license' | Display the license fingerprint & activation key | Yes | Yes | |
'validate' | Check whether IQML is activated on this computer | Yes | Yes | |
'update' | Update the IQML installation to the latest version | Yes | Yes | |
'revert' | Update the IQML installation to a previous version | Yes | Yes | |
'doc' | Display this User Guide in a separate window | - | Yes | Yes |
'quotes' | Fetch quotes/trades information on a ticker | Yes | Yes | |
'fundamental' | Fetch fundamental information on a ticker | Yes | Yes | |
'intervalbars' | Fetch custom streaming interval bars on a ticker | Yes | Yes | |
'marketdepth' | Fetch level 2 market depth information on a ticker | - | Yes | |
'greeks' | Report option Greeks, fair value, implied volatility | - | Yes | |
'history' | Fetch historical or intra-day data bars from IQFeed | Yes | Yes | |
'summary' | Fetch historical market summary data from IQFeed | - | Yes | |
'regional' | Fetch regional update information on a ticker | - | Yes | |
'news' | Fetch news headlines or stories from IQFeed | - | Yes | |
'lookup' | Fetch list of symbols/codes matching a set of criteria | Yes | Yes | |
'chain' | Fetch futures/options chain matching a set of criteria | - | Yes | |
'disconnect' | Disconnect IQML from IQFeed | Yes | Yes | |
'reconnect' | Disconnect and then re-connect IQML to IQFeed | Yes | Yes | |
'time' | Retrieve the latest IQFeed server & message times | Yes | Yes | |
'stats' | Retrieve connection and network traffic statistics | Yes | Yes | |
'command' | Send a custom command to IQFeed | Yes | Yes | |
'registry' | Open Windows Registry Editor at IQFeed’s settings | Yes | Yes | |
'alert' | Alert the users upon IQFeed streaming events | - | Yes | |
'log' | Control IQFeed logging of messages and events | Yes | Yes |
IQML accepts input parameters in several alternative formats, which are equivalent – you can use whichever format that you prefer:
As name-value pairs – for example:
>> IQML('command', 'String',command, 'PortName','Admin'); %2 parameters
As a Matlab struct, with parameters contained in corresponding struct fields e.g.:
>> params = []; % initialize
>> params.String = command;
>> params.PortName = 'Admin';
>> IQML('command', params);
As a Matlab class, with parameters contained in corresponding class properties.
As a Matlab table, with parameters contained in corresponding table variables.
As field-separated rows in an Excel input file – for example:
>> IQML('command', 'C:\MyData\inputFile.xlsx');
where:
Each column contains a separate parameter
Row #1 contains the parameter names, and rows 2+ contain their corresponding values, one row per command
All commands use the same action ('command' in this example)
Each parameter must have an associated value. The value’s data type depends on the specific parameter: it could be numeric, a string, a function handle etc. The definition of all the parameters and their expected data types is listed in the appropriate section in this User Guide that explains the usage for the associated action.
Note: if you specify parameters using a struct/class/table format, and then reuse this object for different IQML commands (by altering a few parameters), the entire set of parameters will be used, possibly including some leftover parameters from previous IQML commands. This may lead to unexpected results. For example:
% 1st IQML command – stop streaming timestamp messages every 1 second
>> params = []; % initialize
>> params.String = 'S,TIMESTAMPSOFF';
>> params.PortName = 'Level1';
>> IQML('command', params);
% 2nd IQML command – stop streaming client stats messages every 1 sec
>> params.String = 'S,CLIENTSTATS OFF'; %reuse existing params struct
>> IQML('command', params);
% 3rd IQML command – start streaming quotes messages for IBM
>> params.Symbol = 'IBM'; %reuse existing params struct
>> IQML('quotes', params);
In this example, the 2nd IQML command above will have no effect, because the PortName parameter in the params struct from the 1st IQML command will be reused in the 2nd command, sending it to the Level1 port, instead of to the Admin port. Similarly, the 3rd IQML command will result in a warning, because the 'quotes' action does not expect the String and PortName parameters that were carried over (reused) from the 2nd command. To avoid such unexpected results, it is therefore best to reset the object (params=[] for a struct) before preparing each IQML command.
IQML is quite tolerant of user input: parameter names (but generally not their values) are case-insensitive, parameter order does not matter, non-numeric parameter values can be specified as either char arrays ('abc') or strings ("abc"), and some of these can be shortened. For example, the following commands are all equivalent:
>> IQML('quotes', 'Symbol','IBM');
>> IQML('quotes', 'symbol','IBM');
>> IQML('Quotes', "Symbol","IBM");
>> IQML('Quotes', 'Symbol','IBM');
>> IQML('QUOTES', 'symbol',"IBM");
The full list of acceptable input parameters, and their expected values, is listed in the following sections, grouped by usage. If you specify an unexpected parameter, it will be ignored and a warning message will be displayed in the Matlab Command Window:
>> IQML('quotes', 'badName',1)
Warning: 'badName' is not a valid parameter for the 'quotes' action in IQML
When using IQML, there is no need to worry about connecting or disconnecting from the IQFeed client – IQML handles these activities automatically, without requiring user intervention. Users only need to ensure that the IQFeed client is active and logged-in when the IQML command is invoked in Matlab.
IQML reads data using the IQFeed account to which the IQFeed client is connected. In other words, the IQFeed account type is transparent to IQML: the only way to control which IQFeed data is available to IQML is to login to the IQFeed client using the appropriate username/password. Refer to §9.1 for additional details.
3.2 Common request properties
The following properties can be specified in IQML, with most actions:
Parameter | Data type | Default | Description |
Symbol / Symbols 8 | string | (none) | The asset symbol, as known by IQFeed.9 |
Timeout | number | 5.0 | Max number of seconds (0-9000) to wait for data in a blocking request (0 means infinite). |
Debug | logical | false or 0 | If true or 1, additional information is displayed. |
MsgParsingLevel | number | 2 | One of: 2 – parse all the data in incoming IQFeed messages (default; most verbose, slowest) 1 – do not parse lookup codes (e.g. trade condition, price formats, market id). 0 – do not parse lookup code; also do not convert string data into numeric values (i.e. all data fields will remain strings: ‘3.14’). This is the fastest but least verbose option. |
RaiseErrorMsgs | logical | true or 1 10 | If true or 1, IQFeed error messages raise a Matlab error in blocking (non-streaming) mode (see §12) |
ProcessFunc | function handle | [] | Custom user callback function to process incoming IQFeed data messages (see §10). |
NumOfEvents | integer | inf | The maximal number of messages to process. |
OutputFormat | string | 'struct' | 'cells', 'struct', 'table' or 'timetable' (see §3.5). |
Additional properties are request-specific and are listed below in the relevant sections. For example, the 'history' action has additional properties that control the parameters of the historic data request (start/end date, data type, etc.).
3.3 Blocking & non-blocking modes
Whenever you specify an output parameter in a call to IQML, the program will block until a response is available (i.e., a synchronous request). If no output parameter is specified, IQML immediately returns (non-blocking, a-synchronous) and additional Matlab commands can immediately be issued. This non-blocking mode is typically useful for sending IQFeed requests to start streaming data (for example, streaming trades/ticks or news headlines), without waiting for any response from IQFeed. The streamed data is accumulated by IQML in the background, and can later be retrieved using the mechanism that is discussed in §6. For example:
>> t = IQML('time'); % blocking until data is available
>> IQML('quotes', 'Symbol','IBM'); %start non-blocking IBM quotes stream
>> IQML('command', 'String',command); %asynchronous/non-blocking command
3.4 Common causes of confusion
A common cause of error is specifying symbols incorrectly. IQFeed is very sensitive about this: if the specified symbol is invalid,11 or if your account does not have the corresponding market subscription, IQFeed will report an error:
>> IQML('quotes', 'Symbol','xyz123')
Symbol 'XYZ123' was not found!
If the request is blocking, an error (exception) will be thrown (raised), which can be trapped and handled by the user, using a Matlab try-catch construct:
try
data = IQML('fundamental', 'Symbol','xyz123');
catch err
fprintf(2, 'Error: %s\n', err.message);
% do some intelligent processing here...
end
IQFeed’s website includes a detailed symbol-lookup search engine.12 If you are still unsure about a symbol name, please contact IQFeed’s customer support.
If any request parameter is invalid or if the request is not accepted by IQFeed, a run-time error will result, which can be trapped as shown above. For example:
IQML historic data query (EURGBP.FXCM) error: Unauthorized user ID (your IQFeed account is not authorized for this data)
A common confusion source is specifying numeric values as strings or vice versa. For example, IQML(...,'Timeout','10') rather than IQML(...,'Timeout',10). Each IQML parameter expects a value of a specific data type, as listed in the parameter tables in this user guide. IQML is sometimes smart enough to automatically convert to the correct data type, but you should not rely on this: it is better to always use the correct data type. Otherwise, Matlab might get confused when trying to interpret the string '10' as a number, and odd results might happen.
While most of IQML’s functionality is available in all license types, some actions/functionality are only available in the Professional IQML license:13
Parallelized queries (§3.6)
Customizable data fields in quotes data (§4.1, §6.1)
Level 2 market depth quotes (§4.4, §6.4, §10.5)
Option Greeks, Fair Value and Implied Volatility (§4.5)
Regional updates (§6.2)
News (§7)
Options/futures chain lookup (§8.2)
Alerts (§11)
If you have a Standard license and try to access Professional-only functionality, a run-time error will result:
>> data = IQML('news');
The 'news' action is not available in your Standard license of IQML, only in the Professional license. Contact info@undocumentedmatlab.net to upgrade your license.
IQFeed reports dates in different formats, depending on the specific query: either in the standard American mm/dd/yyyy format (for example: '01/29/2018'), or in yyyymmdd format (for example: '2018-01-29' or '20180129 12:29:48'). Dates are usually reported as strings. In some cases, a corresponding Matlab datenum value is also reported, for example (§5.5, §6.1):
Symbol: 'IBM'
Timestamp: '2018-03-07 13:23:02.036440'
Datenum: 737126.557662458
...
Symbol: '@VX#'
LatestEventDatenum: 737128.637260451
LatestEventTimestamp: '20180309 15:17:39'
...
Depending on the data field, the timestamp is either your local computer’s time, or IQFeed servers (New York) time – not the exchange time. To get the exchange time, you would need to do the appropriate time-zone arithmetic.
By default, Matlab displays data in the console (“Command Window”) using “short” format, which displays numbers rounded to 4 digits after the decimal. The data actually has higher precision, so when you use it in a calculation the full precision is used, but this is simply not displayed in the console.
IQML does not truncate/round/modify the IQFeed data in any manner!
To display the full numeric precision in the Matlab console, change your Command Window’s Numeric Format from “short” to “long” (or “long g”) in Matlab’s Preferences window, or use the “format long” Matlab command:
>> data = IQML('quotes', 'symbol','ONLIB.X'); %overnight LIBOR rate
>> data.Close % short format (only 4 digits after decimal)
ans =
1.4463
>> format long g % long format (full precision displayed)
>> data.Close
ans =
1.44625
In the few seconds following an initial connection to IQFeed, some queries may return empty data. In such cases, you should try to refetch the data after a short pause.
For example, the following code snippet tries to refetch the S&P index history up to 10 times, with a 0.5-second pause between retries, and raises an error only if no data is available even after all these retries:
data = IQML('history','symbol','SPX.XO');
numTries = 10;
while isempty(data) && numTries > 0
% Wait a bit and then retry
pause(0.5)
data = IQML('history','symbol','SPX.XO');
numTries = numTries - 1;
end
if isempty(data), error('data is not available'); end
3.5 Returned data format
Many queries in IQML return their data in the form of a struct-array (a Matlab array of structs), for example (see §8.6):
>> data = IQML('lookup', 'DataType','NAICS')
data =
1175×1 struct array with fields:
id
description
>> data(1)
ans =
id: 111110
description: 'Soybean Farming'
>> data(2)
ans =
id: 111120
description: 'Oilseed (except Soybean) Farming'
For various purposes (readability, maintainability, performance, usability), users may wish to modify this data structure. You can easily convert the data using Matlab’s builtin functions struct2cell() (which converts the struct-array into a cell-array), or struct2table() (which converts the struct-array into a Matlab table object):
>> disp(struct2cell(data)')
[111110] 'Soybean Farming'
[111120] 'Oilseed (except Soybean) Farming'
[111130] 'Dry Pea and Bean Farming'
...
>> disp(struct2table(data))
id description
______ ___________________________________________________
111110 'Soybean Farming'
111120 'Oilseed (except Soybean) Farming'
111130 'Dry Pea and Bean Farming'
...
Note that empty data cannot be converted using struct2table() or struct2cell():
>> data = IQML('lookup', 'DataType','NAICS', 'Description','xyz')
data =
[]
>> struct2cell(data)
Undefined function 'struct2cell' for input arguments of type 'double'.
>> struct2table(data)
Error using struct2table (line 26)
S must be a scalar structure, or a structure array with one column or one row.
When querying multiple symbols, IQML automatically tries to merge the data into a 2D array of structs (NxM: N data points for each of M symbols). If this is impossible (e.g. different number of data points per symbol), the results are reported as a cell array:
>> data = IQML('history', 'Symbol','IBM,AAPL,T')
data =
100×3 struct array with fields:
Symbol
Datestamp
Datenum
...
>> data = IQML('history', 'Symbol','IBM,AAPL,T,@ESH20C120000')
data =
1×4 cell array
{100×1 struct} {100×1 struct} {100×1 struct} {65×1 struct}
Cell-array results may also occur when some query elements report no results at all:
>> data = IQML('quotes','symbol','IBM,AAPL,FB,XXXXXX')
IQFeed error: Symbol 'XXXXXX' is invalid/unavailable!
data =
1×4 cell array
{1×1 struct} {1×1 struct} {1×1 struct} {0×0 double}
When the results are reported in simple struct format (i.e., scalar or 1D struct array, but not a cell array or 2D struct array), users can request results in equivalent Matlab table or timetable formats,14 using the optional OutputFormat parameter:
>> data = IQML('history', 'symbol','IBM', 'outputformat','table')
data =
100×9 table
Symbol Datestamp Datenum High Low Open ...
_______ ______________ _______ ________ ________ ______ ...
{'IBM'} {'2020-08-11'} 738014 130.47 126.61 128.76 ...
{'IBM'} {'2020-08-12'} 738015 127.79 125.876 127.61 ...
{'IBM'} {'2020-08-13'} 738016 126.39 124.77 125.96 ...
...
>> data = IQML('history', 'symbol','IBM', 'outputformat','timetable')
data =
100×9 timetable
Time Symbol Datestamp Datenum High Low ...
___________ _______ ______________ _______ ________ ________ ...
11-Aug-2020 {'IBM'} {'2020-08-11'} 738014 130.47 126.61 ...
12-Aug-2020 {'IBM'} {'2020-08-12'} 738015 127.79 125.876 ...
13-Aug-2020 {'IBM'} {'2020-08-13'} 738016 126.39 124.77 ...
...
Note: when no timestamp field is reported, the results cannot be converted into a Matlab timetable. In such cases, a regular table will be reported, with a warning message:
>> data = IQML('quotes', 'symbol','IBM', 'outputformat','timetable')
Warning: Cannot convert IQFeed query results into a Matlab timetable:
no timestamp field in results, reverting to regular table
(Type "warning off IQML:CannotConvertToTimetable" to suppress this warning.)
data =
1×19 table
...
When the results cannot be converted into a table, requesting an OutputFormat of table or timetable will result in a warning message and the results will be reported unchanged (as an empty array, a cell-array, or 2D struct array):
>> data = IQML('history', 'symbol','IBM,AAPL,FB', 'outputformat','table')
Warning: Cannot convert IQFeed query results into a Matlab table: Input structture must be a scalar structure or a structure array with one column or one row.
(Type "warning off IQML:CannotConvertToTable" to suppress this warning.)
data =
100×3 struct array with fields:
Symbol
Datestamp
Datenum
...
When OutputFormat is set to 'cells', the results are reported as a cell-array, whose elements match the results columns. This typically equates to a single cell element for each queried symbol. If you query N symbols, a 1xN cell-array will be reported, where each cell element will contain the separate results for each of the N symbols in a struct (or struct-array) format. For example:
>> data = IQML('history', 'symbol','IBM', 'outputformat','cells')
data =
1×1 cell array
{100×1 struct}
>> data = IQML('history', 'symbol','IBM,FB', 'outputformat','cells')
data =
1×2 cell array
{100×1 struct} {100×1 struct}
>> data = IQML('history', 'symbol','IBM,FB,xxxxx', 'outputformat','cells')
data =
1×3 cell array
{100×1 struct} {100×1 struct} {1×0 double}
As can be seen from these examples, the data format for each of the symbols (i.e. the contents of each cell element) remains unchanged: struct array or empty array. To break down the internal struct arrays into their constituents, you can use Matlab’s struct2cell() or struct2table() functions, as explained at the top of this section:
>> data = IQML('history', 'symbol','IBM,FB,xxxxx', 'outputformat','cells');
>> cellfun(@struct2table, data, 'uniform',false, 'ErrorHandler',@(s,v)table)
ans =
1×3 cell array
{100×9 table} {100×9 table} {0×0 table}
The following table compares the possible output types of various IQML queries:
Base IQML query | OutputFormat | |||
'cells' | 'struct' (default) | 'table' | 'timetable' | |
IQML('history','symbol','IBM') | 1×1 cell array | 100×1 | 100×9 | 100×9 timetable |
IQML('history','symbol','xxxx') | 1×0 empty cell array | 1×0 empty double row vector | ||
IQML('history','symbol','IBM,xxxx') | 1×2 cell array | |||
IQML('history','symbol','IBM,FB') | 1×2 cell array | 100×2 | ||
IQML('quotes','symbol','IBM') | 1×1 cell array | 1x1 struct | 1×19 table | |
IQML('quotes','symbol','IBM,FB') | 1×2 cell array | 1x2 struct | 2×19 table | |
IQML('news') | 1×1 cell array | 1000×1 | 1000×6 | 1000×6 timetable |
A second, optional, output parameter of IQML returns the latest error message (if any):15
>> [data, errorMsg] = IQML('quotes', 'Symbol','IBM', 'Timeout',0.1)
data =
[]
errorMsg =
'IQML timeout: either IQFeed has no data for this query, or the Timeout parameter should be set to a value larger than 0.1'
3.6 Run-time performance
3.6.1 General considerations
IQML’s standard processing has an overhead of 1-2 milliseconds per IQFeed message, depending on several factors:
Message type/complexity – simple messages such as the periodic timestamp updates are simpler to process than complex messages (e.g. fundamental data).
The Debug parameter (see §3.2) – A value of 1/true is ~1 msec slower per message, compared to the default value of 0/false (depending on message type).
The MsgParsingLevel parameter (§3.2) – A value of 0 is ~1 msec faster per message, compared to the default value of 2 (depending on message type).
The UseParallel parameter (see below) enables query parallelization (faster).
The Fields parameter (§4.1, §5.1-§5.5, §6.1) – fewer fields are faster.
User-defined callbacks (§10) add their own processing time per message. See §10.2 for suggested ways to speed-up this callback processing overhead.
Each active alert (§11) uses 1-2 msecs per message (depending on alert type, and only for the alert’s corresponding message type). If the alert action is triggered, then its processing time is added. For example, displaying a popup message might take 1 sec, and sending an email might take a few seconds.
Computer capabilities – faster CPU and memory (RAM) enable faster processing, if your computer has enough physical memory to avoid swapping. Adding memory is typically much more cost-effective than upgrading the CPU.
This means that without any defined alert or user-specified callback, nor any other code running in the background (for example, a Matlab data analysis program), we can expect IQML to process up to 500-1000 IQFeed messages per second by default.
This is a relatively fast throughput, but if you stream real-time quotes for hundreds of liquid securities concurrently then you might reach this limit. When this happens, Matlab may be so bogged-down from the flood of incoming messages that it will become unresponsive, and you may need to restart IQConnect and/or Matlab.
Similarly, if you request a blocking (non-streaming) request with multiple data items (for example, thousands of historical data or news items), the query may take a while to process, requiring us to set a higher-than-default Timeout parameter value. For example, if you issue a blocking request for 20K data bars, IQFeed will send 20K data messages (one message per bar). If each of these messages takes 1-2 msecs to process, the total processing time for the IQML query will be 20-40 secs.
When IQFeed is connected, it continuously sends messages to IQML: periodic “heartbeat” and status messages, and messages for any active streaming quotes or news events that you requested. These messages are automatically processed by IQML in the background, reducing the CPU time that is left available to process other IQML queries (e.g., a blocking historical data query) or Matlab analysis functions. It is therefore advisable to stop streaming IQFeed data when not needed, even if only temporarily.
When requesting historic or streaming data items, IQML’s internal data buffer grows with each new item, causing increasing memory reallocation overhead as the buffer gets larger. Depending on your computer speed, the slowdown effect may start to become noticeable when the number of items > 1000. In such cases, setting parameters such as MaxItems may assist to alleviate the problem by limiting memory reallocations.
In cases of high load, numerous streaming message events (quote ticks, news items etc.) may inundate the system, causing Matlab to appear frozen for long minutes, until the events backlog is fully processed. IQML’s built-in overflow handling mechanism (the OverflowMode and MaxEventsBacklog parameters, §6.1) could assist in such cases.
IQFeed includes a built-in logging mechanism. Setting the logger to record fewer data items will improve run-time performance (see §12.2). IQFeed’s log file can become very large very fast and reduce system performance if left unattended. It is intended to be used for troubleshooting purposes only and not on a constant basis. This is especially true if you log streaming data, large historic data, and/or Debug data.
Also, by default IQFeed stores the log file in the user’s Documents folder, which might be automatically synced to OneDrive or Dropbox clouds. The log file can grow to several GBs per day, so such a sync could result in marked slowdown.16 To improve performance, disable this sync, or place the log file in a non-synced folder.
3.6.2 Paralellization
With the Professional and trial IQML licenses, you can use Matlab’s Parallel Computing Toolbox to parallelize IQFeed queries. This can be done both externally (placing IQML commands in parfor/spmd blocks, so that they will run independently), and internally (for some IQML query types, using the UseParallel parameter). If you have the Standard IQML license, or if you do not have Matlab’s Parallel Computing Toolbox, you can still run concurrent IQML commands in separate Matlab sessions, just not in the same session.
IQML automatically tries to parallelize queries when the UseParallel parameter value (default: false) is set to 1 or true. The list of parallelizable queries includes:
Requests resulting in multiple blocking queries in a single IQML command (for example, historical data for multiple symbols or a date range – see §5)
Requests for full news-story of all returned news headlines in a blocking query, using the GetStory parameter (see §7.2)
Requests for fundamental/quotes data on all symbols in an options-chain or futures-chain, using the WhatToShow parameter (see §8.2)
When setting UseParallel to 1 or true, IQML will use parallel Matlab tasks (so-called ‘headless workers’ or ‘labs’) from the currently-active parallel pool created by the Parallel Computing Toolbox. If no pool is active, the default local pool is automatically started.
Here is a run-time example showing the effect of using a 4-worker pool to parallelize a news-story query, resulting in a 3.5x speedup (not 4x, due to parallelization overheads):
>> tic, data = IQML('news', 'DataType','headlines', 'MaxItems',100, ...
'GetStory',1); toc
Elapsed time is 56.311768 seconds.
>> parpool('local',4) % start 4 workers in parallel pool (optional)
>> tic, data = IQML('news', 'DataType','headlines', 'MaxItems',100, ...
'GetStory',1, 'UseParallel',1); toc
Elapsed time is 15.799185 seconds.
Depending on the number of CPU cores and available memory (RAM) on your computer, and the amount of work that the OS and other tasks as doing in the background, a pool of 14 workers (the maximum for IQFeed and IQML) can result in a speedup of up to 10-12x for parallelizable queries (such as historical data fetches), compared to the corresponding non-parallel (serial) query.
IQML parallelization has several performance implications:
Starting the parallel pool can take some time (a few seconds, up to a minute or two, depending on configuration). It is best to start the pool before any time-critical operations to avoid this startup time upon the first parallel query. Starting the pool (with an optional size) can be done using Matlab’s parpool function.
The default pool uses the same number of workers as the number of physical cores on your computer. This makes sense for CPU-intensive programs, but IQML queries are limited by I/O, not CPU. Therefore, unless you also use the parallel pool for CPU-intensive computations in your program, it makes sense to start a pool that has more workers than the number of CPU cores. You can configure your local cluster for this.17 Note that IQFeed limits (throttles) the rate of some query types, limiting the effective parallel pool size.18 Larger worker pools use extra CPU and memory and degrade IQML’s overall performance. Different pool sizes are optimal for different queries; you can try using various pools, to find an optimal size for your specific queries and system.
For any parallelizable IQML command, you can set a MaxWorkers parameter value that limits number of workers for that command (up to the pool size). When MaxWorkers < pool size, only MaxWorkers will be used to process the IQML command; the other workers will remain idle and consume no CPU.
Changing the parallel pool size entails a restart of the entire pool, which could take quite some time. It is advised to pre-allocate the maximal pool size that you will need (using the optional pool size argument of the parpool command, or by configuring the local cluster’s default pool size). When you need fewer workers (for example, to limit the number of parallel historical data queries), use the MaxWorkers parameter to limit the number of active workers, without modifying the pool size.
Note that the more parallel workers that you use, the more memory will be used by the program, and the higher the risk that some internal program elements might conflict with each other. You can test various pool sizes to check which size fits best for your particular computer and query types.
In addition to the workers startup time, each worker independently connects to IQFeed upon the first IQML command it encounters, taking a few extra seconds.
Due to parallelization overheads, inter-task memory transfers, and CPU task-switches (especially in a case of more workers than cores), speedup will always be smaller than the number of workers. The actual speedup will depend on query type and computer/OS configuration. Parallelization may even cause slowdown in some cases (e.g. quote queries, due to waiting for market events, not CPU).
It is only possible to parallelize workers on the local computer, not on other computers in a distributed grid/cluster/cloud. This is due to IQFeed/exchange limitations, which prohibit distribution of data to other computers.
Historical data queries have two parallelization variants:
When requesting data for multiple symbols, the symbols are split across the available parallel workers (up to the specified MaxWorkers value or the active pool size, whichever is lower), one or more complete symbol query per worker. Therefore, if you query fewer symbols than workers, some workers will remain idle (unused). In such a case, consider using a smaller parallel pool, with fewer workers, taking up less CPU and memory resources. Alternatively, run a serial for-loop over the symbols, parallelizing the date/time dimension (next bullet).
When requesting data for just a single symbol, the requested date/time segment (BeginDate to EndDate etc.) is evenly split across the workers. You should always set BeginDate to the earliest time of actual available data: without a date/time segment (for example no BeginDate), parallelization is impossible, so IQML reverts to slow serial mode. Likewise, when data is limited (for example, 1-sec intervals are only available for the past 180 days) and BeginDate is older than the earliest data, some workers will remain idle (processing a date/time range with little or no data); the actual work will be done by fewer workers than optimal, resulting in slower-than-optimal performance.
3.6.3 Reported data-fields
Also in the Professional IQML license, you can customize the fields that IQML reports for market data quotes and historic data bars. The Fields parameter can be set to a cell-array of strings ({'Bid','Ask','Last'}), or a comma-separated string ('Bid,Ask,Last'). All subsequent queries of the same type will only report the requested fields. For example:
>> data = IQML('quotes', 'Symbol','AAPL', 'Fields',{'Bid','Ask','Last'})
>> data = IQML('quotes', 'Symbol','AAPL', 'Fields','Bid,Ask,Last') %equivalent
data =
Symbol: 'AAPL'
Bid: 222.71
Ask: 222.91
Last: 222.11
Note: the fewer fields that you request, the faster the required processing time, by both IQFeed and IQML (see §4.1, §5.1-§5.5, §6.1 for details). Requesting fewer fields (as in the example above, which only requested 3 quote fields) results in faster processing and reduced memory consumption. To improve the run-time and reduce memory usage it is advisable to request only those data fields that your program actually needs.
7 IQConnect.exe on Windows, IQFeed application on MacOS. or ran as a Windows app on Mac/Linux using Parallels/Wine
8 In IQML, the Symbol and Symbols parameters are synonymous – you can use either of them, in any capitalization.
10 Using the 2nd (optional) output parameter of IQML implies a default value of false (0) for RaiseErrorMsgs (see §3.5)
11 For example, EURUSD.FXCM is a valid symbol, but EURUSD and USDEUR.FXCM are invalid
13 Standard license can be converted into a Professional license at any time; contact info@undocumentedmatlab.com for details.
14 Note: table format is only available on Matlab releases R2013b or newer; timetables only on R2016b or newer.
15 Using the 2nd (optional) output parameter of IQML implies a default value of false (0) for the RaiseErrorMsgs parameter.
18 Until Dec. 2020, IQFeed allowed up to 15 concurrent (parallel) historical data queries; since then 50 queries/sec are allowed, regardless of the number of concurrent queries. See http://forums.iqfeed.net/index.cfm?page=topic&topicID=5832 for details.