Typo corrected, meant ASCII of course (ASCII is represented by 1 byte in UTF-8, and 2 bytes in UTF-16)

I’ve recently came across a MATLAB bug that completely blew my mind (for over a year)… Imagine you have an MCR application that executes for the first time, fails the second time (some windows DLL error), works the third time, fails the forth time, so on so forth…

Talked to MATLAB support and they guided us to run tests and gather information, the standard stuff that you do to try and find out what’s wrong. This bug is especially difficult to figure out because the application runs fine with the majority of our clients and we cannot duplicate the issue in house… The clients that ran fine are US clients (hint, hint…) the clients that had the problem are all from outside US (hint, hint…). Imagine we have to coordinate with our clients to perform the information gathering across timezones (we are US Mountain time) and our clients having the issue are from England, Denmark, etc…

So one night as I sit watching TV late at night, a light bulb suddenly lit up… It has something to do with Locale settings (actually it has to do with the default character set) within MATLAB… Our development machines are all running Windows US locale, the machine that produces the MCR executable is also a US locale machine. So after some quick checking setting Windows to non-US locale and running our US locale compiled MCR app, we finally reproduced the error. Subsequently, talking to MathWorks and they admit that it is a known bug and has to do with running a compiled MCR application on another computer with a different locale cause an error message to be displayed in the DOS command window if the MCR application is compiled as a DOS application and we can see the error message. But if the app is compiled as a Windows application (no DOS window while the app is running), the output error message causes MCR to crash…

The workaround is to compile our application as a traditional app with a DOS command window, or alternatively patch our MATLAB R2013b (MCR) or upgrade to MATLAB R2014a (where the bug is fixed).

So the lesson of this story is that it is really unfortunate that MathWorks is not sharing bug reports which can cause a lot of grief for MATLAB programmers…

Lastly, thank you Amro for sharing how to change the default character sets within MATLAB which actually might help us to avoid this bug if you had posted this a couple of weeks earlier…

Very good and insightful comments here. Personally, I feel that nowadays the most sensible text encoding is UTF-8 and I wish it would become the default in MATLAB sooner rather than later. My workflow is very English-centric, but even I want to credit someone or reference an article once in a while, at which point I want to use all kinds of diacritics and maybe even different alphabets.

My approach to ensure this is: Check in my startup.m that

feature('DefaultCharacterSet')

returns UTF-8. If not, check via

feature('Locale')

which locale MATLAB has detected for my system and alter $MLROOT/bin/lcdata.xml accordingly (change the encoding attribute of the matching locale element).

I haven’t really tested it, but I could imagine that there are less issues with this method than changing the encoding via the above feature command, which would only happen after all of MATLAB is loaded and the editor has already reopened my files with the wrong encoding.

I guess the choice of UTF-16 as the internal encoding is understandable, given that this is the encoding on an API level on Windows and also the encoding underlying all of Java.

@SteveEddins:

Thanks for the response Steve. I took another shot at the problem, and I’ve managed to work with code points outside the BMP plane this time (i.e U+010000 to U+10FFFF)!

As I’ve shown before, the CHAR function unfortunately does not know how to deal with those code points directly; the trick is to use NATIVE2UNICODE function to convert a vector of bytes (encoded using any of the supported encodings) back to the abstract character they represent.

So my previous example can be written as:

% U+10437 is encoded in UTF-8 as 4-bytes: 0xF0 0x90 0x90 0xB7
cc = native2unicode(hex2dec({'F0' '90' '90' 'B7'})', 'UTF-8')

% MATLAB internally uses UTF-16, which in this case
% is encoded as a surrogate pair: 0xD801 0xDC37
>> whos cc
  Name      Size            Bytes  Class    Attributes
  cc        1x2                 4  char               

>> double(cc)
ans =
       55297       56375      % high/low surrogates

% using an appropriate font, R2014b is now capable of showing it in a plot!
title(cc, 'FontSize',144, 'FontName','Segoe UI Symbol')

This does confirm that MATLAB is using UTF-16 internally to represent characters, even though there are a couple of places that still need attention… Sorry for doubting you guys 🙂

—

I hope Yair doesn’t mind me documenting this stuff here, but in the spirit of filing bugs, here are some of my findings:

– CHAR function truncates characters outside the range of BMP plane.
– even if we managed to store a non-BMP character like I did in the above example, there is a problem of “leaky abstractions”; the resulting character has length(cc)==2, while it’s supposed to be a single character! This means that indexing into a string is messed up as well, and won’t always be what we expect: str(5:10)
– the MATLAB editor does not handle files with BOM markers well (either UTF-8 or UTF-16).
– reading files with UTF-16 text using FOPEN/FREAD functions has some issues. I guess that’s why UTF-16 is not listed as a supported encoding in FREAD doc page, even though it is accepted as an option value.
– the command window in the IDE has small rendering issues with Unicode text that involves combining diacritical-marks. Take for example the letter A with accents above and below: char(hex2dec({‘0061’, ‘0301’, ‘0317’})’) or ‘á̗’
– other advanced Unicode functionality: collation, case mapping, text segmentation, regular expressions, etc.. For instance sort({‘a’, ‘z’, ‘e’, ‘é’, ‘ä’}) sorts by code point value, rather than alphabetically, where the accented characters should come after the regular ones, not last after the ‘z’. Another example is case conversion where in German upper(‘ß’) should be ‘SS’!

It’s worth mentioning that MATLAB is not the only language that suffers from the “UTF-16 curse” (JavaScript comes to mind!). I only wish TMW had chosen UTF-8 instead, which many agree is a better encoding in general:
– http://utf8everywhere.org/
– http://programmers.stackexchange.com/questions/102205/should-utf-16-be-considered-harmful
This is especially true when working at the level of MEX-files.

@Amro – where have you been all these years?! – this would have deserved a dedicated post…

Anything else you’re keeping up your sleeve?

MATLAB uses UTF-16 internally for character representation. Some parts of MATLAB don’t know yet what to do with code points about 65535.

@Yair: you can also make it work in R2014a and earlier, you just have to change the default character set (undocumented of course!):

% works in R2014a
feature('DefaultCharacterSet','UTF-8')
uicontrol('String',char(1495:1499))

On my Windows machine with “en-US” locale, the default charset was set to “windows-1252”.

@Amro – you’re right, my bad: I was working on 14a this morning, where uicontrols indeed do not display the non-Latin text without the Java hack. It works correctly in 14b. Sorry for the mixup.

For the reference of others, in 14a and earlier we can use the underlying Java components of the uicontrols to display non-Latin labels (unnecessary in 14b):

hButton = uicontrol('String',char(1495:1499));  % no label displayed in 14a; ok in 14b
jButton = findjobj(hButton);
jButton.setText(char(1495:1499));  % label shown ok