One basic fact about floating point arithmetic is that it is not associated. That is, order of operations matters. For example, (1e-20 + 1) – 1 returns 0 while 1e-20 + (1 – 1) would return 1e-20. In the case of for vs parfor, the different number of threads used by MATLAB in the two loop settings change the order of floating point operations. This cause the results to be different. There is nothing wrong with parfor. For double precision, both computed results are numerically correct. As a good practice with floating point arithmetic, we should always test the computed results to be accurate within a tolerance rather than testing for equality.

As to the complete loss of precision in std for single precision data, we can trace it back to the computation of sum. The root of the problem is that the current algorithm used by sum simply does not have enough precision to cope with the amount of single precision data. This makes the results seemingly sensitive to the number of computational threads. I will explain more below. But first I would admit that MATLAB needs to do better and we plan to improve the algorithm to better cope with this amount of data in a future release.

To understand what is happening, consider the case of a vector of all ones. If you simply summing up the numbers, like this

x = ones(1, 1e8, 'single');
s = 0;
for k = 1:numel(x)
   s = s + x(k);
end
 
>> s
s =
  single
    16777216

The number 16777216 is not a random integer. It is 2^24. This simple implementation of sum starts to lose precision after 2^24 (more than 16 million) iterations. This is because the distance between single(2^24) and the next larger floating point number is 2. Adding 1 to single(2^24) would result in single(2^24) due to round off error. This is what you would get with sum if you use only one computational thread, like what happens when you use parfor.

>> maxNumCompThreads(1);
>> sum(x)
ans =
  single
    16777216

When you run sum in a for loop, the computation will be multithreaded and you will see a different set of round off error. Look at the output of sum when running with different number of threads.

for k = [1 2 4 8]
   maxNumCompThreads(k);
   sum(x)
end
 
ans =
  single
    16777216
 
ans =
  single
    33554432
 
ans =
  single
    67108864
 
ans =
  single
   100000000

Only 8 threads gets the right answer because each thread gets 12500000 elements, which is less than 2^24 elements, so no error in the partial sum, and no error when combining the partial sums from each thread.

This problem is not unique to single precision. The limitation is present for double precision as well, however, since there is more than twice the precision, it takes substantially more numbers to see the issue.

Regards,
Bobby Cheng, MathWorks.

After some checks, I think it could be a matter associated with loss of significance in case of calculations (like std, mean, sum, etc.) which involve summing up relatively small numbers (here we are randomly generating from U(0,1)) to (partial) summations which are large with respect to the available precision.

In case of multi-thread computations, since the summations are (likely) carried out in separate pieces, the effect is seen “later” in terms of dimension of the array under analysis. Instead, in single-thread calculations this is seen “before”.

I have prepared a script + a function which might be useful to show this effect (I used R2016a – change the first switch to change the test function):

— Script —

clear
rng('shuffle','twister');
 
switch 1, %change the function here
    case 1, %STD
        f=@(x) std(x); %function to be evaluated
        fname='std(x(1:i))';
    case 2, %MEAN
        f=@(x) mean(x); %function to be evaluated
        fname='mean(x(1:i))';
    case 3, %SUM
        f=@(x) sum(x); %function to be evaluated
        fname='sum(x(1:i))';        
end
 
NTO = maxNumCompThreads; %number of threads in console - it is assumed that NTO>1
 
Nd = 5*10^7; %Number of data - At least abt 4*10^7 is needed to see the problem in my case. Be very careful with RAM limitations (try smaller values first, and then increase progressively)! 
arr_double = rand(1,Nd);
arr_single = single(arr_double);
ind = round(linspace(100,Nd,100));
 
f_double_MT = f_test_progressive(f,ind,arr_double);
f_single_MT = f_test_progressive(f,ind,arr_single);
 
%go to single thread:
maxNumCompThreads(1);
 
f_double_ST = f_test_progressive(f,ind,arr_double);
f_single_ST = f_test_progressive(f,ind,arr_single);
 
%go back to original number of threads:
maxNumCompThreads(NTO);
 
%now use a parfor:
[f_double_PF, f_single_PF] = deal(cell(1));
parfor i = 1
    f_double_PF{i} = f_test_progressive(f,ind,arr_double);
    f_single_PF{i} = f_test_progressive(f,ind,arr_single);    
end
f_double_PF = f_double_PF{1};
f_single_PF = f_single_PF{1};
 
figure;
hold all;
plot(ind,f_double_MT,'-o', 'Displayname','Double-MT');
plot(ind,f_single_MT,'--o','Displayname','Single-MT');
plot(ind,f_double_ST,'-s', 'Displayname','Double-ST');
plot(ind,f_single_ST,'--s','Displayname','Single-ST');
plot(ind,f_double_PF,'-d', 'Displayname','Double-PF');
plot(ind,f_single_PF,'--d','Displayname','Single-PF');
xlabel('Progressive index - i');
ylabel(fname);
legend('Location','Best');

— End script —

— Support function —

function out = f_test_progressive(fh,ind,x)
   ind = unique(ind);
   Ntest = numel(ind);
 
   switch class(x),
      case 'double',  out = nan(1,Ntest);
      case 'single',  out = single(nan(1,Ntest));
      otherwise,      error('f_test_progressive:ErrorDataType','Data type not allowed');                
   end
 
   for jt=1:Ntest
      out(jt) = fh(x(1:ind(jt)));
   end
end

— End support function —

Basically a function specified by handle “fh” (first switch) is calculated on the first ind(jt) elements of the array, considering single/double precision, and single-/multi-thread, as well as parfor (by default single-thread).

In my case, with single-thread, I see the issue occurring particularly with single-precision, single-thread calculations. It is interesting to see how the effect is different depending on the specific function which is evaluated. “std”, in my case, is particularly problematic, while “mean” and “sum” in single-precision/single-thread basically agree with double precision results up to a dimension of about 3.3e7 (in my case).

I hope it can be useful.
Gabriele

Perhaps I’m wrong, but it seems to be something related to intrinsic parallelization combined with single accuracy. In fact, as a basis, the main console is multi-thread. On the other hand workers are started, as a basis, using a single thread. Changing from multi-thread to single-thread computations in the console, I can reproduce the issue you have identified with parfor.

This example reproduces the issue from within the console:

clear
rng('shuffle','twister');
 
NTO = maxNumCompThreads; %number of threads in console - it is assumed that NTO>1
 
arr_double = rand(1,10000000*5);
arr_single = single(arr_double);
 
std_double_MT = std(arr_double);
std_single_MT = std(arr_single);
 
%go to single thread:
maxNumCompThreads(1);
 
std_double_ST = std(arr_double);
std_single_ST = std(arr_single);
 
%go back to original number of threads:
maxNumCompThreads(NTO);
 
%now use a parfor:
std_double_PF = NaN;
std_single_PF = NaN;
parfor i=1
    std_double_PF(i) = std(arr_double);
    std_single_PF(i) = std(arr_single);
end;
 
fprintf('\n\nResults from console with intrinsic parallelization (multi-thread):');
fprintf('\nstd_double_MT=%f',std_double_MT);
fprintf('\nstd_single_MT=%f',std_single_MT);
 
fprintf('\n\nResults from console with single thread:');
fprintf('\nstd_double_ST=%f',std_double_ST);
fprintf('\nstd_single_ST=%f',std_single_ST);
 
fprintf('\n\nResults from parfor:');
fprintf('\nstd_double_PF=%f',std_double_PF);
fprintf('\nstd_single_PF=%f',std_single_PF);
 
fprintf('\n\nAnalytical result:');
fprintf('\n       std_an=%f',sqrt(1/12));

———————————————————

Outcome in my case (I have NTO=4):

Results from console with intrinsic parallelization (multi-thread):
std_double_MT=0.288674
std_single_MT=0.286997
 
Results from console with single thread:
std_double_ST=0.288674
std_single_ST=0.309665
 
Results from parfor:
std_double_PF=0.288674
std_single_PF=0.309665
 
Analytical result:
       std_an=0.288675

I played with different sizes for arr_double: [2e7 5e7 8e7 1e8 2e8],
parforSingleAccuracy-1 gives following:
[0.0001 0.4901 1.3841 1.9798 0]

parforDoubleAccuracy-1 gives following:
[0.1350 0.1619 0.1648 0.4738 0] * 1e-12

The last (for size 2e8) is zero because it exceeds memory, and matlab gives a warning and switches to local computation.

parfor ii=1:10; q(ii).process; q(ii).returnVal=ii; end

I end up debugging this over and over though.

–zed.