# Std Deviation and Variance.vi outputs erroneous value in corner cases

## Recommended Posts

Posted (edited)

Feed an array comprised of non trivial equal numbers (e.g. a 1,000 values of 1.2345) to the Std Deviation and Variance.vi of the Mathematics>>Probability & Statistics palette and look at the standard deviation output using a display format such as that shown below:

You will be surprised to see that the standard deviation is non zero and within the range of the precision for the type (~1E-15).

While this might not be an issue in most use cases, it is still not an optimal result, and suggests that the calculation done in the called DLL is not optimal (it probably uses the standard formula, but because of round-off errors, the outcome is non-zero).

I am posting this in the "bug" category, although it is not strictly speaking one, just to warn unsuspecting users (like me), that dealing with floating point numbers comes with its load of gotchas.

Here is a snippet to experiment with the phenomenon:

Again the display format of the standard deviation indicator needs to be chosen wisely to reveal the round off error.

Edited by X___
typo correction

I do not think it is a bug, just floating point math. You don't have infinite precision. See below for another example.

Yes, as a general rule, it is fairly unusual to stick an equal sign on a floating point number. This applies to most programming languages, not just LabVIEW. So, I would never stick an equals zero on the output of std dev but I have once or twice broken the rule like this:

• 1
On 6/21/2024 at 8:40 PM, X___ said:

You will be surprised to see that the standard deviation is non zero and within the range of the precision for the type (~1E-15).

Even though many mathematic operators are commutative (meaning the order of execution of those operators does not change the result) this only applies if you have infinite resolution. But in order for a computer to have infinite resolution when doing numerical calculations, it needs also infinite amount of resources such as memory. Obviously that can't be done! 😁

• ### Similar Content

• #### Units and compound arithmetic bug

By X___,

• 0 replies
• 1,186 views
• #### Bug/Documentation Error in Continuous Random VI: Gamma Random

By X___,

• 2 replies
• 967 views
• #### Bookmarks in malleable VIs do no show up in the bookmark manager

By X___,

• 0 replies
• 981 views
• #### Matrix !@#\$%^&*()_+ Compound Arithmetic

By X___,

• 2 replies
• 1,201 views
×