# Std Deviation and Variance.vi outputs erroneous value in corner cases

## Recommended Posts

Posted (edited)

Feed an array comprised of non trivial equal numbers (e.g. a 1,000 values of 1.2345) to the Std Deviation and Variance.vi of the Mathematics>>Probability & Statistics palette and look at the standard deviation output using a display format such as that shown below:

You will be surprised to see that the standard deviation is non zero and within the range of the precision for the type (~1E-15).

While this might not be an issue in most use cases, it is still not an optimal result, and suggests that the calculation done in the called DLL is not optimal (it probably uses the standard formula, but because of round-off errors, the outcome is non-zero).

I am posting this in the "bug" category, although it is not strictly speaking one, just to warn unsuspecting users (like me), that dealing with floating point numbers comes with its load of gotchas.

Here is a snippet to experiment with the phenomenon:

Again the display format of the standard deviation indicator needs to be chosen wisely to reveal the round off error.

Edited by X___
typo correction

I do not think it is a bug, just floating point math. You don't have infinite precision. See below for another example.

Yes, as a general rule, it is fairly unusual to stick an equal sign on a floating point number. This applies to most programming languages, not just LabVIEW. So, I would never stick an equals zero on the output of std dev but I have once or twice broken the rule like this:

On 6/21/2024 at 8:40 PM, X___ said:

You will be surprised to see that the standard deviation is non zero and within the range of the precision for the type (~1E-15).

Even though many mathematic operators are commutative (meaning the order of execution of those operators does not change the result) this only applies if you have infinite resolution. But in order for a computer to have infinite resolution when doing numerical calculations, it needs also infinite amount of resources such as memory. Obviously that can't be done! 😁

## Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

×   Pasted as rich text.   Paste as plain text instead

Only 75 emoji are allowed.

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
• ### Similar Content

• #### Units and compound arithmetic bug

By X___,

• 0 replies
• 1,166 views
• #### Bug/Documentation Error in Continuous Random VI: Gamma Random

By X___,

• 2 replies
• 939 views
• #### Bookmarks in malleable VIs do no show up in the bookmark manager

By X___,

• 0 replies
• 966 views
• #### Matrix !@#\$%^&*()_+ Compound Arithmetic

By X___,

• 2 replies
• 1,180 views
×