Jump to content

Std Deviation and Variance.vi outputs erroneous value in corner cases

Recommended Posts



Feed an array comprised of non trivial equal numbers (e.g. a 1,000 values of 1.2345) to the Std Deviation and Variance.vi of the Mathematics>>Probability & Statistics palette and look at the standard deviation output using a display format such as that shown below:



You will be surprised to see that the standard deviation is non zero and within the range of the precision for the type (~1E-15).

While this might not be an issue in most use cases, it is still not an optimal result, and suggests that the calculation done in the called DLL is not optimal (it probably uses the standard formula, but because of round-off errors, the outcome is non-zero).

I am posting this in the "bug" category, although it is not strictly speaking one, just to warn unsuspecting users (like me), that dealing with floating point numbers comes with its load of gotchas.


Here is a snippet to experiment with the phenomenon:


Again the display format of the standard deviation indicator needs to be chosen wisely to reveal the round off error.



Edited by X___
typo correction
Link to comment

Yes, as a general rule, it is fairly unusual to stick an equal sign on a floating point number. This applies to most programming languages, not just LabVIEW. So, I would never stick an equals zero on the output of std dev but I have once or twice broken the rule like this:

std dev round fix.png

  • Haha 1
Link to comment
On 6/21/2024 at 8:40 PM, X___ said:

You will be surprised to see that the standard deviation is non zero and within the range of the precision for the type (~1E-15).

Even though many mathematic operators are commutative (meaning the order of execution of those operators does not change the result) this only applies if you have infinite resolution. But in order for a computer to have infinite resolution when doing numerical calculations, it needs also infinite amount of resources such as memory. Obviously that can't be done! 😁

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Create New...

Important Information

By using this site, you agree to our Terms of Use.