Jump to content

Expected zero isn't


Recommended Posts

Hi all,

Stumbled across this today in a larger application that I'm working on. Have stripped out the core and put it in the attached very simple VI. The VI takes a double precision number (which should be a multiple of 0.1) and, via a While loop with a shift register, successively subtracts 0.1 (also a double precision number) from it. When the result is zero the While loop should stop, except it doesn't: the result is never zero, only very nearly, e.g. 1.23456789E-16, or something similar.

I'm a bit baffled. As ever, you can work around it, but I'd like to understand what's going on. If anyone can enlighten me I'd appreciate it.

Regards.

Link to comment

You are right, this will (most likely) never work. Since a double is a floating point binary representation of a decimal number you always get rounding errors.

If you want to do something like this use integers!

Get your DBL multiply it by 10 and use round to nearest and then convert it to I16. Now do in every iteration a subtraction of 1 (one) and you will hit 0!

Ton

Link to comment

QUOTE(Graeme @ Apr 11 2007, 05:32 PM)

Hi all,

Stumbled across this today in a larger application that I'm working on. Have stripped out the core and put it in the attached very simple VI. The VI takes a double precision number (which should be a multiple of 0.1) and, via a While loop with a shift register, successively subtracts 0.1 (also a double precision number) from it. When the result is zero the While loop should stop, except it doesn't: the result is never zero, only very nearly, e.g. 1.23456789E-16, or something similar.

I'm a bit baffled. As ever, you can work around it, but I'd like to understand what's going on. If anyone can enlighten me I'd appreciate it.

Regards.

I saw this quite a few years ago. If you extend your significant digits out to about 18 you'll see where the "error" is coming from, but it's not really an error. It has something to do with the way your PC deals with dbl's. It can only handle so many bits.

Link to comment

As ever, many thanks to those that replied. Well, you learn something every day, don't you! Thanks for your comments and links. Reading these it all seems rather obvious now, but it never crosses your (well, my) mind initially. So, unless I've misread the links, floating point maths is fine but, be careful comparing such numbers.

Regards, Graeme.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use.