Strange behavior with numeric coercion

[TobyD]

I don't know if this is a bug or if I'm missing something (maybe a rounding issue)...

1. Place a numeric control on the Front Panel
   
2. On the Data Entry tab of the properties window, uncheck Use Default Limits
   
3. Set Minimun to 0.08 (coerce), Maximum to 2.0 (coerce), Increment to 0.08 (coerce to nearest) and click OK
   
4. Type 0 into the control and it should coerce to 0.08
   
5. Now use the increment/decrement arrows to adjust the value. Once you get to 0.24 or above, it is not possible to decrement below 0.16
   

I would expect to be able to step back down to 0.08. Does anyone have an explanation for this? Should I file a bug report? This happens on 8.5.1 and 8.6.1f1

Thanks,

Toby

[Mark Smith]

QUOTE (TobyD @ Apr 7 2009, 11:24 AM)

I don't know if this is a bug or if I'm missing something (maybe a rounding issue)...

1. Place a numeric control on the Front Panel
   
2. On the Data Entry tab of the properties window, uncheck Use Default Limits
   
3. Set Minimun to 0.08 (coerce), Maximum to 2.0 (coerce), Increment to 0.08 (coerce to nearest) and click OK
   
4. Type 0 into the control and it should coerce to 0.08
   
5. Now use the increment/decrement arrows to adjust the value. Once you get to 0.24 or above, it is not possible to decrement below 0.16
   

I would expect to be able to step back down to 0.08. Does anyone have an explanation for this? Should I file a bug report? This happens on 8.5.1 and 8.6.1f1

Thanks,

Toby

It's numeric roundoff due to floating point imprecision

As type double, 0.08 entered becomes 0.0800000000000000017, increment up, you get 0.160000000000000003. Increment up again you get 0.239999999999999991 - now, when you increment down you get 0.159999999999999976 and you can't step down another 0.08 and stilll be above the allowed minimum

Mark

[TobyD]

QUOTE (mesmith @ Apr 7 2009, 10:35 AM)

It's numeric roundoff due to floating point imprecision

As type double, 0.08 entered becomes 0.0800000000000000017, increment up, you get 0.160000000000000003. Increment up again you get 0.239999999999999991 - now, when you increment down you get 0.159999999999999976 and you can't step down another 0.08 and stilll be above the allowed minimum

Mark

Thanks Mark. I figured it was something like this. It seems to me like it should do the math first and then coerce to the nearest "in-range" value though.

[Rolf Kalbermatter]

QUOTE (TobyD @ Apr 7 2009, 01:22 PM)

Thanks Mark. I figured it was something like this. It seems to me like it should do the math first and then coerce to the nearest "in-range" value though.

I think you are right but that might have other implications somewhere they tried to fix by ordering the operations like this. With such things it is often very hard to fix something without breaking something else, and even harder to think of possibilities what might break by the fix.

Rolf Kalbermatter