djolivet Posted April 17, 2006 Report Share Posted April 17, 2006 Find attached a VI that takes a number, increments it, and then displays it. I find that when I perform this operation on the numbers 16,777,216 or 36,564,036 the result is incorrect. This seems to occur on various ranges around this number as well. I'd appreciate it if a few people would have a look at this VI to confirm that I'm not missing something obvious. Thanks Download File:post-1881-1145285924.vi Quote Link to comment
Guillaume Lessard Posted April 17, 2006 Report Share Posted April 17, 2006 On a single, you have only 23 bits available for mantissa (significant digits) of your number. Your first example is 2^24, and adding 1 is below the precision of the format: it gets rounded off. Same thing for the second example. More verbosely but more precisely: http://en.wikipedia.org/wiki/Single_precision Quote Link to comment
djolivet Posted April 17, 2006 Author Report Share Posted April 17, 2006 That explains it. So this is not a bug. Michael, can you move/delete this post to somewhere more appropriate? Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.