piclist 2000\11\07\074807a >
Thread: AN575, FP24.A16, INT24 ROUNDING PROBLEM
www.piclist.com/techref/microchip/devices.htm?key=pic
flavicon
face BY : James Hillman email (remove spam text)



Dear All,

I am using the the latest version of Microchip's AN575. I found a rounding error in file FP24.A16, subroutine INT24 when FPFLAGS,RND is set.

Decimal     24bit Float    INT24 result    Decimal result
9.4999      8217FF        0009              9
9.50          821800         000A             10
9.9999       821FFF        000A             10        10.4999     8227FF        000A              10
10.50        822800         000A              10        (should be 10 because it is equidistant from 10 and 11 so rounds to the number with bit 0 0)
10.999      822FFF        000A               10        (should be 11 !)
11.000      823000         000B              11
11.4999    8237FF        000B              11
11.50        823800         000C              12        (should be 12 because it is equidistant from 11 and 12 so rounds to the number with bit 0 0)
11.999      823FFF        000C              12        12.0         824000         000C              12
12.4999    8247FF         000C              12
12.50       824800          000C              12
12.9999    824FFF         000C              12       (should be 13 !)
13.0          825000         000D              13  etc

Has anyone else who has noticed this error fixed it or know where I can get a working version ? I contacted Microchip and they could not help me. I am using this version :
FP24.A16        07/10/96 07:50
(RCS Header $Id: fp24.a16 2.7 1996/10/07 13:50:29 F.J.Testa Exp $)



James Hillman

--
http://www.piclist.com hint: To leave the PICList
piclist-unsubscribe-request.....spammitvma.mit.edu




<006001c048b9$2e21f1a0$0800a8c0@IIRACS> quoted-printable

See also: www.piclist.com/techref/microchip/devices.htm?key=pic
Reply You must be a member of the piclist mailing list (not only a www.piclist.com member) to post to the piclist. This form requires JavaScript and a browser/email client that can handle form mailto: posts.
Subject (change) AN575, FP24.A16, INT24 ROUNDING PROBLEM

month overview.

new search...