Searching \ for '[PIC]: AN575, FP24.A16, INT24 ROUNDING PROBLEM' in subject line. ()
Make payments with PayPal - it's fast, free and secure! Help us get a faster server
FAQ page: www.piclist.com/techref/microchip/devices.htm?key=pic
Search entire site for: 'AN575, FP24.A16, INT24 ROUNDING PROBLEM'.

Exact match. Not showing close matches.
PICList Thread
'[PIC]: AN575, FP24.A16, INT24 ROUNDING PROBLEM'
2000\11\07@074807 by James Hillman

flavicon
face
Dear All,

I am using the the latest version of Microchip's AN575. I found a rounding error in file FP24.A16, subroutine INT24 when FPFLAGS,RND is set.

Decimal     24bit Float    INT24 result    Decimal result
9.4999      8217FF        0009              9
9.50          821800         000A             10
9.9999       821FFF        000A             10        10.4999     8227FF        000A              10
10.50        822800         000A              10        (should be 10 because it is equidistant from 10 and 11 so rounds to the number with bit 0 0)
10.999      822FFF        000A               10        (should be 11 !)
11.000      823000         000B              11
11.4999    8237FF        000B              11
11.50        823800         000C              12        (should be 12 because it is equidistant from 11 and 12 so rounds to the number with bit 0 0)
11.999      823FFF        000C              12        12.0         824000         000C              12
12.4999    8247FF         000C              12
12.50       824800          000C              12
12.9999    824FFF         000C              12       (should be 13 !)
13.0          825000         000D              13  etc

Has anyone else who has noticed this error fixed it or know where I can get a working version ? I contacted Microchip and they could not help me. I am using this version :
FP24.A16        07/10/96 07:50
(RCS Header $Id: fp24.a16 2.7 1996/10/07 13:50:29 F.J.Testa Exp $)



James Hillman

--
http://www.piclist.com hint: To leave the PICList
spam_OUTpiclist-unsubscribe-requestTakeThisOuTspammitvma.mit.edu




2000\11\09@014946 by Nikolai Golovchenko

flavicon
face
part 1 3306 bytes content-type:text/plain; charset=Windows-1251 (decoded quoted-printable)

Hi James,

I've found also a few more problems with the INT24 routine:

1) when AEXP==0, which means zero floating number, but
AARGB0:1 is not zero, return will be incorrect. So AARGB0:1
have to be cleared in that case.

2) problem with overflow checking (for example -32769
(0x8e8001) will result in 0xC78e, instead of 0x8000)

There is really something wrong with rounding. They teach in
school that if a number is in the middle between two
integers, the highest by absolute value is taken. For
example, MATLAB has four rounding functions:

ROUND - round to closest integer (like in school)
FIX - round towards zero (ignore fractional part)
CEIL - round towards plus infinity
FLOOR - round towards minus infinity

let x = [-1:0.25:1]

x = [-1:0.25:1]
x =
   -1 -0.75  -0.5 -0.25     0  0.25   0.5  0.75     1
round(x)
ans =
   -1    -1    -1     0     0     0     1     1     1
fix(x)
ans =
   -1     0     0     0     0     0     0     0     1
ceil(x)
ans =
   -1     0     0     0     0     1     1     1     1
floor(x)
ans =
   -1    -1    -1    -1     0     0     0     0     1

I assume that the INT24 routine should implement ROUND
function. But why there is a check for AARGB1.0 (LSb of
integer result)?...

Check attachment for a fixed version (I hope) of INT24
routine (fpint.asm).

Hope it helps,
Nikolai

---- Original Message ----
From: James Hillman <.....jamesKILLspamspam@spam@IND-INTERFACE.CO.UK>
Sent: Tuesday, November 07, 2000 14:49:34
 To: PICLISTspamKILLspamMITVMA.MIT.EDU
Subj: [PIC]: AN575, FP24.A16, INT24 ROUNDING PROBLEM

{Quote hidden}


part 2 6916 bytes content-type:application/octet-stream; name="fpint.asm"
(decode)

part 3 146 bytes
--
http://www.piclist.com hint: The list server can filter out subtopics
(like ads or off topics) for you. See http://www.piclist.com/#topics




2000\11\09@065531 by James Hillman

flavicon
face
Just for the record. To convert 24bit float to 16bit integer without a
rounding error :

As a quick fix I added 0.5 to the 24bit float using FPA24, cleared
FPFLAGS,RND to turn rounding off and converted the number to a 16bit integer
using INT24. I found that this always rounded correctly, but incurred the
delay introduced by the call to FPA24.

James Hillman

--
http://www.piclist.com hint: The list server can filter out subtopics
(like ads or off topics) for you. See http://www.piclist.com/#topics




More... (looser matching)
- Last day of these posts
- In 2000 , 2001 only
- Today
- New search...