Re: [SLUG] Math Precision

From: Eben King (eben1@tampabay.rr.com)
Date: Sun Feb 06 2005 - 11:28:09 EST


On Sun, 06 Feb 2005 06:24:45 -0500, Kwan Lowe <kwan@digitalhermit.com>
wrote:

> Paul M Foster wrote:
>> Programmers:
>>
>> I'm using SQLite for some payroll database work, and seeing a lot of
>> rounding errors, or what are really errors caused by trying to store
>> decimals as binaries. Things like 2.45 being stored and then getting
>> 2.4500000001 back out. I'd like an uncomplicated way of avoiding this,
>> _and_ avoiding math/rounding errors when dealing with decimals.
>>
>> Any help?
>
> You'd probably want to use some for of binary coded decimal. It stores
> the exact number rather than a decimal approximation.

Or fixed point.

-- 
-eben    ebQenW1@EtaRmpTabYayU.rIr.OcoPm    home.tampabay.rr.com/hactar
PISCES:  Try to avoid any Virgos or Leos with the Ebola virus.
You are the Lord of the Dance, no matter what those idiots at
work say.  -- Weird Al, _Your Horoscope for Today_
-----------------------------------------------------------------------
This list is provided as an unmoderated internet service by Networked
Knowledge Systems (NKS).  Views and opinions expressed in messages
posted are those of the author and do not necessarily reflect the
official policy or position of NKS or any of its employees.



This archive was generated by hypermail 2.1.3 : Fri Aug 01 2014 - 18:39:57 EDT