[SLUG] Re: slug] Math Precision

From: R P Herrold (herrold@owlriver.com)
Date: Sun Feb 06 2005 - 09:16:44 EST


On Sun, 6 Feb 2005, Paul M Foster wrote:

> I'm using SQLite for some payroll database work, and seeing a lot of
> rounding errors, or what are really errors caused by trying to store

> Any help?

Not that you are going to like. sqlite does not use SQL'92
typeing data -- it handles type conversions as it deems
appropriate, as part of its approach to simplicity. It is
part of its design.

As a workaround, working in pennies, always truncating (using
int() to force all math results to integers), and then only
'punctuating' those pennies in $zzz.yy form (although
maintaned internally as 'zzzyy') will work.

I first hit this issue back in '74, as I was doing the
computer work, dividing a legal setlement between a couple
thousand bricklayers, based on hours worked and wage rate
seniority rules. We had to go to truncating off fractional
cents, and getting the other party to concur on how we handled
the 'breakage'; The movie 'Office Space' ['... excuse me,
but you seem to have my stapler...', mumbled] also dealt with
the topic.

-- Russ Herrold
-----------------------------------------------------------------------
This list is provided as an unmoderated internet service by Networked
Knowledge Systems (NKS). Views and opinions expressed in messages
posted are those of the author and do not necessarily reflect the
official policy or position of NKS or any of its employees.



This archive was generated by hypermail 2.1.3 : Fri Aug 01 2014 - 18:39:19 EDT