On Mon, 2004-12-06 at 11:09, Chuck Hast wrote:
> First I know nil about DB stuff. I am trying to learn though.
I defer to formal DBAs regularly. I'm mainly a platform guy. I know
how to write SQL like I know how to write Perl. It's an end to a means
and there are far better gurus who could do it 4x faster and better than
I.
> Here is the scenario: ... cut ...
> So based on the above info (probably not enough but hope it sheds
> light on what direction I need to go) what db should I look at using for
> this application. The data needs to get in there fast enough so the
> mappers can update the view without the vehicle having gone too far
> down the road.
Memory. ;->
In reality, don't think you need a formal DB. Sometimes memory
suffices. And then dumb it to a file when the capture is complete.
Of course, that file might get imported into a formal DB for reading.
The solution would vary on many details.
There is one thing I wish to note however:
> At the same time there will be at least 1 machine looking at the db
> and mapping the data as it comes in, that is so the person monitoring
> the data can stop the drive test and go back and check a questionable
> area if needed. There may be other mappers looking at the db as the
> data is being captured.
How much "analysis" do you need to do in "real-time"? That's the crux
right there. In a nutshell, there are a _lot_ of memory-only DBs out
there. Some even take SQL commands.
> I want this to be a Linux machine obviously.
> At the end of the data the db will be copied to another machine for
> other people to map all of the data the next day while the drive testing
> continues.
Sounds like much of what I was exposed to in aerospace -- data
acquisition (DAQ). You grab and write in real-time, provides some basic
info (for real-time tracking), and then you parse it more completely
_off-line_.
In my case, it was telemetry. Capture all of it, provide some rough
IRIG time -- launch, staging, terminal, intercept. Then go back later,
off-line, and parse for readings from thermocouplings and other sensors.
Because while the "cool" part is watching a TMD system turn millions of
public dollars into scrap with pure kinetic energy, the "fun" part is to
see where it actually hit and if the payload could be considered
"destroyed" from further analysis.
There's far more to the entire DAQ-to-analysis solution than just the DB
components. I've have to understand the the interfaces, etc...
But given the fact that you are writing at such a massive rate, if you
_do_ end up going directly to DB during the DAQ, then I'd lean towards
PostgreSQL. But I'm really "jumping ahead" there -- you may need to
have a distributed DAQ setup that buffers in memory, then commits to the
DB back-end.
Again, the entire DAQ-to-analysis solution is more than just the DB.
;->
-- Bryan J. Smith b.j.smith@ieee.org ------------------------------------------------------------------ Beware of advocates who justify their preference not in terms of what they like about their "choice," but what they did not like about another option. Such advocacy is more hurtful than helpful.----------------------------------------------------------------------- This list is provided as an unmoderated internet service by Networked Knowledge Systems (NKS). Views and opinions expressed in messages posted are those of the author and do not necessarily reflect the official policy or position of NKS or any of its employees.
This archive was generated by hypermail 2.1.3 : Fri Aug 01 2014 - 20:28:13 EDT