On Mon, 06 Dec 2004 11:44:39 -0500, Bryan J. Smith <b.j.smith@ieee.org> wrote:
> On Mon, 2004-12-06 at 11:09, Chuck Hast wrote:
> > First I know nil about DB stuff. I am trying to learn though.
>
> I defer to formal DBAs regularly. I'm mainly a platform guy. I know
> how to write SQL like I know how to write Perl. It's an end to a means
> and there are far better gurus who could do it 4x faster and better than
> I.
>
> > Here is the scenario: ... cut ...
> > So based on the above info (probably not enough but hope it sheds
> > light on what direction I need to go) what db should I look at using for
> > this application. The data needs to get in there fast enough so the
> > mappers can update the view without the vehicle having gone too far
> > down the road.
>
> Memory. ;->
>
> In reality, don't think you need a formal DB. Sometimes memory
> suffices. And then dumb it to a file when the capture is complete.
> Of course, that file might get imported into a formal DB for reading.
The present application writes the data to a flat file. The guy who does
the app is going re-work the app so that it can write directly to a db
using ODBC or probably JDBC, as he is looking at re-writing in Java
so that we are not chained to the windows environment.
I will have a ODBC or JDBC connection to a db, that will populate that
db as the drive test progresses, at the same time the mappers will be
looking at the data in the db, and will map the data that we have selected
as critital, examples are that are RSSI, BER/BERT, RTT, for CDMA
systems we have to add Ec/Io, FFER and the call setup time/error codes
if a call fails. We already have all our tools and whatnot, and in the past
we have been importing the flat file into various db after the capture.
We want to have the app just dump it directly into the db and be done with
it, at the same time while driving down the road we will have one or more
mapping machines to use to look at the data, there are preset levels that
will either show a obnoxious color or a strange symbol when some level or
other data is off.
>
> The solution would vary on many details.
>
> There is one thing I wish to note however:
> > At the same time there will be at least 1 machine looking at the db
> > and mapping the data as it comes in, that is so the person monitoring
> > the data can stop the drive test and go back and check a questionable
> > area if needed. There may be other mappers looking at the db as the
> > data is being captured.
>
> How much "analysis" do you need to do in "real-time"? That's the crux
> right there. In a nutshell, there are a _lot_ of memory-only DBs out
> there. Some even take SQL commands.
Perhaps it would be good for me to send you a sample db and some
map images so you can see what we are doing.
>
> > I want this to be a Linux machine obviously.
> > At the end of the data the db will be copied to another machine for
> > other people to map all of the data the next day while the drive testing
> > continues.
>
> Sounds like much of what I was exposed to in aerospace -- data
> acquisition (DAQ). You grab and write in real-time, provides some basic
> info (for real-time tracking), and then you parse it more completely
> _off-line_.
>
> In my case, it was telemetry. Capture all of it, provide some rough
> IRIG time -- launch, staging, terminal, intercept. Then go back later,
> off-line, and parse for readings from thermocouplings and other sensors.
>
> Because while the "cool" part is watching a TMD system turn millions of
> public dollars into scrap with pure kinetic energy, the "fun" part is to
> see where it actually hit and if the payload could be considered
> "destroyed" from further analysis.
Yes, that must be fun to watch, any mass destruction is fun to watch.
Imploded buildings, vehicles, of course as long as there is no loss of
life involved.
>
> There's far more to the entire DAQ-to-analysis solution than just the DB
> components. I've have to understand the the interfaces, etc...
Basicly the data will enter via odbc/jdbc and exit the same way. We
pull the data into the mapping tools and create nice maps of the different
data fields. Right now we are doing it with Access and Oracle, on Windows
as soon as my cohort in crime re-does the app to where it uses the odbc
drivers we will deposit it directly to a db. I am just trying to pick the right
one from the start for the mobile piece, once we get it back to the office it
will probably be exported to something else depending on what the customer
wants it in. But internally I want it captured to a server in the
vehicle so that
a. It can be viewed by mappers while we are doing the testing
b. It can be exported out to a machine out on our network somewhere so the
data can be mapped (it is really already processed once in the db)
for delivery
to the customer.
c. Want to use Linux as I feel it is a much more stable environment than windows
I have had windows do too many strange things while doing the capture.
have even had viruses come down the pipe, and we had all of the virus stuff
on the machines.
I hope this is clear (probably like mud) most of the analysis is done
on the maps
which are linked to the db. The area I am worried about is in the vehicle once I
pass it off to the fixed side at the end of the day they will put it
on a machine and
link the mappers to it, they will then map each important field of
data, then we look
at it, but my immediate worry is the vehicular db. I have a fixed
format, and the
in and out points are odbc type devices. I am just wanting to
establish the db that
I should use in that mobile server.
>
> But given the fact that you are writing at such a massive rate, if you
> _do_ end up going directly to DB during the DAQ, then I'd lean towards
> PostgreSQL. But I'm really "jumping ahead" there -- you may need to
> have a distributed DAQ setup that buffers in memory, then commits to the
> DB back-end.
>
> Again, the entire DAQ-to-analysis solution is more than just the DB.
> ;->
>
Yes, but I will be handing the db off to others who will then put it in whatever
they want.
-- Chuck Hast To paraphrase my flight instructor; "the only dumb question is the one you DID NOT ask resulting in my going out and having to identify your bits and pieces in the midst of torn and twisted metal." ----------------------------------------------------------------------- This list is provided as an unmoderated internet service by Networked Knowledge Systems (NKS). Views and opinions expressed in messages posted are those of the author and do not necessarily reflect the official policy or position of NKS or any of its employees.
This archive was generated by hypermail 2.1.3 : Fri Aug 01 2014 - 20:33:49 EDT