Eben King wrote:
> I got a dual-core CPU (etc) a month or so ago. I need to rewrite some
> of my scripts to take advantage of the second CPU. Is there a
> relatively easy way of doing that? I've thought about:
>
> - generating a Makefile on the fly, and then calling "make -j
> 2". This requires learning enough about Makefiles to twist it to do my
> nefarious deeds. I know very little of Makefiles, so the learning
> curve
> is a definite downside.
It's easier than you think. Really. Most of my automation is Makefile
based.
That, or a bash script with "set -e" to error out on a non-zero result
from a scripted command (the same behavior as a Makefile), though you
need to if-then nest everything and dependencies don't work themselves out.
Makefiles really are ideal for most batch based automation. Dependency
resolution is a wonderful thing.
Speaking of Makefiles, has anyone run across a dot creator from parsed
Makefile dependencies? I'd love to have a way to visualize things
automatically.
> - "xargs -n 2"; what it'll do is run pairs at a time, and if one of
> the two
> takes longer than the other, tough noogies; it won't run the next pair
> until _both_ current processes exit. So it's better than serial
> execution, but not optimal.
xargs -P 2, will run 2 in parallel at the same time. If one ends,
another is spawned in its place.
> - all get backgrounded. This works, but uses a _lot_ of RAM, maybe more
> than I have, which will make it swap and maybe take longer than if I
> had
> run one after the other.
>
> So what's a dual-core-havin' scripter to do?
xargs -P is a godsend for parallelizing tasks.
There are many other job scheduling systems out there, particularly in
the distributed/GRID computing space (Condor, etc etc). That is, if
you're working on creating a complex batch scheduling platform.
This question (and answer) seems to pop up 3 or 4 times a year anymore.
- Ian C. Blenke <ian@blenke.com>
-----------------------------------------------------------------------
This list is provided as an unmoderated internet service by Networked
Knowledge Systems (NKS). Views and opinions expressed in messages
posted are those of the author and do not necessarily reflect the
official policy or position of NKS or any of its employees.
This archive was generated by hypermail 2.1.3 : Fri Aug 01 2014 - 16:32:09 EDT