Re: [SLUG] perl [pig] duplicate removal with a twist

From: Kwan Lowe (kwan@digitalhermit.com)
Date: Mon Jul 24 2006 - 17:31:20 EDT


> I'm writing a perl program that takes input from one file, processes it
> (line by line), and then
> puts in into another text file. I'm trying to figure out how I can
> search the second text file to see
> if a particular part that line (at the beginning of the line) has
> already been put into the file, and if
> it has, don't put it in.
>
> What I'm having trouble on is how to search the output file in perl for
> that text before writing to it.
> -any ideas?

The best way IMHO is to hash the second file. Then you can just do a hash comparison
to check, rather than relatively expensive regex's.

-- 
* The Digital Hermit   http://www.digitalhermit.com
* Unix and Linux Solutions   kwan@digitalhermit.com
-----------------------------------------------------------------------
This list is provided as an unmoderated internet service by Networked
Knowledge Systems (NKS).  Views and opinions expressed in messages
posted are those of the author and do not necessarily reflect the
official policy or position of NKS or any of its employees.



This archive was generated by hypermail 2.1.3 : Fri Aug 01 2014 - 15:02:38 EDT