[SLUG] Invasive botageq

From: Logan Tygart (thelogan@allyourbase-arebelongto.us)
Date: Tue Jun 15 2004 - 20:31:33 EDT

Fellow sysadmins,

I have a server farm dishing out about 100 domains' webpages using
LAMP. Have you noticed since Google announce gmail, the other search
engine bots (in particular msnbot and yahoo slurp), have become
inordinately intrusive? They obey robots.txt, but they now hit sites as
if they were visiting as opposed to just gathering relevant info e.g.
using ?=phpsession yada yada etc.

Possibly I shouldn't scrutinize my logs so hard, but I have just noticed
a change in behavior over the last few months.

Just curious,
The Logan 

-- 
20:20:01 up 115 days, 13:42, 5 users, load average: 0.19, 0.23, 0.35
Mynd you, mse bites Kan be pretty nasti. -- Monty Python
ICQ: 72101412 AIM/MSN/Yahoo/Jabber: logantheclever
Registered Linux User: 277727
----------------------------------------------------------------------- This list is provided as an unmoderated internet service by Networked Knowledge Systems (NKS). Views and opinions expressed in messages posted are those of the author and do not necessarily reflect the official policy or position of NKS or any of its employees.

This archive was generated by hypermail 2.1.3 : Fri Aug 01 2014 - 17:43:46 EDT