Search engine spider identification
The following is a basic listing of search engine spider names and their "owners". This is by no means complete, as there are many thousands of search engines on the Internet, but it covers the more common beneficial spiders. Look for these in your traffic reports or search for the names through your server logs to discover which pages they have been spidering. You'll find that many of the entries will also have accompanying numbers or letters e.g Googlebot/2.1 or Slurp.so/1.0
Spider name-- Spider owner
Googlebot ---Google.com
MSNbot ---Search.msn.com
Ask Jeeves/Teoma --Ask.com
Architext spider --Excite.com
FAST-WebCrawler --FAST (AllTheWeb.com)
Slurp ---Inktomi.com
Yahoo Slurp ---Yahoo Web Search
ia_archiver --Alexa.com
Scooter ---AltaVista.com
crawler@fast --FAST (AllTheWeb.com)
Crawler ---Crawler.de
InfoSeek sidewinder --InfoSeek.com
Lycos_Spider_(T-Rex)--- Lycos.com
No comments:
Post a Comment