Spiders?

MaxForce · 10-12-2005, 07:28 PM

I have just made a quick check on Who's Online (Under Quick Links) and I noticed that there are 21 Yahoo Slurp Spiders.
I was wondering, does this mean that the forum will be indexed much faster compared to one that has only say, 4 Yahoo Slurp Spiders?
If so, then how do we get more of this "spiders" to come?

***Brian Turner*** · 10-12-2005, 07:42 PM

The Yahoo! spiders are basically just spidering the site one URL at a time - so now you've started this topic, one of the slurps spiders will notice this new URL - then at some future point, another Yahoo! slurp spider will come along and try to index the content.

Yahoo! used to have problems indexing deep content, but it's much better now.

MaxForce · 10-12-2005, 07:46 PM

So there more they are, the better?

***Brian Turner*** · 10-12-2005, 08:22 PM

Well...the more there are, then the more often you're being indexed by Yahoo!. Unfortunately, spidering isn't necessarily proportional to ranking - though of course, it's an important precursor.

Interestingly enough, I'm reading three different sets of IP's as carrying the Yahoo! slurp user-agent:

202.160.xx.xx
68.142.xx.xx
66.196.xx.xx

I'll have to keep my eye out for further information on that.

MaxForce · 11-12-2005, 10:43 AM

And the impact of more indexing by Yahoo! brings to ?

vpssupport · 11-12-2005, 11:21 AM

Spiders are looking for fresh content on your site, the way I understand it is the more a spider crawls your site and the more up to date content you have = more chance of the content being listed.
On one of my sites yahoo took a long time to crawl the whole site but eventually we came up really well with our chosen keywords/phrase.
Although on another site which happens to be writen differently, the yahoo spider seems to only index the top level.

There is lots of info about this, here's just one:
http://www.marketingsource.com/articles/view/1931

Depending on how you view your whos online, you could setup a comprehensive robots.txt file listing all known spiders, which should help in determining what is a spider and what isn't in your report. You will also be able to allow/disallow the ones you dont want to come.

stoner3221 · 11-12-2005, 03:47 PM

Yahoo can be a very powerful marketing asset if you can get a lot of pages cached by them. One of my websites has over 900,000 pages cached by Yahoo verses a recent large jump to over 390,000 cached pages by Google. Yahoo hits have been out performing or equaling Google hits and has generated a great deal of traffic for this website. Unfortunately Yahoo has a bad habit of indexing pages tagged not to be indexed or pages without content. Google seems to be far more efficient in not indexing low or no content pages.
I think a lot of webmasters are dropping the ball by concentrating so much on Google when Yahoo can be such a good traffic source as well.

MaxForce · 11-12-2005, 08:31 PM

Alright, THANKS!!!

Finally got it... hehe I AM kind of slow...

10-12-2005, 07:28 PM	#1 (permalink)
MaxForce Senior Member Join Date: Oct 2005 Location: Malaysia Posts: 147	Spiders? I have just made a quick check on Who's Online (Under Quick Links) and I noticed that there are 21 Yahoo Slurp Spiders. I was wondering, does this mean that the forum will be indexed much faster compared to one that has only say, 4 Yahoo Slurp Spiders? If so, then how do we get more of this "spiders" to come? __________________ "The reasonable man adapts to the world, The unreasonable man adapts the world to himself, Therefore, all progress depends on the unreasonable man." Visit my newly launched blog: A World Class Mentality

10-12-2005, 07:42 PM	#2 (permalink)
*Brian Turner* Business Guru Join Date: Dec 2003 Location: Near Inverness, Highlands, Scotland Posts: 7,719	Re: Spiders? The Yahoo! spiders are basically just spidering the site one URL at a time - so now you've started this topic, one of the slurps spiders will notice this new URL - then at some future point, another Yahoo! slurp spider will come along and try to index the content. Yahoo! used to have problems indexing deep content, but it's much better now. __________________ SEO specialist

10-12-2005, 07:46 PM	#3 (permalink)
MaxForce Senior Member Join Date: Oct 2005 Location: Malaysia Posts: 147	Re: Spiders? So there more they are, the better? __________________ "The reasonable man adapts to the world, The unreasonable man adapts the world to himself, Therefore, all progress depends on the unreasonable man." Visit my newly launched blog: A World Class Mentality

10-12-2005, 08:22 PM	#4 (permalink)
*Brian Turner* Business Guru Join Date: Dec 2003 Location: Near Inverness, Highlands, Scotland Posts: 7,719	Re: Spiders? Well...the more there are, then the more often you're being indexed by Yahoo!. Unfortunately, spidering isn't necessarily proportional to ranking - though of course, it's an important precursor. Interestingly enough, I'm reading three different sets of IP's as carrying the Yahoo! slurp user-agent: 202.160.xx.xx 68.142.xx.xx 66.196.xx.xx I'll have to keep my eye out for further information on that. __________________ SEO specialist

11-12-2005, 10:43 AM	#5 (permalink)
MaxForce Senior Member Join Date: Oct 2005 Location: Malaysia Posts: 147	Re: Spiders? And the impact of more indexing by Yahoo! brings to ? __________________ "The reasonable man adapts to the world, The unreasonable man adapts the world to himself, Therefore, all progress depends on the unreasonable man." Visit my newly launched blog: A World Class Mentality

11-12-2005, 11:21 AM	#6 (permalink)
vpssupport Senior Member Join Date: Mar 2005 Location: Bedfordshire Posts: 155	Re: Spiders? Spiders are looking for fresh content on your site, the way I understand it is the more a spider crawls your site and the more up to date content you have = more chance of the content being listed. On one of my sites yahoo took a long time to crawl the whole site but eventually we came up really well with our chosen keywords/phrase. Although on another site which happens to be writen differently, the yahoo spider seems to only index the top level. There is lots of info about this, here's just one: http://www.marketingsource.com/articles/view/1931 Depending on how you view your whos online, you could setup a comprehensive robots.txt file listing all known spiders, which should help in determining what is a spider and what isn't in your report. You will also be able to allow/disallow the ones you dont want to come. __________________ Regards, Clive Rutherford www.vitalproductsolutions.co.uk

11-12-2005, 03:47 PM	#7 (permalink)
stoner3221 Member Join Date: Jul 2004 Location: South Glens Falls, NY USA Posts: 74	Re: Spiders? Yahoo can be a very powerful marketing asset if you can get a lot of pages cached by them. One of my websites has over 900,000 pages cached by Yahoo verses a recent large jump to over 390,000 cached pages by Google. Yahoo hits have been out performing or equaling Google hits and has generated a great deal of traffic for this website. Unfortunately Yahoo has a bad habit of indexing pages tagged not to be indexed or pages without content. Google seems to be far more efficient in not indexing low or no content pages. I think a lot of webmasters are dropping the ball by concentrating so much on Google when Yahoo can be such a good traffic source as well. __________________ WoW Directory & WoW Yellow Pages Business Directory

11-12-2005, 08:31 PM	#8 (permalink)
MaxForce Senior Member Join Date: Oct 2005 Location: Malaysia Posts: 147	Re: Spiders? Alright, THANKS!!! Finally got it... hehe I AM kind of slow... __________________ "The reasonable man adapts to the world, The unreasonable man adapts the world to himself, Therefore, all progress depends on the unreasonable man." Visit my newly launched blog: A World Class Mentality

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode