Yahoo’s Recent Spider Improvement Beats Google’s
Yahoo!’s Search Blog announced yesterday that they were making some final changes to their spider, (named “Slurp”), standardizing their crawlers to provide a common DNS signature for identification/authorization purposes.
Previously, Slurp’s requests may have come from IP addresses associated with inktomisearch.com, and now they should all come from IPs associated with domains in this standard syntax:
[something].crawl.yahoo.net
Possible Related Posts
Posted by Chris of Silvery on 06/06/2007
Permalink | | 
Print
| Trackback | Comments Off on Yahoo’s Recent Spider Improvement Beats Google’s | Comments RSS
Filed under: Google, Spiders, Yahoo bot-detection, bots, Googlebot, slurp, spidering, Spiders, user-agents
AdSense Spider Cross-Pollinates for Google
A few bloggers such as Jenstar have just posted that pages spidered by Google’s AdSense bot are appearing in Google’s regular search results pages. Shoemoney just blogged that Matt Cutts has officially verified that this is happening, saying that this was done so that they wouldn’t have to spider the same content twice, and that Google did this as part of their recent Big Daddy infrastructure improvements.
This has a couple of interesting ramifications for SEO professionals and those of us who are optimizing our sites for Google, since bot detection systems may now need to be updated and since this may essentially be a new way of committing site/page submissions into Google’s indices. And we all thought automated URL submissions were dead! I’ll explain further…. (more…)
Possible Related Posts
Posted by Chris of Silvery on 04/19/2006
Permalink | | Print
| Trackback | Comments Off on AdSense Spider Cross-Pollinates for Google | Comments RSS
Filed under: Google, Spiders AdSense, bots, Googlebot, Robots.txt, Spiders, URL-submission
