AdSense Speeds Up Search Engine Crawling & Indexing

Since this is a new site it occurred to me to put AdSense ads on it as I do with most of my sites. When Google first started AdSense I had been in the middle of starting up some other sites which I put AdSense on within the first week of launching. One week from that point I found I had been indexed on Google. I will be testing this theory again but now with RobMalon.com. I will report my findings in a later post but for now I did some research on the subject.

There are actually two bots that google uses. A MediaBot and a GoogleBot each with their respective purpose and own database. Google claims there isn’t a link between them and this has always been topic of suspicion. While this could be true to an extent I still see an advantage in having AdSense on your page for indexing. This is based upon an announcement by Google’s Matt Cutts. At the Boston PubCon in 2006 he verified MediaBot pages can also be used for Google’s main index:

“Pages with AdSense will not be indexed more frequently. It’s literally just a crawl cache, so if e.g. our news crawl fetched a page and then Googlebot wanted the same page, we’d retrieve the page from the crawl cache. But there’s no boost at all in rankings if you’re in AdSense or Google News. You don’t get any more pages crawled either.”

Technically this means GoogleBot wont grab your information any sooner then it would have if it truly is separate. But why stuff your mouth with more food when you already have something to chew on. Its more practical for Google to go after things they already have in their possession. In either case here is the worse case scenario which still points to early implementation of AdSense being beneficial:

Faster indexing - Once Google realizes your site exists it does not need to crawl it. It will just grab the cache copy that MediaBot uses. Thus proving the two databases do interact.
Faster updating - Once GoogleBot does grab MediaBots content data (assuming it took a while) it will likely see that the content is outdated depending on the frequency of MediaBots updates. It could think “Hey lets go check this site out again for an update”. Thus reindexing your page and probably digging a bit further into your site for more pages of content to index.

Having your data (cached pages) exist somewhere within Googles own databases allows a high probability that by including AdSense on your page you will be spidered/indexed sooner. Whether that means 1 day or 3 weeks sooner is still up to speculation. Regardless its still pretty harmless to ad even Googles link ads to the the bottom of your page and well worth it if not just for a bit of revenue. Test it out yourself and let us know your results in the comments.

Helpful hint: If your site doesn’t already have a date display add one to it. Then when you find your results in Google you can click the “cached” link under the link result and view this date to know EXACTLY when the snapshot of your site was taken.