Reducing Page Load Times Dramatically Increased my Googlebot Crawl Rate

We’ve heard it many times before: Google loves fast loading websites. Towards the end of this past June, I completed a move of my site Wantbox from a Dreamhost shared server to a dedicated Linode virtual server.

I really like Dreamhost’s hosting service, their nice dashboard and great customer service, but Wantbox had grown beyond shared hosting. I needed an environment that I could fully control and could no longer tolerate the downtime caused by other sites on my shared server.

I chose to move to Linode because of the competitive price, the positive experience from some friends and the great Django/Apache/MySQL StackScript setup script by Filip.

Perceptually, I noticed that the pages seemed to be loading faster, and confirmed an approximate 35% page load improvement using

I didn’t pay much attention to GoogleBot’s reaction to this faster site hosting environment until I happened to check out Wantbox’s crawl stats today:

Not too shabby: an immediate drop in the time Googlebot spent downloading pages resulting in more pages crawled per day.

In real numbers (approximated based on the above graphs),  I saw an 83% drop in the time to download a Wantbox page which has resulted in a 200% increase in pages crawled per day. Booya!


Wantbox is a place to post all of the things that you want and have your friends, neighbors and local merchants help you find them. Learn from other people’s shopping experiences to help you make more informed decisions. On Wantbox you can find someone to design a bathroom, install dog electric fencing or learn everything you need to find a domestic house cleaning service.


Tommy asked a good question in the comments, namely, have my keyword rankings improved as a result of this speed improvement. After checking, over half of them have indeed improved while only 14% have degraded. The rest either have not changed or still do not rank.