We’ve heard it many times before: Google loves fast loading websites. Towards the end of this past June, I completed a move of my site Wantbox from a Dreamhost shared server to a dedicated Linode virtual server.
I really like Dreamhost’s hosting service, their nice dashboard and great customer service, but Wantbox had grown beyond shared hosting. I needed an environment that I could fully control and could no longer tolerate the downtime caused by other sites on my shared server.
I chose to move to Linode because of the competitive price, the positive experience from some friends and the great Django/Apache/MySQL StackScript setup script by Filip.
Perceptually, I noticed that the pages seemed to be loading faster, and confirmed an approximate 35% page load improvement using WebPageTest.org.
I didn’t pay much attention to GoogleBot’s reaction to this faster site hosting environment until I happened to check out Wantbox’s crawl stats today:
Not too shabby: an immediate drop in the time Googlebot spent downloading pages resulting in more pages crawled per day.
In real numbers (approximated based on the above graphs), I saw an 83% drop in the time to download a Wantbox page which has resulted in a 200% increase in pages crawled per day. Booya!
ABOUT WANTBOX:
Wantbox is a place to post all of the things that you want and have your friends, neighbors and local merchants help you find them. Learn from other people’s shopping experiences to help you make more informed decisions. On Wantbox you can find someone to design a bathroom, install dog electric fencing or learn everything you need to find a domestic house cleaning service.
UPDATE:
Tommy asked a good question in the comments, namely, have my keyword rankings improved as a result of this speed improvement. After checking, over half of them have indeed improved while only 14% have degraded. The rest either have not changed or still do not rank.
4 Responses to “Reducing Page Load Times Dramatically Increased my Googlebot Crawl Rate”
Interesting article. Any news on Google ranking improvements as a result?
Thanks Tommy. I actually hadn’t checked on that until now. I track 74 bellweather keywords for the site. Since the end of June (when I made the move) 49% of them are up, 14% are down, 14% are unchanged and 24% still are not ranked. So, yeah, I’d say it has had an big improvement on ranking as well!
Very useful! Thanks for the update, appreciate it.
To put some specific figures on it, any reduction in speed down to about 400ms seems to be worth doing and does indeed seem to increase the crawl rate. I’ve recently been trying to reduce page download time and my site’s average crawl rate is currently at 344 ms. I think the only options left to me now are to buy a beefier server or get a wider network pipe. Throughout this improvement period, from an initial 700 or 800 ms, the crawl rate has steadily increased but now seems to have hit a plateau. Anything below 400ms hasn’t made much difference. Perhaps a step jump is required now.