July 9th, 2005

Googlebot broken?

While my server frequently sees a decent number of hits from the googlebot crawlers, they have always been very good about spreading out the hits so that they don't put a large load on the server or network.

So imagine my surprise to find that for the last three hours straight, a single googlebot crawler at crawl-66-249-65-166.googlebot.com has constantly had multiple connections open to my web server continuously downloading file after file with no pauses, using up all my bandwidth. I had to do something I never thought I would have to, and had to put google IPs into the firewall.

Anyone else getting pounded by googlebot today?