Google Blocks SERP Tracker for Bad Behavior

First, the bad news....

If you're a frequent user of Sitening's SERP Tracker, you may have noticed a lack of results over the past week. It looks like the sleeping giant has finally noticed us. If you've ever managed to get blocked, you may recognize this message:

We're sorry...

... but your query looks similar to automated requests from a computer
virus or spyware application. To protect our users, we can't process your
request right now.

Fan-freakin-tastic.

SEO Analyzer and Page Rank Checker were not affected. Backlink Analyzer is now working completely, we switched around some of the inner workings and started relaying all Google traffic through proxy servers. SERP Tracker has been disabled for a few days while we figured out what was going on. As it turns out, our scraper may not have been very polite with Google, and we have addressed that issue. I just reactivated the SERP Tracker scheduler, but it will only be working with Yahoo and Microsoft until I get some more proxy resources in place.

Now some good news!!!

Sitening SEO Tools are really popular! So popular, in fact, that we need more servers! We've actually been anticipating this day for months, and with the faltering Google API program and the consistent growth in SERP Tracker volume we've been designing a new system. Two words: grid computing. Stick that in your firewall and smoke it, Google.

Since the great Google blockage of July 2007, we've started running a new scraper inside our grid computing environment that is performing even better than expected. We've got the capability now of spreading our requests out in a way that will be much more polite, and will make it extremely difficult for the search engines to detect or disable (think "moving target"). For testing we've been tracking every keyword you've ever thrown at us on a daily basis. The results from the new crawler will be stored indefinitely, along with the entire HTML results in case we need to recrawl the page for any reason.

For existing SERP Tracker users, you'll continue to see Yahoo and Microsoft results as before. In the upcoming weeks we plan on moving the Google results seamlessly over to the new system, at which point you'll start seeing Google results again with just a short gap in data. After that, we'll be launching a completely new set of SEO Tools which will utilize our new technology and build off our experience from the original SEO Tools.

Thank you for you patience, and please keep sending us your comments and feedback. We appreciate your continued support.

If you're interested in writing your own scraper, check out Webbots, Spiders, and Screen Scrapers from No Starch Press.