The Axandra newsletter archive - 5 October 2004
Welcome to the latest issue of the Search Engine Facts newsletter.

This week, we're trying to find out if Google has sent out spiders to find spammers.

In the news: Yahoo releases new search tools, MSN Search launches a new beta version, there's a new article about Google's web browser plans and more.

Table of contents:

We hope that you enjoy this newsletter and that it helps you to get more out of your web site. Please pass this newsletter on to your friends.

Best regards,
Andre Voget, Johannes Selbach, Axandra CEO

1. Facts of the week: Google's new web page spider
Search engines use automated software programs that crawl the web. These programs called "crawlers" or "spiders" go from link to link and store the text and the keywords from the pages in a database. "Googlebot" is the name of Google's spider software.

Many webmasters have noticed that there are now two different Google spiders that index their web pages. At least one of them is performing a complete site scan:

    The normal Google spider: - "GET /robots.txt HTTP/1.0" 404 1227 "-" "Googlebot/2.1 (+"

    The additional Google spider: - "GET / HTTP/1.1" 200 38358 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +"

What is the difference between these two Google spiders?

    The new Google spider uses a slightly different user agent: "Mozilla/5.0 (compatible; Googlebot/2.1; +".

    This means that Googlebot now also accepts the HTTP 1.1 protocol. The new spider might be able to understand more content formats, including compressed HTML.

Why does Google do this?

    Google hasn't revealed the reason for it yet. There are two main theories:

    The first theory is that Google uses the new spider to spot web sites that use cloaking, JavaScript redirects and other dubious web site optimization techniques. As the new spider seems to be more powerful than the old spider, this sounds plausible.

    The second theory is that Google's extensive crawling might be a panic reaction because the index needs to be rebuilt from the ground up in a short time period. The reason for this might be that the old index contains too many spam pages.

What does this mean to your web site?

    If you use questionable techniques such as cloaking or JavaScript redirects, you might get into trouble. If Google really uses the new spider to detect spamming web sites, it's likely that these sites will be banned from the index.

    To obtain long-term results on search engines, it's better to use ethical search engine optimization methods. General information about Google's web page spider can be found here.

It's likely that the new spider announces a major Google update. We'll have to see what this means in detail.

Back to table of contents - Visit

2. Search engine news of the week

Personal Search: My Yahoo! Search

    "My Yahoo! Search addresses a whole host of everyday search problems. If you've ever experienced the tedium of searching again for something you found before but can't quite remember, if you've ever been annoyed by search results you don't like but show up again and again, if you ever wanted to share something you found in search but were forced to cut-and-paste... give My Yahoo! Search a shot."

A new MSN Search tech preview is ready for testing

    "What do we have under the hood? Our new algorithmic search engine with a few more features and a bigger index than last time you visited this site. We used your comments to make it better so give it a spin."

MSN forms search focus group

    "Microsoft's Internet division has invited dozens of Webloggers, researchers and others to its Redmond, Wash., campus next week for an event called 'Search Champs.' They are scheduled to hear about and test upcoming features for MSN Search and to meet researchers from Microsoft Research, according to several people invited to participate."

Alexa Web Information Service (Beta)

    "The Alexa Web Information Service (AWIS) provides developers with programmatic access to the information Alexa Internet collects from its Web Crawl, which currently encompasses more than 100 terabytes of data from over 4 billion Web pages. "

    Editor's note: and Amazon's other search engine currently use Google's index. This beta service might be a first indicator that Amazon will be switching to its own search technology.

Blogger found leaves Google

    "When I started at Google, I knew I was giving up my independence and knew I probably wouldn't like that eventually. So I promised myself I'd stay at least a year. I stayed for a year and eight months and have had a fun, fascinating, and extremely educational time. I'm honored to have been a part of Google for such a historic period."

Search engine newslets

3. Articles of the week

Microsoft researcher questions search engine business model

    "One Microsoft technologist said web search technology will be a big money maker for Google and others in the short term but he predicted the party won't last long."

Google should be in the browser biz

    "The move would be key in helping the outfit expand beyond search, tie its many offerings together, and hold off Microsoft."

Google Search: best-laid plans

    "All five gave Google high ratings, even though it was a far more expensive stock than at the IPO just six weeks earlier, having risen almost 40 percent. Why any investor took these opinions seriously is a mystery to me—but the stock soared around 8 percent. It's as if people have forgotten the whole unseemly spectacle of analysts' hyping stocks their firms helped underwrite. Does anything ever change? Maybe not."

Google challenge: keeping wealthy workers

    "Based on Google's share price of around $130, senior vice president Bill Coleman estimated that around 60 percent of the 1,900 Google employees with the company at the end of March each now hold stock options worth at least $1 million."

Are you Google employee material?

    "A new aptitude test Google is circulating purports to find the best and brightest. But is it really a brilliant ad campaign?"

Back to table of contents - Visit

4. Recommended resources

Another 5 star rating for IBP

    Our popular web site promotion tool IBP has received another 5 star rating. This time it's from

    "IBP is the multi award winning web site promotion tool that helps you to get more revenue with high search engine rankings in Google, Yahoo and all other major search engines.

    The suite of 10 professional website promotion tools includes a search engine submission tool, a search engine ranking checker, a top 10 optimizer and a keyword generator. IBP creates professional reports in PDF, HTML and Microsoft Word format."

    5 out of 5 -

Are you more popular than your competitors?

    How does your web site compare to your competitors? Does your web site have more incoming links? How much traffic do your competitors get?

    The new version 3.0 of the freeware tool Link Popularity Check helps you to get the answers quickly and easily. Click here to download your free copy.

Want to be mentioned in this newsletter?

    Just send us some words about your successes with IBP or ARELIS and you might get featured in this newsletter along with your web site address.

Back to table of contents - Visit

5. Previous articles

Back to table of contents - Visit