The Axandra newsletter archive

21 May 2002

Hello!

Welcome to the new issue of Weekly Search Engine Facts! Again, we'll draw conclusions from the biggest search engine study ever conducted. In the study, we analyzed more than 103,320 top 10 Web pages to discover why they have top rankings. This weekly newsletter tells you about some of our findings.

This is a plain text file (without HTML code) that allows the Web site administrator to define which parts of the site robots may access and which not.

Rumor has it that some search engine robots do not index Web pages that lack the robots.txt file as they don't know whether it's allowed to access your Web site or not.

In our search engine ranking study, we examined 103,260 top 10 Web pages on Google, AltaVista, iWon/Inktomi, AllTheWeb, Teoma and Wisenut. Here are the results for the robots.txt file:

AllTheWeb: 30.5% have it, 69.5% don't have it
AltaVista: 36.3% have it, 63.7% don't have it
Google: 35.7% have it, 64.3% don't have it
iWon/Inktomi: 32.7% have it, 67.3% don't have it
Teoma: 30.4% have it, 69.6% don't have it
Wisenut: 31.4% have it, 68.6% don't have it

As you can see, the majority of the top 10 Web pages don't have the robots.txt file and they are still indexed.

Sometimes, it's necessary to include a robots.txt file in the root directory of your server. For example, you don't want the search engine robot to index your log files so that anyone can find your logs in the search engines. In addition, you may want to exclude robots from accessing dynamically created pages because of the heavy load for your server.

If you already have a robots.txt file, it's very important that you check its syntax. Here's a very good free tool:

http://www.tardis.ed.ac.uk/~sxw/robots/check/

Martijn Koster, author of the Robots Exclusion Protocol, has compiled information on robots:

http://www.robotstxt.org/wc/robots.html

Listing of robot names (so that you can recognize them in your Web server logs):

http://www.jafsoft.com/searchengines/webbots.html

Source of the search engines percentages above: Search Engine Ranking Studies Q2/2002 .

Reason #3: The submitted Web page is only a redirection.

If the Web page you submit contains a redirection to another Web site, most search engines will skip your Web site completely. Do not submit a redirection Web page.

Many webmasters tried to cheat search engines with redirection pages in the past. The search engines companies discovered that and they decided to totally skip Web pages with redirections.

Submit a real Web page that contains the product description visible to the reader.

Sometimes, you have old Web pages listed on search engines and you want them to redirect to the new Web site. There are several ways to do it:

1. You can implement a server side redirect on the old Web page, using the 301 Moved Permanently error message. This will redirect users to the new Web site, but also tells the search engines that this page has moved permanently. Some search engines will drop the page from their index, and some will eventually replace the old page with the new one without hurting your rankings.

2. You can use the META Refresh tag on the old Web page, for example <META HTTP-EQUIV=Refresh CONTENT="5; URL=http://www.axandra.com"> tells the browser to load www.axandra.com 5 seconds after the current document has finished loading. However, some old Web browsers don't support that tag, and some search engines penalize pages that use a refresh of a few seconds or less (more about this in our search engine ranking report, see http://www.axandra.com/search-engine-studies/index.htm ).

3. Instead of the META Refresh tag, you can also use JavaScript to load a new document:

<SCRIPT LANGUAGE="JavaScript">
<!--
location.replace("http://www.your-new-site.com");
-->
</SCRIPT>

Most search engine robots ignore JavaScript so this method of redirection is unlikely to be penalized.

4. You can also delete the old Web page and create a custom 404 error page. This ensures that visitors will be redirected to the new site if they click on a broken link or enter an incorrect URL. The 404 error page should contain a link to your home page and to the primary sections of your Web site.

To move to a new Web site and to keep your old search engine rankings, I recommend using one of the methods 1 or 3.

* ARELIS - Axandra's Reciprocal Links Solution

ARELIS is the first software program that provides a serious, spam-free reciprocal links solution for professional webmasters. Reciprocal links can get you a lot of targeted traffic and they improve the link popularity of your Web site in the search engines.

Version 3.0 is planned to be released in about 2-3 weeks. Buy ARELIS now and you'll get version 3.0 for free.

Free download:
http://www.Axandra.com/download.htm

* Did you know?

Our two-tier affiliate program pays you 35% + 10% for the sale of our products:
http://www.Axandra.com/affiliates.htm

* Marketing Web sites that we use

http://www.axandra.com/webmaster-resources/index.htm

Copyright © 1997- Axandra GmbH • Internet marketing pioneers since 1997
About us • Customer Support • Code of ethics • Privacy policy • Affiliate program • SEO Software

Free Ranking News

More than 300,000 subscribers. Get weekly search engine ranking news, facts, tips and tricks by email.

Download

Order now

Free content for your own website

You can publish the articles of our newsletters on your own web site as long as you follow some simple rules:

• The articles must not be changed.
• The links in the articles must remain intact.
• You must add the following link below the article:
  Article by Axandra <a href="http://www.ibusinesspromoter.com" target=_blank>SEO software</a>

That's all. Improve your web pages with our free content!