Update as of 11/21/2011:
Yahoo! Site Explorer is no longer in use as it has combined with Bing Webmaster Tools.
—
Top Organic Search Engine Rankings is No Longer an Option
SEO is the holy grail of free, relevant traffic. You’re dead if you aren’t in the top 3 positions for your keyword search. In the late 1990’s dinosaur days of the Internet you had to be in the first 3 pages. Later it was the first page to get any clicks. Now its the top 3 positions or die.
This definitive search engine registrations article explains how to make sure your website will always be set up for maximum organic rankings and Internet exposure. These practices are usually known by SEO companies or web development companies but amazingly it isn’t known by most of them.
You can do many things both on-site and off-site to get your rankings, but if you don’t do what is described below then your site will never get the rankings that you would otherwise deserve. You just have to do what I describe below to get the organic rankings traction you expect from your website.
SEO Files Required for Optimum Indexing
Search engines require 2 files to be placed into your root directory:
1. The robots.txt file tells the search engines what you don’t want placed into their indexes. This is accomplished by placing a properly formatted robots.txt file into the root folder of your website. See these Google guidelines for creating a robots.txt file.
2. The sitemap.xml file tells the search engines exactly what you do want to be indexed and found on the Internet. A properly formatted sitemap.xml file should be placed into the root folder of your website. Here are some Google guidelines on sitemap.xml files.
Point robots.txt to the sitemap.xml File
A standard method of ensuring your sitemap is recognized by the search engines is to add this line to your robots.txt file:
Sitemap: http://www.yourdomain.com/sitemap.xml
Where to Register Your Website and Sitemap in the Top 3 Search Engines
These details are critical but elusive to find. To register your website and sitemap.xml in each of the search engines you need to go to each of their webmaster pages and create files or meta-tags to place on your website. This validates your ownership and points to your sitemap.xml file. Note that you typically don’t have to register the robots.txt file. Register with each of the sites below and add your site to your account. This can either mean that you upload a file with contents that the search engine can recognize, or add a meta-tag to the header of your sites home page that they can recognize. Go to each of these pages, register for an account, register your website and validate your sitemap.xml file:
Google Webmaster Central– http://www.google.com/webmasters/
Yahoo! Site Explorer – http://siteexplorer.search.yahoo.com/
Bing Webmaster Center – http://www.bing.com/webmaster
Canonicalization – Be Consistent in Your Website Addressing for Maximum Exposure
This issue is about how you reference (address) your website. For example, all of the following are ways to reference your home page and are usually the same page, but they could actually be a different page to you need to be careful in how you reference your website. If you aren’t consistent in addressing your website then you’ll most likely dilute the SEO value of most of the links to your website.
Here are a few examples of how you can lose SEO value by inconsistently addressing just the home page of your site :
1. http://www.yourdomain.com
2. http://yourdomain.com
3. http://www.yourdomain.com/index.php
If you use all of these methods of addressing your home page then your SEO value will be divided 3 ways thus reducing the overall SEO value of the links and minimizing your organic search engine results page (SERP) rankings.
Matt Cutts is a member of the Google Search Quality group that specializes in search engine optimization issues and has commented on this canonical topic a few times.
So, now that you understand the issue and how to do it right the first time, how do you manage an existing site with many pages? Use a 301 SEO friendly redirect. In Linux this is easily done with the .htaccess file with this syntax:
# redirect yourdomain.com to www.yourdomain.com
# RewriteCond %{HTTP_HOST} ^yourdomain.com [NC]
# RewriteRule ^(.*)$ http://www.yourdomain.com/$1 [L,R=301]
In Windows you have to place code on each page that redirects to the new page. Alternatively, if you have full control of the server you can install an application that provides similar functionality as the Linux .htaccess file.
In Google’s Webmaster Tools register both versions of your domain, one with www. and the other without it. For example http://www.yourdonain.com and http://yourdomain.com Next click Site Configuration > Settings > Preferred Domain and select the your preferred domain name, either with or without the www. This will consolidate all links to your site into one form thus getting credit for all links and increasing your SEO site worth.
More Matt Cutts comments on the subject:
http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=139394
Local SEO