Utilize the Power of Robots. txt

Robots.txt

Once we have an internet site working, we need to make certain that all visiting search engines can access all the webpages we want those to look at.

Sometimes, we may want search engines to not index certain elements of the site, or even ban other SE from the site all together.

This is where a straightforward, little 2 collection text file called automated programs. txt comes in.

Key phrases:

Domains, search engines, web permissions, search engine optimization, seo

Article Body:

Once we have a website up and running, we need to make sure that visiting search motors can access all the pages we would like them to look at.

Sometimes, we may want search engines to not index certain components of the site, or even ban other ZE from the site all together.

This is where a simple, little 2 line text message file called robots. txt comes in.

Robots. txt resides in your websites main directory (on APACHE systems this is your /public_html/ directory), and looks something like these:

User-agent: *

Disallow:

The very first collection controls the "bot" that will be visiting your site, the second collection controls if they happen to be allowed in, or which parts of the site they are not in order to visit...

When you want to deal with multiple "bots", then simple repeat the above ranges.

So the:

User-agent: googlebot

Disallow:

User-agent: askjeeves

Disallow: /

This will allow Goggle (user-agent name Google Bot) to visit every page and directory site, while at the same time banning Ask Jeeves from the site completely.

To find a "reasonably" up to date list of robot user names this visit http://www.robotstxt.org/wc/active/html/index.html

Even if you would like to allow every robotic to index every page of your site, it's still very advisable to put a robots. txt file on your site. It will stop your error logs completing with entries from search engines like google seeking to access your programs. txt file that doesn't exist.

For more information on robots. txt see, the full list of resources about robots. txt at  <a href="http://www.websitesecrets101.com/robotstxt-further-reading-resources/"> http://www.websitesecrets101.com/robotstxt-further-reading-resources </a>