Welcome Guest Search | Active Topics | Members

robots.txt Options
bigbrennan
Posted: Wednesday, August 31, 2005 5:01:33 PM
Rank: Larger than Life

Joined: 11/15/2003
Posts: 1,649
Location: United Kingdom
I have been recently thinking up ways of increasing my google listings.

Whilst reading a few tutorials, I came across the robots.txt which is a simple file on the server that tells the Spiders where to look and not to look for information about the site.


Does anyone use this on their site? If so can I have an example of it, as I would like to implement it, but not 100% on how to code it
Steviepunk
Posted: Wednesday, August 31, 2005 6:28:35 PM

Rank: Admin

Joined: 8/26/2002
Posts: 3,467
Location: United Kingdom
I have one, however it is just a basic one:
-----
User-agent: *
Allow: /
Disallow: /Application.cfm
Disallow: /swoj.css
-----

It just allows all robots to access the site but not the two files mentioned (why I listed the .css file I can't remember!)

Another useful tool to use to make sure Google finds everything on your site is the Sitemaps feaure of Google. You generate a file that Google checks, in that file you create a list of all the pages in your site and the suggested refresh for the Googlebot.
http://www.google.com/webmasters/sitemaps

I've not set one up yet for Swoj.com, though we do use it for work.

I've also found that having the Google Ads on the site helps as well, anywhere you display an Ad Banner, Google will index the page so that it knows what types of Ads to put on it.
rogem002
Posted: Wednesday, August 31, 2005 8:31:16 PM

Rank: Martian Estate Agent

Joined: 9/14/2003
Posts: 3,358
Location: United Kingdom
Sitemaps - If you need a Sitemap that makes 'Google friendly' check out http://www.xml-sitemaps.com/ they have some online and server tools that do it for you.

Also see RSS and Meta Tags, I'm sure if you looks hard enough you will find a robots.txt generator.


I don't use Google Ads as they don't pay you using PayPal.

Full On Design
tom_h_500
Posted: Monday, September 26, 2005 5:53:33 PM

Rank: L4: post-aholic

Joined: 5/13/2004
Posts: 1,437
Location: United Kingdom
if you go to www.whitehouse.gov/robots.txt they have a MASSIVE one
Users browsing this topic
Guest


Forum Jump
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

Main Forum RSS : RSS

Powered by Yet Another Forum.net version 1.9.1.8 (NET v2.0) - 3/29/2008
Copyright © 2003-2008 Yet Another Forum.net. All rights reserved.