Catching a spider advice

1

I have just been going through this article :here

And I am a bit confused, when adding the meta tags: <META NAME=“ROBOTS” CONTENT=“INDEX, FOLLOW”>

Do i add this to every page within my site, because i just looked on my works server and they already had a robot.txt file and this meta tag was within that, so is it both or have I lost the plot :slight_smile:

Thanks for any advice

2

You actually want them spiders going to the whole site ???

3

there is usually only one Robots.txt, and it usually doesn’t contain a meta tag. Instead: http://www.searchengineworld.com/robots/robots_tutorial.htm. The robots.txt files tells the search engine spiders(bots) where they can and can’t look for information/search terms.

4

Thanks guyz, Ithought the robot.txt file should not contain a meta file instruction!

I’ll set the file to (try) stop spambots and restrict access to certain folders such as cgibin etc

Cheers