Catching a spider advice

programming
1

I have just been going through this article :here

And I am a bit confused, when adding the meta tags: <META NAME=“ROBOTS” CONTENT=“INDEX, FOLLOW”>

Do i add this to every page within my site, because i just looked on my works server and they already had a robot.txt file and this meta tag was within that, so is it both or have I lost the plot :slight_smile:

Thanks for any advice

2

You actually want them spiders going to the whole site ???

3

there is usually only one Robots.txt, and it usually doesn’t contain a meta tag. Instead: http://www.searchengineworld.com/robots/robots_tutorial.htm. The robots.txt files tells the search engine spiders(bots) where they can and can’t look for information/search terms.

4

Thanks guyz, Ithought the robot.txt file should not contain a meta file instruction!

I’ll set the file to (try) stop spambots and restrict access to certain folders such as cgibin etc

Cheers

5

Been through all the articles and my robots.txt file is ok now, got it validated also to check it works :slight_smile:

But one thing I am still confused is that should I also add the meta tag to my html pages aswell as having the robots.txt file?

6

The robots.txt file will work, but most bots nowadays don’t even read meta tags. So if you want to put meta tags on all your pages… by all means.

7

Thanks, Anygood manual submission websites about, and what’s the best software to get? Also any tips that you use?

8

Bump