# Catching a spider advice **URL:** \ **Category:** programming\ **Created:** [August 25, 2004, 3:56pm UTC](https://forum.kirupa.com/t/catching-a-spider-advice/62178 "2004-08-25T15:56:13Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![kev](https://avatars.discourse-cdn.com/v4/letter/k/e9bcb4/32.png) [@kev](https://forum.kirupa.com/u/kev)\ **Post date:** [August 25, 2004, 3:56pm UTC](https://forum.kirupa.com/t/catching-a-spider-advice/62178/1 "2004-08-25T15:56:13Z")
I have just been going through this article :[here](http://nl.internet.com/ct.html?rtr=on&s=1,12xo,1,hb6k,dmml,em9a,m7na) And I am a bit confused, when adding the meta tags: \ Do i add this to every page within my site, because i just looked on my works server and they already had a robot.txt file and this meta tag was within that, so is it both or have I lost the plot 🙂 Thanks for any advice ---
**Author:** ![system](https://canada1.discourse-cdn.com/flex011/uploads/kirupa/original/3X/6/2/621e5c11736f46532526e61be85940af4230f3e5.png) [@system](https://forum.kirupa.com/u/system)\ **Post date:** [August 25, 2004, 6:11pm UTC](https://forum.kirupa.com/t/catching-a-spider-advice/62178/2 "2004-08-25T18:11:55Z")
You actually want them spiders going to the whole site ??? ---
**Author:** ![system](https://canada1.discourse-cdn.com/flex011/uploads/kirupa/original/3X/6/2/621e5c11736f46532526e61be85940af4230f3e5.png) [@system](https://forum.kirupa.com/u/system)\ **Post date:** [August 25, 2004, 6:16pm UTC](https://forum.kirupa.com/t/catching-a-spider-advice/62178/3 "2004-08-25T18:16:53Z")
there is usually only one Robots.txt, and it usually doesn’t contain a meta tag. Instead: [http://www.searchengineworld.com/robots/robots\_tutorial.htm](http://www.searchengineworld.com/robots/robots_tutorial.htm). The robots.txt files tells the search engine spiders(bots) where they can and can’t look for information/search terms. ---
**Author:** ![system](https://canada1.discourse-cdn.com/flex011/uploads/kirupa/original/3X/6/2/621e5c11736f46532526e61be85940af4230f3e5.png) [@system](https://forum.kirupa.com/u/system)\ **Post date:** [August 25, 2004, 8:18pm UTC](https://forum.kirupa.com/t/catching-a-spider-advice/62178/4 "2004-08-25T20:18:02Z")
Thanks guyz, Ithought the robot.txt file should not contain a meta file instruction! I’ll set the file to (try) stop spambots and restrict access to certain folders such as cgibin etc Cheers ---
**Author:** ![system](https://canada1.discourse-cdn.com/flex011/uploads/kirupa/original/3X/6/2/621e5c11736f46532526e61be85940af4230f3e5.png) [@system](https://forum.kirupa.com/u/system)\ **Post date:** [August 26, 2004, 10:53am UTC](https://forum.kirupa.com/t/catching-a-spider-advice/62178/5 "2004-08-26T10:53:35Z")
Been through all the articles and my robots.txt file is ok now, got it validated also to check it works 🙂 But one thing I am still confused is that should I also add the meta tag to my html pages aswell as having the robots.txt file? ---
**Author:** ![system](https://canada1.discourse-cdn.com/flex011/uploads/kirupa/original/3X/6/2/621e5c11736f46532526e61be85940af4230f3e5.png) [@system](https://forum.kirupa.com/u/system)\ **Post date:** [August 26, 2004, 3:19pm UTC](https://forum.kirupa.com/t/catching-a-spider-advice/62178/6 "2004-08-26T15:19:49Z")
The robots.txt file will work, but most bots nowadays don’t even read meta tags. So if you want to put meta tags on all your pages… by all means. ---
**Author:** ![system](https://canada1.discourse-cdn.com/flex011/uploads/kirupa/original/3X/6/2/621e5c11736f46532526e61be85940af4230f3e5.png) [@system](https://forum.kirupa.com/u/system)\ **Post date:** [August 27, 2004, 7:02am UTC](https://forum.kirupa.com/t/catching-a-spider-advice/62178/7 "2004-08-27T07:02:11Z")
Thanks, Anygood manual submission websites about, and what’s the best software to get? Also any tips that you use? ---
**Author:** ![system](https://canada1.discourse-cdn.com/flex011/uploads/kirupa/original/3X/6/2/621e5c11736f46532526e61be85940af4230f3e5.png) [@system](https://forum.kirupa.com/u/system)\ **Post date:** [August 31, 2004, 8:02am UTC](https://forum.kirupa.com/t/catching-a-spider-advice/62178/8 "2004-08-31T08:02:16Z")
Bump