# Catching a spider advice

**URL:** <https://forum.kirupa.com/t/catching-a-spider-advice/120581>\
**Category:** Uncategorized\
**Created:** [August 25, 2004, 3:56pm UTC](https://forum.kirupa.com/t/catching-a-spider-advice/120581 "2004-08-25T15:56:13Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![kev](https://avatars.discourse-cdn.com/v4/letter/k/e9bcb4/32.png) [@kev](https://forum.kirupa.com/u/kev)\
**Post date:** [August 25, 2004, 3:56pm UTC](https://forum.kirupa.com/t/catching-a-spider-advice/120581/1 "2004-08-25T15:56:13Z")

</div>

I have just been going through this article :[here](http://nl.internet.com/ct.html?rtr=on&s=1,12xo,1,hb6k,dmml,em9a,m7na)

And I am a bit confused, when adding the meta tags: \<META NAME=“ROBOTS” CONTENT=“INDEX, FOLLOW”\>

Do i add this to every page within my site, because i just looked on my works server and they already had a robot.txt file and this meta tag was within that, so is it both or have I lost the plot 🙂

Thanks for any advice

---

<div class="post-metadata">

**Author:** ![ol4pr0](https://avatars.discourse-cdn.com/v4/letter/o/5fc32e/32.png) [@ol4pr0](https://forum.kirupa.com/u/ol4pr0)\
**Post date:** [August 25, 2004, 6:11pm UTC](https://forum.kirupa.com/t/catching-a-spider-advice/120581/2 "2004-08-25T18:11:55Z")

</div>

You actually want them spiders going to the whole site ???

---

<div class="post-metadata">

**Author:** ![norie](https://avatars.discourse-cdn.com/v4/letter/n/3ec8ea/32.png) [@norie](https://forum.kirupa.com/u/norie)\
**Post date:** [August 25, 2004, 6:16pm UTC](https://forum.kirupa.com/t/catching-a-spider-advice/120581/3 "2004-08-25T18:16:53Z")

</div>

there is usually only one Robots.txt, and it usually doesn’t contain a meta tag. Instead: [http://www.searchengineworld.com/robots/robots\_tutorial.htm](http://www.searchengineworld.com/robots/robots_tutorial.htm). The robots.txt files tells the search engine spiders(bots) where they can and can’t look for information/search terms.

---

<div class="post-metadata">

**Author:** ![kev](https://avatars.discourse-cdn.com/v4/letter/k/e9bcb4/32.png) [@kev](https://forum.kirupa.com/u/kev)\
**Post date:** [August 25, 2004, 8:18pm UTC](https://forum.kirupa.com/t/catching-a-spider-advice/120581/4 "2004-08-25T20:18:02Z")

</div>

Thanks guyz, Ithought the robot.txt file should not contain a meta file instruction!

I’ll set the file to (try) stop spambots and restrict access to certain folders such as cgibin etc

Cheers