Creating a search engine

talk random
1

Hey everybody :smiley:

I am developing my own search engine as a way for me to learn more programing languages and more about the web. I’ve come to a halt tho because I cant decide what language to make the web crawler in. Has anybody had experience with this or know of any websites on how to make web crawlers or tutorials that deal with scanning the web. I know… indexing the entire web requires many servers ect. but this is only for me to test and expand on ect.

Thanks a lot!

2

I’m guessing python or perl but not really sure, I haven’t used either. PHP is a definate no imo though.

3

Lol yea… php is out… i was thinking at first C# but I havent found many refrences to building a web spider in it…

4

C, C++, Perl, visiual, All very good… Personally… Combination of Perl and C.
Making a webcrawler is harder then you might think… Scanning IPs for websites then checking there content… I mean… hard stuff… but good luck :slight_smile:

5

:smiley: thanks - looking to be a slow start though because of lack of refrences on the web, altho I have books on C and visual but they dont help with connecting to sites and downloading info. Im also thinking java, any idea with that?

6

C‡ !!

If it were too easy to make a web crawler, Google would have more competitors.

7

yep Java… actually should of put that in the list to… but no experiance with it… or little…

I made an apication… that went trow a web database and gatherd information…

basicly… it was… the C exe opening the perl script then storing the perl output in the database on my pc… thus making a back up… worked really nice…

8

Google would have more competitors.

Google has no competitors :beam:

As for the webcrawler though, good luck! It’s a very complex task, but i wish you luck nonetheless (hell, i know I couldn’t do it :P)

9

:smiley: C# sounds best to me along with java… I’m doing more research on those topics… I will be googles competition :wink: lol

10

looking at people who have made active content filters for ISP’s I would suggest Perl…but good luck :slight_smile:

as long as you don’t name it plex haha.

11

Perl is good. Lisp is also good for recursive stuff such as search engines :slight_smile:

12

Google actually uses Java, perl, and c++ (all three), though I’ve seen many in pure C++ too.

LISP would kill you (it’s like suggesting SML/NJ) and provide you with very little worthwhile experience.

Definitely look into Java and/or C++ - they’re both widely used in the ‘real world’, so what you learn will benefit you in the future.

13

:smiley: thanks a lot for the suggestions and support! ill let you know how it goes.