I feel special surely theirs more?
Also what does this crawler do?
Just collect outbound links or does it do a collect outbound links then crawl that outbound link for outbound links? and whats it for?
I feel special surely theirs more?
Also what does this crawler do?
Just collect outbound links or does it do a collect outbound links then crawl that outbound link for outbound links? and whats it for?
Currently it grabs all the weblinks on the website/page. I have been playing with the coding for a short while and am working on a system that goes on for 10 terms.
eg.
website.com would have 3 websites (website1.com, website2.com and website3.com), they would find the websites lets say they find 9 (website4.com, website5.com, website6.com, website7.com, website8.com, website9.com, website10.com, website11.com, website12.com) and so on...
What is this for...
well I am wanting to build a search engine so I have started collecting webpages.
R-ob.com
Robertoland.com
Qontrol.net
Ouruploads.com
Robmail.info
Cheeseandcrackers.info
Youtool.us
And whats this spider gonna do for you/us??
Oh.. What like google?
Thats a point why not run google on it?
Last edited by Ryan.; 29-10-2006 at 10:52 AM.
Hey Danny - tis Craig
www.bbc.co.uk
www.bbc.co.uk/sport [subdomain onto it lol]
www.newsground.com
www.play.com
www.amazon.co.uk
Last 2 should reach loads
yes blud
Want to hide these adverts? Register an account for free!