I was looking at getting a web-crawler which would be really easy to use, linux-based and have a zero learning curve and found w-get really good. I remember trying out all those other open source crawlers and getting lost in trying them out.
http://en.wikipedia.org/wiki/Wget
Subscribe to:
Post Comments (Atom)
3 comments:
you may want to look at Hyper Estraier I know it is a great search engine, but it does have crawling features too.. mebbe it could be patched up also..
Check with online crawler to find out how many links a crawler finds in your website...
http://www.mallutalk.com/bmallu/bcrawler/
Thanks Boby. I will look at it
Post a Comment