OpenWebSpider is a multi-threaded Web spider (also known as a robot or crawler) and search engine with a lot of interesting features.