webbase is an Internet crawler. It is able to crawl and maintain millions of URLs and store information about them in a MySQL database. The interface is either a command line program or a C library. It contains hooks to plug a full text indexing database.
