whalebot C/C++ web crawler

Summary

Whalebot is open-source web crawler. It is intended to be simple, fast and memory efficient. It was created as a targeted spider, but you may use it as common.

Current release 0.02

Current state. Bold - done, normal - TODO

If something broken or you have an idea, please visit http://groups.google.com/group/whalebot

Usages

  • It was used for collecting papers on target thematic from http://citeseerx.ist.psu.edu for my master degree work
  • Candidates for logo were collected using whalebot
  • Eating own dogs food (links for url parsing benchmark)

Features

  • Simple configuration from command line and text files
  • Start/Stop/Resume fetching sessions

你可能感兴趣的:(c/c++)