A complete rewrite for higher performance, including support for all media types, asynchronous DNS lookups, and optional wait time between hosts. The configuration file specifies permittable size of media objects depending on media-type, etc.
This release fixes a bug where crawl would stop
early when encountering errors on subsequent
connections. The verbosity level and number of
concurrent connections are tunable now
