GitLab

Since such bandwidth limiting is not provided by crawl directly, tell
users there is another solution. Once/if crawl implements that on its
own, that notice could be removed.

Defaults that are more suitable to real-world site archiving.

This change allows more complex scope boundaries, including loosening
edges a bit to include related resources of HTML pages (which makes
for more complete archives if desired).

The native Go implementation of LevelDB.

The queuing code now performs proper lease accounting, and it will not
return a URL twice if the page load is slow.

The WARC-Date fields now are UTC times in proper ISO-8601 format. This
makes pywb and other tools happy.

Now generates well-formed, indexable WARC files.