Skip to content
Snippets Groups Projects
Commit 86a0bd2d authored by ale's avatar ale
Browse files

Mention trickle as a possible bandwidth limiter

Since such bandwidth limiting is not provided by crawl directly, tell
users there is another solution. Once/if crawl implements that on its
own, that notice could be removed.
parent 19863316
No related branches found
No related tags found
No related merge requests found
......@@ -3,8 +3,11 @@ A very simple crawler
This tool can crawl a bunch of URLs for HTML content, and save the
results in a nice WARC file. It has little control over its traffic,
save for a limit on concurrent outbound requests. Its main purpose is
to quickly and efficiently save websites for archival purposes.
save for a limit on concurrent outbound requests. An external tool
like `trickle` can be used to limit bandwidth.
Its main purpose is to quickly and efficiently save websites for
archival purposes.
The *crawl* tool saves its state in a database, so it can be safely
interrupted and restarted without issues.
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment