Skip to content
Snippets Groups Projects
Select Git revision
  • master default protected
  • better-queue
2 results

crawler.go

  • ale's avatar
    b6901eee
    Enable proper rate limiting functionality · b6901eee
    ale authored 
    Use the --qps option to control per-hostname rate limits. The limit is
currently global and applied independently to each hostname, before
DNS resolution (due to the current structure of the crawler that
performs DNS resolution in the worker), but it should be definitely
better than nothing.
    b6901eee
    History
    Enable proper rate limiting functionality
    ale authored 
    Use the --qps option to control per-hostname rate limits. The limit is
currently global and applied independently to each hostname, before
DNS resolution (due to the current structure of the crawler that
performs DNS resolution in the worker), but it should be definitely
better than nothing.