Skip to content
  • ale's avatar
    Enable proper rate limiting functionality · b6901eee
    ale authored
    Use the --qps option to control per-hostname rate limits. The limit is
    currently global and applied independently to each hostname, before
    DNS resolution (due to the current structure of the crawler that
    performs DNS resolution in the worker), but it should be definitely
    better than nothing.
    b6901eee