Use the --qps option to control per-hostname rate limits. The limit is
currently global and applied independently to each hostname, before
DNS resolution (due to the current structure of the crawler that
performs DNS resolution in the worker), but it should be definitely
better than nothing.