- Feb 17, 2020
-
-
ale authored
In order to do this we have to plumb it through the queue and the Handler interface, but it should allow fetches of the resources associated with a page via the IncludeRelatedScope even if it's behind a redirect.
-
- Jan 20, 2019
-
-
ale authored
Introduce an interface to decouple the Enqueue functionality from the Crawler implementation.
-
- Aug 31, 2018
- Dec 19, 2017
-
-
ale authored
This change allows more complex scope boundaries, including loosening edges a bit to include related resources of HTML pages (which makes for more complete archives if desired).
-