Commits · 5f82887302cfc65975f793fc7242f1423dbac18b · ale / crawl

Aug 26, 2020
- Minor logging fixes · a3019b17
  ale authored 4 years ago
  
  a3019b17
Aug 23, 2020
- Fix the crawl.go tests · d771e9f9
  ale authored 4 years ago
  
  d771e9f9
- Allow setting DNS overrides using the --resolve option · 37c649a8
  ale authored 4 years ago
  
  37c649a8
Jul 30, 2020
- Retry requests on transport-level errors · 9d07df09
  ale authored 4 years ago
  
  9d07df09
Feb 17, 2020
- Propagate the link tag through redirects · 533f4725
  ale authored 5 years ago
  533f4725
Jan 20, 2019
- Refactor Handlers in terms of a Publisher interface · 64eb5fb2
  ale authored 6 years ago
  64eb5fb2
Jan 02, 2019
- Add multi-file output · c5ec7eb8
  ale authored 6 years ago
  c5ec7eb8
Dec 06, 2018
- Apply --excludes to related resources too · 8ca8cc1f
  ale authored 6 years ago
  
  8ca8cc1f
Sep 02, 2018
- Add --exclude and --exclude-file options · 66ce654d
  ale authored 6 years ago
  66ce654d
Aug 31, 2018
- Add a simple test for the full WARC crawler · 23a80bd6
  ale authored 6 years ago
  
  23a80bd6
- Explicitly delegate retry logic to handlers · 98253349
  ale authored 6 years ago
  98253349
- Improve error handling, part two · 70c12b7a
  ale authored 6 years ago
  70c12b7a
- Improve error checking · ee1a3d8e
  ale authored 6 years ago
  ee1a3d8e
Dec 19, 2017
- Provide better defaults for command-line options · df800e15
  ale authored 7 years ago
  df800e15
- Exit gracefully on signals · ddecd376
  ale authored 7 years ago
  
  ddecd376
- Use a global http.Client with sane settings · 6f5bef5f
  ale authored 7 years ago
  
  6f5bef5f
- Add tags (primary/related) to links · 4cd67e72
  ale authored 7 years ago
  4cd67e72
Jul 03, 2015
- minor golint fixes · 4fc0b1d2
  ale authored 9 years ago
  
  4fc0b1d2
Jun 29, 2015
- clean up the state directory when done · b06e5a29
  ale authored 9 years ago
  
  b06e5a29
Jun 28, 2015
- add ignore list from ArchiveBot · 63bd51e0
  ale authored 9 years ago
  
  63bd51e0
Dec 20, 2014
- move URLInfo logic into the Crawler itself · 3af8c9a2
  ale authored 10 years ago
  
  3af8c9a2
- make Scope checking more modular · 4c82422d
  ale authored 10 years ago
  
  4c82422d
- move link extraction to a common location · f0c14e5e
  ale authored 10 years ago
  
  f0c14e5e
- move the WARC code into its own package · d4c561c2
  ale authored 10 years ago
  d4c561c2
Dec 19, 2014
- initial commit · b09f05f8
  ale authored 10 years ago
  
  b09f05f8