Commits · 74e5aaf573232bb7b5117d94fdcb80a48256b609 · ale / crawl

Aug 23, 2020
- Minor fixes to Debian packaging · 74e5aaf5
  ale authored 4 years ago
  
  74e5aaf5
- Fix the crawl.go tests · d771e9f9
  ale authored 4 years ago
  
  d771e9f9
- Add minimal Debian packaging · 929da174
  ale authored 4 years ago
  
  929da174
- Allow setting DNS overrides using the --resolve option · 37c649a8
  ale authored 4 years ago
  
  37c649a8
Aug 20, 2020
- Panic instead of just dying with fatal error · cf35cce6
  ale authored 4 years ago
  
  cf35cce6
Jul 30, 2020
- Retry requests on transport-level errors · 9d07df09
  ale authored 4 years ago
  
  9d07df09
- Panic on fatal errors · de767cce
  ale authored 4 years ago
  de767cce
Feb 17, 2020
- Fix the Handler in cmd/links · 082784b7
  ale authored 5 years ago
  
  082784b7
- Propagate the link tag through redirects · 533f4725
  ale authored 5 years ago
  533f4725
Dec 04, 2019
- Fix installation instructions · fec78595
  ale authored 5 years ago
  
  fec78595
Nov 13, 2019
- Add contact email address · 2c9f8b07
  ale authored 5 years ago
  
  2c9f8b07
- Update dependencies (legacy and go.mod) · f4455ea6
  ale authored 5 years ago
  
  f4455ea6
Oct 07, 2019
- Add a vendor dependency used for tests · a67708b7
  ale authored 5 years ago
  
  a67708b7
- Parse links in inline style blocks · 6dc36ab4
  ale authored 5 years ago
  
  6dc36ab4
Sep 26, 2019
- Switch to latest Go image for CI test · f4ef521d
  ale authored 5 years ago
  
  f4ef521d
- Add Go module support · 4b3b1290
  ale authored 5 years ago
  
  4b3b1290
- Update vendored dependencies · 97221e85
  ale authored 5 years ago
  
  97221e85
Jan 20, 2019
- Refactor Handlers in terms of a Publisher interface · 64eb5fb2
  ale authored 6 years ago
  64eb5fb2
Jan 19, 2019
- Replace URLInfo with a simple URL presence check · cce28f44
  ale authored 6 years ago
  cce28f44
Jan 02, 2019
- Add multi-file output · c5ec7eb8
  ale authored 6 years ago
  c5ec7eb8
Dec 28, 2018
- Updated dependencies · 3518feaf
  ale authored 6 years ago
  
  3518feaf
Dec 27, 2018
- Normalize URLs before checking if they are in scope · 2f3ca2f8
  ale authored 6 years ago
  
  2f3ca2f8
- Merge branch 'master' of git.autistici.org:ale/crawl · 52eba2bb
  ale authored 6 years ago
  
  52eba2bb
Dec 06, 2018
- Apply --excludes to related resources too · 8ca8cc1f
  ale authored 6 years ago
  
  8ca8cc1f
Sep 02, 2018
- Fix typo · bbc09df4
  ale authored 6 years ago
  
  bbc09df4
- Explicitly mention the crawler limitations · 59f3725f
  ale authored 6 years ago
  
  59f3725f
- Add --exclude and --exclude-file options · 66ce654d
  ale authored 6 years ago
  66ce654d
- Minimal support for <video> and <object> tags · a5d20a9a
  ale authored 6 years ago
  
  a5d20a9a
Aug 31, 2018
- Do not drop /index.html at the end of URLs · 81e16ffb
  ale authored 6 years ago
  
  81e16ffb
- Add a simple test for the full WARC crawler · 23a80bd6
  ale authored 6 years ago
  
  23a80bd6
- Explicitly delegate retry logic to handlers · 98253349
  ale authored 6 years ago
  98253349
- Improve error handling, part two · 70c12b7a
  ale authored 6 years ago
  70c12b7a
- Use a buffered Writer for WARC output · 98e2528f
  ale authored 6 years ago
  
  98e2528f
- Improve error checking · ee1a3d8e
  ale authored 6 years ago
  ee1a3d8e
- Update dependencies · b3d41948
  ale authored 6 years ago
  
  b3d41948
Aug 30, 2018
- Mention trickle as a possible bandwidth limiter · 86a0bd2d
  ale authored 6 years ago
  86a0bd2d
- Improve install instructions a bit more · 19863316
  ale authored 6 years ago
  
  19863316
- Update installation instructions · d434efe8
  ale authored 6 years ago
  
  d434efe8
Dec 19, 2017
- Provide better defaults for command-line options · df800e15
  ale authored 7 years ago
  df800e15
- Merge branch 'master' of git.autistici.org:ale/crawl · b06b0cd4
  ale authored 7 years ago
  
  b06b0cd4