Commits · 37c649a8b693ba65a59eab5de3c01bf212f791ad · ale / crawl

Aug 20, 2020
- Panic instead of just dying with fatal error · cf35cce6
  ale authored 4 years ago
  
  cf35cce6
Jul 30, 2020
- Panic on fatal errors · de767cce
  ale authored 4 years ago
  de767cce
Feb 17, 2020
- Propagate the link tag through redirects · 533f4725
  ale authored 5 years ago
  533f4725
Jan 20, 2019
- Refactor Handlers in terms of a Publisher interface · 64eb5fb2
  ale authored 6 years ago
  64eb5fb2
Jan 19, 2019
- Replace URLInfo with a simple URL presence check · cce28f44
  ale authored 6 years ago
  cce28f44
Dec 27, 2018
- Normalize URLs before checking if they are in scope · 2f3ca2f8
  ale authored 6 years ago
  
  2f3ca2f8
Aug 31, 2018
- Do not drop /index.html at the end of URLs · 81e16ffb
  ale authored 6 years ago
  
  81e16ffb
- Explicitly delegate retry logic to handlers · 98253349
  ale authored 6 years ago
  98253349
- Improve error handling, part two · 70c12b7a
  ale authored 6 years ago
  70c12b7a
- Improve error checking · ee1a3d8e
  ale authored 6 years ago
  ee1a3d8e
Dec 19, 2017
- Exit gracefully on signals · ddecd376
  ale authored 7 years ago
  
  ddecd376
- Simplify redirectHandler.Handle · 3ff5e7b8
  ale authored 7 years ago
  
  3ff5e7b8
- Add tags (primary/related) to links · 4cd67e72
  ale authored 7 years ago
  4cd67e72
Dec 18, 2017
- Switch to github.com/syndtr/goleveldb · 394de2d9
  ale authored 7 years ago
  394de2d9
Jul 03, 2015
- minor golint fixes · 4fc0b1d2
  ale authored 9 years ago
  
  4fc0b1d2
Jun 29, 2015
- clean up the state directory when done · b06e5a29
  ale authored 9 years ago
  
  b06e5a29
- improve queue code; golint fixes · 9fbc656c
  ale authored 9 years ago
  9fbc656c
Dec 20, 2014
- move URLInfo logic into the Crawler itself · 3af8c9a2
  ale authored 10 years ago
  
  3af8c9a2
- add a prefix iterator to gobDb · 7c1b1f70
  ale authored 10 years ago
  
  7c1b1f70
- make Scope checking more modular · 4c82422d
  ale authored 10 years ago
  
  4c82422d
- move the WARC code into its own package · d4c561c2
  ale authored 10 years ago
  d4c561c2
Dec 19, 2014
- initial commit · b09f05f8
  ale authored 10 years ago
  
  b09f05f8