Commits · cce28f44e7ad88900e6c53394a8e496f2955b784 · ale / crawl

Jan 19, 2019
- Replace URLInfo with a simple URL presence check · cce28f44
  ale authored 6 years ago
  cce28f44
Jan 02, 2019
- Add multi-file output · c5ec7eb8
  ale authored 6 years ago
  c5ec7eb8
Dec 28, 2018
- Updated dependencies · 3518feaf
  ale authored 6 years ago
  
  3518feaf
Dec 27, 2018
- Normalize URLs before checking if they are in scope · 2f3ca2f8
  ale authored 6 years ago
  
  2f3ca2f8
- Merge branch 'master' of git.autistici.org:ale/crawl · 52eba2bb
  ale authored 6 years ago
  
  52eba2bb
Dec 06, 2018
- Apply --excludes to related resources too · 8ca8cc1f
  ale authored 6 years ago
  
  8ca8cc1f
Sep 02, 2018
- Fix typo · bbc09df4
  ale authored 6 years ago
  
  bbc09df4
- Explicitly mention the crawler limitations · 59f3725f
  ale authored 6 years ago
  
  59f3725f
- Add --exclude and --exclude-file options · 66ce654d
  ale authored 6 years ago
  66ce654d
- Minimal support for <video> and <object> tags · a5d20a9a
  ale authored 6 years ago
  
  a5d20a9a
Aug 31, 2018
- Do not drop /index.html at the end of URLs · 81e16ffb
  ale authored 6 years ago
  
  81e16ffb
- Add a simple test for the full WARC crawler · 23a80bd6
  ale authored 6 years ago
  
  23a80bd6
- Explicitly delegate retry logic to handlers · 98253349
  ale authored 6 years ago
  98253349
- Improve error handling, part two · 70c12b7a
  ale authored 6 years ago
  70c12b7a
- Use a buffered Writer for WARC output · 98e2528f
  ale authored 6 years ago
  
  98e2528f
- Improve error checking · ee1a3d8e
  ale authored 6 years ago
  ee1a3d8e
- Update dependencies · b3d41948
  ale authored 6 years ago
  
  b3d41948
Aug 30, 2018
- Mention trickle as a possible bandwidth limiter · 86a0bd2d
  ale authored 6 years ago
  86a0bd2d
- Improve install instructions a bit more · 19863316
  ale authored 6 years ago
  
  19863316
- Update installation instructions · d434efe8
  ale authored 6 years ago
  
  d434efe8
Dec 19, 2017
- Provide better defaults for command-line options · df800e15
  ale authored 7 years ago
  df800e15
- Merge branch 'master' of git.autistici.org:ale/crawl · b06b0cd4
  ale authored 7 years ago
  
  b06b0cd4
- Exit gracefully on signals · ddecd376
  ale authored 7 years ago
  
  ddecd376
- Add a README · 03f5e296
  ale authored 7 years ago
  
  03f5e296
- Use a global http.Client with sane settings · 6f5bef5f
  ale authored 7 years ago
  
  6f5bef5f
- Crawl IFRAMEs as related resources · 979f2e8d
  ale authored 7 years ago
  
  979f2e8d
- Simplify redirectHandler.Handle · 3ff5e7b8
  ale authored 7 years ago
  
  3ff5e7b8
- Add license · 912a2926
  ale authored 7 years ago
  
  912a2926
- Update cmd/links to new scope syntax · 06f170e9
  ale authored 7 years ago
  
  06f170e9
- Skip data: URLs · 665f7ba1
  ale authored 7 years ago
  
  665f7ba1
- Add tags (primary/related) to links · 4cd67e72
  ale authored 7 years ago
  4cd67e72
Dec 18, 2017
- Add CI configuration (test only) · 77211d4f
  ale authored 7 years ago
  
  77211d4f
- Add support for @import syntax in css · 680d0e4b
  ale authored 7 years ago
  
  680d0e4b
- Update location of the uuid package · b4be821f
  ale authored 7 years ago
  
  b4be821f
- Add vendor deps · 445bf898
  ale authored 7 years ago
  
  445bf898
- Switch to github.com/syndtr/goleveldb · 394de2d9
  ale authored 7 years ago
  394de2d9
Jul 03, 2015
- minor golint fixes · 4fc0b1d2
  ale authored 9 years ago
  
  4fc0b1d2
Jun 29, 2015
- clean up the state directory when done · b06e5a29
  ale authored 9 years ago
  
  b06e5a29
- improve queue code; golint fixes · 9fbc656c
  ale authored 9 years ago
  9fbc656c
Jun 28, 2015
- add ignore list from ArchiveBot · 63bd51e0
  ale authored 9 years ago
  
  63bd51e0