Commits · 979f2e8d216629f8beea4bb224686c73ddd0c4b6 · ale / crawl

Dec 19, 2017
- Crawl IFRAMEs as related resources · 979f2e8d
  ale authored 7 years ago
  
  979f2e8d
- Simplify redirectHandler.Handle · 3ff5e7b8
  ale authored 7 years ago
  
  3ff5e7b8
- Update cmd/links to new scope syntax · 06f170e9
  ale authored 7 years ago
  
  06f170e9
- Skip data: URLs · 665f7ba1
  ale authored 7 years ago
  
  665f7ba1
- Add tags (primary/related) to links · 4cd67e72
  ale authored 7 years ago
  4cd67e72
Dec 18, 2017
- Add CI configuration (test only) · 77211d4f
  ale authored 7 years ago
  
  77211d4f
- Add support for @import syntax in css · 680d0e4b
  ale authored 7 years ago
  
  680d0e4b
- Update location of the uuid package · b4be821f
  ale authored 7 years ago
  
  b4be821f
- Add vendor deps · 445bf898
  ale authored 7 years ago
  
  445bf898
- Switch to github.com/syndtr/goleveldb · 394de2d9
  ale authored 7 years ago
  394de2d9
Jul 03, 2015
- minor golint fixes · 4fc0b1d2
  ale authored 9 years ago
  
  4fc0b1d2
Jun 29, 2015
- clean up the state directory when done · b06e5a29
  ale authored 9 years ago
  
  b06e5a29
- improve queue code; golint fixes · 9fbc656c
  ale authored 9 years ago
  9fbc656c
Jun 28, 2015
- add ignore list from ArchiveBot · 63bd51e0
  ale authored 9 years ago
  
  63bd51e0
- fix timestamp format · aa6e67d7
  ale authored 9 years ago
  aa6e67d7
Dec 20, 2014
- move URLInfo logic into the Crawler itself · 3af8c9a2
  ale authored 10 years ago
  
  3af8c9a2
- add a prefix iterator to gobDb · 7c1b1f70
  ale authored 10 years ago
  
  7c1b1f70
- add tests to scope.go · c31b2d91
  ale authored 10 years ago
  
  c31b2d91
- make Scope checking more modular · 4c82422d
  ale authored 10 years ago
  
  4c82422d
- relax the CSS url() regexp · efe98903
  ale authored 10 years ago
  
  efe98903
- move link extraction to a common location · f0c14e5e
  ale authored 10 years ago
  
  f0c14e5e
- move the WARC code into its own package · d4c561c2
  ale authored 10 years ago
  d4c561c2
Dec 19, 2014
- initial commit · b09f05f8
  ale authored 10 years ago
  
  b09f05f8