Commits · 06f170e9ca181f145a57150c111a0982db32d023 · ale / crawl

Dec 19, 2017

Update cmd/links to new scope syntax · 06f170e9
ale authored 7 years ago

06f170e9

Add tags (primary/related) to links · 4cd67e72

ale authored 7 years ago

This change allows more complex scope boundaries, including loosening
edges a bit to include related resources of HTML pages (which makes
for more complete archives if desired).

4cd67e72

Jul 03, 2015
- minor golint fixes · 4fc0b1d2
  ale authored 9 years ago
  
  4fc0b1d2
Jun 29, 2015
- clean up the state directory when done · b06e5a29
  ale authored 9 years ago
  
  b06e5a29
Jun 28, 2015
- add ignore list from ArchiveBot · 63bd51e0
  ale authored 9 years ago
  
  63bd51e0
Dec 20, 2014
- move URLInfo logic into the Crawler itself · 3af8c9a2
  ale authored 10 years ago
  
  3af8c9a2
- make Scope checking more modular · 4c82422d
  ale authored 10 years ago
  
  4c82422d
- move link extraction to a common location · f0c14e5e
  ale authored 10 years ago
  
  f0c14e5e
- move the WARC code into its own package · d4c561c2
  ale authored 10 years ago
  
  Now generates well-formed, indexable WARC files.
  d4c561c2
Dec 19, 2014
- initial commit · b09f05f8
  ale authored 10 years ago
  
  b09f05f8