Commits · 63bd51e06b32d48878da68df8931809d42996df1 · ale / crawl

Jun 28, 2015
- add ignore list from ArchiveBot · 63bd51e0
  ale authored 9 years ago
  
  63bd51e0
- fix timestamp format · aa6e67d7
  ale authored 9 years ago
  
  The WARC-Date fields now are UTC times in proper ISO-8601 format. This makes pywb and other tools happy.
  aa6e67d7
Dec 20, 2014
- move URLInfo logic into the Crawler itself · 3af8c9a2
  ale authored 10 years ago
  
  3af8c9a2
- add a prefix iterator to gobDb · 7c1b1f70
  ale authored 10 years ago
  
  7c1b1f70
- add tests to scope.go · c31b2d91
  ale authored 10 years ago
  
  c31b2d91
- make Scope checking more modular · 4c82422d
  ale authored 10 years ago
  
  4c82422d
- relax the CSS url() regexp · efe98903
  ale authored 10 years ago
  
  efe98903
- move link extraction to a common location · f0c14e5e
  ale authored 10 years ago
  
  f0c14e5e
- move the WARC code into its own package · d4c561c2
  ale authored 10 years ago
  
  Now generates well-formed, indexable WARC files.
  d4c561c2
Dec 19, 2014
- initial commit · b09f05f8
  ale authored 10 years ago
  
  b09f05f8