Allow users to add to the exclude regexp lists easily.

This change allows more complex scope boundaries, including loosening
edges a bit to include related resources of HTML pages (which makes
for more complete archives if desired).

The queuing code now performs proper lease accounting, and it will not
return a URL twice if the page load is slow.