Provide "litestream" as an alternative backup option
Many of our apps have a backend which consists of a simple SQLite database. In this scenario, we can do a lot better than float's default "one day acceptable data loss on reschedule" guarantee by using litestream, an asynchronous replication solution for sqlite that can use S3 or SFTP as a backend, which would give us full point-in-time restore capabilities (within a few seconds) in case of service reschedule or hw failure!
This should be an alternative backup mechanism, to replace "tabacco" for those datasets. The main issue is how to run litestream itself, I see two options:
- single host-wide litestream process configured to watch all databases
- litestream sidecar containers, automagically added by float.py to the service specification when we see one of these datasets