diff --git a/doc/BACKUP.md b/doc/BACKUP.md index 55f1c2a45..a607b1416 100644 --- a/doc/BACKUP.md +++ b/doc/BACKUP.md @@ -30,7 +30,9 @@ For example, if you are running `--mainnet`, it will be ## `hsm_secret` -`/!\` WHO SHOULD DO THIS: Everyone. +!!! note + + WHO SHOULD DO THIS: Everyone. You need a copy of the `hsm_secret` file regardless of whatever backup strategy you use. @@ -84,12 +86,14 @@ backup strategies below. ## SQLITE3 `--wallet=${main}:${backup}` And Remote NFS Mount -`/!\` WHO SHOULD DO THIS: Casual users. +!!! note -`/!\` **CAUTION** `/!\` This technique is only supported after the version v0.10.2 (not included) -or later. -On earlier versions, the `:` character is not special and will be -considered part of the path of the database file. + WHO SHOULD DO THIS: Casual users. + +!!! warning + + This technique is only supported after the version v0.10.2 (not included) or later. + On earlier versions, the `:` character is not special and will be considered part of the path of the database file. When using the SQLITE3 backend (the default), you can specify a second database file to replicate to, by separating the second @@ -100,11 +104,15 @@ For example, if the user running `lightningd` is named `user`, and you are on the Bitcoin mainnet with the default `${LIGHTNINGDIR}`, you can specify in your `config` file: - wallet=sqlite3:///home/user/.lightning/bitcoin/lightningd.sqlite3:/my/backup/lightningd.sqlite3 +```bash +wallet=sqlite3:///home/user/.lightning/bitcoin/lightningd.sqlite3:/my/backup/lightningd.sqlite3 +``` Or via command line: - lightningd --wallet=sqlite3:///home/user/.lightning/bitcoin/lightningd.sqlite3:/my/backup/lightningd.sqlite3 +```bash +lightningd --wallet=sqlite3:///home/user/.lightning/bitcoin/lightningd.sqlite3:/my/backup/lightningd.sqlite3 +``` If the second database file does not exist but the directory that would contain it does exist, the file is created. @@ -173,7 +181,9 @@ like fire or computer confiscation. ## `backup` Plugin And Remote NFS Mount -`/!\` WHO SHOULD DO THIS: Casual users. +!!! note + + WHO SHOULD DO THIS: Casual users. You can find the full source for the `backup` plugin here: https://github.com/lightningd/plugins/tree/master/backup @@ -221,8 +231,9 @@ like fire or computer confiscation. ## Filesystem Redundancy -`/!\` WHO SHOULD DO THIS: Filesystem nerds, data hoarders, home labs, -enterprise users. +!!! note + + WHO SHOULD DO THIS: Filesystem nerds, data hoarders, home labs, enterprise users. You can set up a RAID-1 with multiple storage devices, and point the `$LIGHTNINGDIR` to the RAID-1 setup. @@ -336,7 +347,9 @@ of new storage devices to set up a new node. ## PostgreSQL Cluster -`/!\` WHO SHOULD DO THIS: Enterprise users, whales. +!!! note + + WHO SHOULD DO THIS: Enterprise users, whales. `lightningd` may also be compiled with PostgreSQL support. PostgreSQL is generally faster than SQLITE3, and also supports running a @@ -420,10 +433,75 @@ This can be difficult to create remote replicas due to the latency. [pqsyncreplication]: https://www.postgresql.org/docs/13/warm-standby.html#SYNCHRONOUS-REPLICATION +## SQLite Litestream Replication + +!!! warning + + Previous versions of this document recommended this technique, but we no longer do so. + According to [issue 4857][], even with a 60-second timeout that we added + in 0.10.2, this leads to constant crashing of `lightningd` in some + situations. + This section will be removed completely six months after 0.10.3. + Consider using + + ``` + --wallet=sqlite3://${main}:${backup} + ``` + + above, instead. + +[issue 4857]: https://github.com/ElementsProject/lightning/issues/4857 + +One of the simpler things on any system is to use Litestream to replicate the SQLite database. +It continuously streams SQLite changes to file or external storage - the cloud storage option +should not be used. +Backups/replication should not be on the same disk as the original SQLite DB. + +You need to enable WAL mode on your database. +To do so, first stop `lightningd`, then: + + $ sqlite3 lightningd.sqlite3 + sqlite3> PRAGMA journal_mode = WAL; + sqlite3> .quit + +Then just restart `lightningd`. + +/etc/litestream.yml : + + dbs: + - path: /home/bitcoin/.lightning/bitcoin/lightningd.sqlite3 + replicas: + - path: /media/storage/lightning_backup + + and start the service using systemctl: + + $ sudo systemctl start litestream + +Restore: + + $ litestream restore -o /media/storage/lightning_backup /home/bitcoin/restore_lightningd.sqlite3 + +Because Litestream only copies small changes and not the entire +database (holding a read lock on the file while doing so), the +60-second timeout on locking should not be reached unless +something has made your backup medium very very slow. + +Litestream has its own timer, so there is a tiny (but +non-negligible) probability that `lightningd` updates the +database, then irrevocably commits to the update by sending +revocation keys to the counterparty, and *then* your main +storage media crashes before Litestream can replicate the +update. +Treat this as a superior version of "Database File Backups" +section below and prefer recovering via other backup methods +first. + ## Database File Backups -`/!\` WHO SHOULD DO THIS: Those who already have at least one of the -other backup methods, those who are #reckless. +!!! note + + WHO SHOULD DO THIS: Those who already have at least one of the + other backup methods, those who are #reckless. This is the least desirable backup strategy, as it *can* lead to loss of all in-channel funds if you use it. @@ -528,3 +606,38 @@ still not assured with this backup strategy. `sqlite3` has `.dump` and `VACUUM INTO` commands, but note that those lock the main database for long time periods, which will negatively affect your `lightningd` instance. + +### `sqlite3` `.dump` or `VACUUM INTO` Commands + +!!! warning + + Previous versions of this document recommended + this technique, but we no longer do so. + According to [issue 4857][issue 4857], even with a 60-second timeout that we added + in 0.10.2, this may lead to constant crashing of `lightningd` in some + situations; this technique uses substantially the same techniques as + `litestream`. + This section will be removed completely six months after 0.10.3. + Consider using `--wallet=sqlite3://${main}:${backup}` above, instead. + +Use the `sqlite3` command on the `lightningd.sqlite3` file, and +feed it with `.dump "/path/to/backup.sqlite3"` or `VACUUM INTO +"/path/to/backup.sqlite3";`. + +These create a snapshot copy that, unlike the previous technique, +is assuredly uncorrupted (barring any corruption caused by your +backup media). + +However, if the copying process takes a long time (approaching the +timeout of 60 seconds) then you run the risk of `lightningd` +attempting to grab a write lock, waiting up to 60 seconds, and +then failing with a "database is locked" error. +Your backup system could `.dump` to a fast `tmpfs` RAMDISK or +local media, and *then* copy to the final backup media on a remote +system accessed via slow network, for example, to reduce this +risk. + +It is recommended that you use `.dump` instead of `VACUUM INTO`, +as that is assuredly faster; you can just open the backup copy +in a new `sqlite3` session and `VACUUM;` to reduce the size +of the backup.