Commit Graph

42 Commits

Author SHA1 Message Date
Christian Decker
d28d68cb16 backup: Ensure entries in the tx are bytes
We could end up in some cases with string-encoded statements.
2021-01-16 13:29:37 +01:00
Christian Decker
a097606495 ci: Increase timeout so we don't kill tests too often 2021-01-13 11:23:56 +01:00
Christian Decker
b0784e4ae2 meta: Add development requirements where missing
We're about to start testing in isolation, so better make sure that
each plugin can be tested in isolation.
2021-01-13 11:23:56 +01:00
svewa
d48a8d128d Update backup/README.md
Co-authored-by: Christian Decker <decker.christian@gmail.com>
2021-01-03 16:44:04 +01:00
Sven
a60d8efcf0 clarify backup location to be a file 2021-01-03 16:44:04 +01:00
Christian Decker
70aeb75365 backup: Be more explicit about requirements in readme 2020-12-19 10:59:24 +01:00
Wladimir J. van der Laan
7678fe1ed1 backup: Some improvements to text of README.md 2020-12-19 10:59:24 +01:00
Wladimir J. van der Laan
a6afc5a612 backup: Increase version_count after each version
Fix #183.
2020-12-18 19:42:16 +01:00
Michael Schmoock
1e7701f033 backup: fix flake8 nits 2020-12-13 15:36:15 +01:00
Michael Schmoock
1f05674b5b backup: cleanup flake8 nits 2020-12-11 11:01:02 +01:00
Michael Schmoock
be523aa54f backup: restore default filename lightningd.sqlite3 2020-12-11 11:01:02 +01:00
Michael Schmoock
9591c1b31e backup: doc fix that restore needs a file not a dir 2020-12-11 11:01:02 +01:00
Christian Decker
092b295d06 backup: Rewrite erroneously expanded SQL stmts on-the-fly
Seems to have little to no performance impact.
2020-12-11 10:52:23 +01:00
Christian Decker
e6d26b479c backup: Add a test to reproduce #158 2020-12-11 10:52:23 +01:00
Michael Schmoock
0cbee3fc48 backup: fix loglevel to just INFO 2020-11-27 12:05:51 +01:00
Michael Schmoock
040b3a3a35 backup: adds a basic README.md 2020-11-27 12:05:51 +01:00
Christian Decker
3a1a89ecf8 backup: Add a warning about the deprecated backup-destination flag 2020-11-25 19:26:44 +01:00
Christian Decker
f34787d065 backup: Rely solely on backup.lock and do not cache writes
Caching writes was causing us some issues if the startup of
`lightningd` didn't complete, i.e., the writes would happen on the
live DB, but we'd forget about them. Since we have the backup.lock
file in same directory as the plugin is running in, we no longer have
to defer the writes. This hugely simplifies our logic.

Fixes #155

Changelog-Fixed: backup: The plugin doesn't lose sync anymore if the startup is interrupted
2020-11-25 19:26:44 +01:00
Christian Decker
630d385a0b backup: Mark the backup-destination option as UNUSED
We'll start relying solely on the `backup.lock` file generated by the
`backup-cli`. But we keep the option so we don't automatically break
all configs that simply updated.
2020-11-25 19:26:44 +01:00
Christian Decker
b7804408f4 noise: Add missing dependency on psutil>=5 2020-10-21 16:13:29 +02:00
Christian Decker
b9a8622422 backup: Remove temporary node_factory shim
This was required in order to avoid cleaning the datadir on creation.
2020-08-28 23:06:30 +02:00
Christian Decker
a1f005012a backup: Perform preflight checks before entering the plugin loop 2020-05-18 22:44:49 +02:00
Christian Decker
fbc494ec23 backup: Kill lightningd instead of the roundabout way around
Looking up the process parents and searching for `lightnignd` allows us to
kill directly. It also means we no longer have to wait for the RPC to be
enabled to abort.
2020-05-18 22:44:49 +02:00
Christian Decker
9b5a9076d4 backup: Fix flaky test due to logs being dropped on shutdown
We were writing logs directly before killing `lightningd` which resulted in
the logs being lost. The subsequent check for the log-lines then would fail
obviously.
2020-05-18 22:44:49 +02:00
Christian Decker
41e440522e backup: NF wrapper was missing a keyword arg 2020-04-30 17:52:16 +02:00
Michal Rostecki
8438491350 backup: Add flaky to requirements
Signed-off-by: Michal Rostecki <mrostecki@mailfence.com>
2020-04-15 21:00:30 +02:00
Michal Rostecki
e7c1ae60a1 backup: Remove duplicate call of initialize()
Before this change, backup backend was initialized twice - in on_init()
and get_backend() functions. This change leaves only one call in
get_backend() and adds a kwarg determining whether init failure should
be fatal.

Signed-off-by: Michal Rostecki <mrostecki@mailfence.com>
2020-04-15 21:00:30 +02:00
Christian Decker
ef31a4ac85 backup: Refuse to create a new FileBakend if the backing file exists 2020-04-12 19:35:21 +02:00
Christian Decker
ce49404eb4 backup: Add PRAGMA foreign_keys=on when restoring so cascades work
We heavily rely on foreign key constraints to keep the DB consistent and drop
dependent objects such as UTXO set entries when we roll back a block. We also
need to enable the foreign key constraints when restoring otherwise these
dependents are not removed and we run into constraint violations.
2020-04-12 19:35:21 +02:00
Christian Decker
3a399fd55c backup: Temporarily use a wrapper of the node_factory
The node_factory from pyln-client clears the node directory when provisioning
a new node, which interferes with the backup-cli creation of the initial
snapshot. This just disables the cleaning until we can release the next
pyln-client version
2020-04-12 19:35:21 +02:00
Christian Decker
0ed717b3d1 backup: Add restore functionality to the backup-cli utility 2020-04-12 19:35:21 +02:00
Christian Decker
a68758b1d0 backup: Implement the restore method for all backends
Also implements the `stream_changes` function in the FileBackend which is used
by `restore` to get all the changes.
2020-04-12 19:35:21 +02:00
Christian Decker
caa5c0a16b backup: Add the version to the FileBackend format 2020-04-12 19:35:21 +02:00
Christian Decker
8f2eea2840 backend: Make sure that initialize initializes the metadata 2020-04-12 19:35:21 +02:00
Christian Decker
1e6b55b3cb backup: If a database exists write an intial snapshot 2020-04-12 19:35:21 +02:00
Christian Decker
a44284d04d pytest: Test starting without having initialized the backup 2020-04-12 19:35:21 +02:00
Christian Decker
c8502cd09d backup: Add missing Click dependency for the backup-cli tool 2020-04-12 19:35:21 +02:00
Christian Decker
4c146bf2a4 backend: Use th backup URL in backup-cli and add snapshot type to FileBackend 2020-04-12 19:35:21 +02:00
Christian Decker
c58e675877 backup: Ensure at startup that the backup backend was initialized 2020-04-12 19:35:21 +02:00
Christian Decker
97c702731a backup: Add a bit more documentation to the abstract backend class 2020-04-12 19:35:21 +02:00
Christian Decker
4e19c32444 backup: Add backup-cli tool and use it to initialize the backups 2020-04-12 19:35:21 +02:00
Christian Decker
c049069cff backup: Initial version of the backup plugin 2020-04-12 19:35:21 +02:00