datastore: add generation, simple atomicity.

We add a generation counter, and allow update or del conditional
on a given generation.

Formalizes error codes, too, since we have more now.

Suggested-by: @shesek
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

common/jsonrpc_errors.h

@@ -86,6 +86,13 @@ static const errcode_t OFFER_ROUTE_NOT_FOUND = 1003;
 static const errcode_t OFFER_BAD_INVREQ_REPLY = 1004;
 static const errcode_t OFFER_TIMEOUT = 1005;
 
+/* Errors from datastore command */
+static const errcode_t DATASTORE_DEL_DOES_NOT_EXIST = 1200;
+static const errcode_t DATASTORE_DEL_WRONG_GENERATION = 1201;
+static const errcode_t DATASTORE_UPDATE_ALREADY_EXISTS = 1202;
+static const errcode_t DATASTORE_UPDATE_DOES_NOT_EXIST = 1203;
+static const errcode_t DATASTORE_UPDATE_WRONG_GENERATION = 1204;
+
 /* Errors from wait* commands */
 static const errcode_t WAIT_TIMEOUT = 2000;
 

doc/lightning-datastore.7

@@ -3,7 +3,7 @@
 lightning-datastore - Command for storing (plugin) data
 .SH SYNOPSIS
 
-\fBdatastore\fR \fIkey\fR [\fIstring\fR] [\fIhex\fR] [\fImode\fR]
+\fBdatastore\fR \fIkey\fR [\fIstring\fR] [\fIhex\fR] [\fImode\fR] [\fIgeneration\fR]
 
 .SH DESCRIPTION
 
@@ -21,6 +21,12 @@ plugin name (e\.g\. \fBsummary.\fR) is recommended\.
 append this to what's already there) or "create-or-append" (append if
 anything is there, otherwise create)\.
 
+
+\fIgeneration\fR, if specified, means that the update will fail if the
+previously-existing data is not exactly that generation\.  This allows
+for simple atomicity\.  This is only legal with \fImode\fR "must-replace"
+or "must-append"\.
+
 .SH RETURN VALUE
 
 On success, an object is returned, containing:
@@ -29,20 +35,25 @@ On success, an object is returned, containing:
 .IP \[bu]
 \fBkey\fR (string): The key which has been added to the datastore
 .IP \[bu]
+\fBgeneration\fR (u64): The number of times this has been updated
+.IP \[bu]
 \fBhex\fR (hex): The hex data which has been added to the datastore
 .IP \[bu]
 \fBstring\fR (string, optional): The data as a string, if it's valid utf-8
 
 .RE
 
-The main cause of failure is an already-existing entry\.
-
-
 The following error codes may occur:
 
 .RS
 .IP \[bu]
--32602: invalid parameters, including already-existing/not-existing key\.
+1202: The key already exists (and mode said it must not)
+.IP \[bu]
+1203: The key does not exist (and mode said it must)
+.IP \[bu]
+1204: The generation was wrong (and generation was specified)
+.IP \[bu]
+-32602: invalid parameters
 
 .RE
 .SH AUTHOR
@@ -57,4 +68,4 @@ Rusty Russell \fI<rusty@rustcorp.com.au\fR> is mainly responsible\.
 
 Main web site: \fIhttps://github.com/ElementsProject/lightning\fR
 
-\" SHA256STAMP:4bb1369465ffb76e8e1962bd9e242159e579bb3af6e01c9d1461e519d8721769
+\" SHA256STAMP:0ef09e6f98d7e34e7d8339351c29ffc70be71fbf9f05f581488e3c7f603d3721

doc/lightning-datastore.7.md

@@ -4,7 +4,7 @@ lightning-datastore -- Command for storing (plugin) data
 SYNOPSIS
 --------
 
-**datastore** *key* [*string*] [*hex*] [*mode*]
+**datastore** *key* [*string*] [*hex*] [*mode*] [*generation*]
 
 DESCRIPTION
 -----------
@@ -21,20 +21,27 @@ plugin name (e.g. `summary.`) is recommended.
 append this to what's already there) or "create-or-append" (append if
 anything is there, otherwise create).
 
+*generation*, if specified, means that the update will fail if the
+previously-existing data is not exactly that generation.  This allows
+for simple atomicity.  This is only legal with *mode* "must-replace"
+or "must-append".
+
 RETURN VALUE
 ------------
 
 [comment]: # (GENERATE-FROM-SCHEMA-START)
 On success, an object is returned, containing:
 - **key** (string): The key which has been added to the datastore
+- **generation** (u64): The number of times this has been updated
 - **hex** (hex): The hex data which has been added to the datastore
 - **string** (string, optional): The data as a string, if it's valid utf-8
 [comment]: # (GENERATE-FROM-SCHEMA-END)
 
-The main cause of failure is an already-existing entry.
-
 The following error codes may occur:
-- -32602: invalid parameters, including already-existing/not-existing key.
+- 1202: The key already exists (and mode said it must not)
+- 1203: The key does not exist (and mode said it must)
+- 1204: The generation was wrong (and generation was specified)
+- -32602: invalid parameters
 
 AUTHOR
 ------
@@ -51,4 +58,4 @@ RESOURCES
 
 Main web site: <https://github.com/ElementsProject/lightning>
 
-[comment]: # ( SHA256STAMP:5eda4592b0a5e893853ea15ce7e800bb94e3a26ebd932507c2a55890f56fee14)
+[comment]: # ( SHA256STAMP:0867f9910b75ef66e640a92aad55dbab7ce0b3278fd1fb200f91c2a1a6164409)

doc/lightning-deldatastore.7

@@ -3,7 +3,7 @@
 lightning-deldatastore - Command for removing (plugin) data
 .SH SYNOPSIS
 
-\fBdeldatastore\fR \fIkey\fR
+\fBdeldatastore\fR \fIkey\fR [\fIgeneration\fR]
 
 .SH DESCRIPTION
 
@@ -11,7 +11,8 @@ The \fBdeldatastore\fR RPC command allows plugins to delete data it has
 stored in the c-lightning database\.
 
 
-The command fails if the \fIkey\fR isn't present\.
+The command fails if the \fIkey\fR isn't present, or if \fIgeneration\fR
+is specified and the generation of the data does not exactly match\.
 
 .SH RETURN VALUE
 
@@ -21,20 +22,23 @@ On success, an object is returned, containing:
 .IP \[bu]
 \fBkey\fR (string): The key which has been removed from the datastore
 .IP \[bu]
+\fBgeneration\fR (u64): The number of times this has been updated
+.IP \[bu]
 \fBhex\fR (hex): The hex data which has removed from the datastore
 .IP \[bu]
 \fBstring\fR (string, optional): The data as a string, if it's valid utf-8
 
 .RE
 
-The main cause of failure is an non-existing entry\.
-
-
 The following error codes may occur:
 
 .RS
 .IP \[bu]
--32602: invalid parameters, including non-existing key\.
+1200: the key does not exist
+.IP \[bu]
+1201: the key does exist, but the generation is wrong
+.IP \[bu]
+-32602: invalid parameters
 
 .RE
 .SH AUTHOR
@@ -49,4 +53,4 @@ Rusty Russell \fI<rusty@rustcorp.com.au\fR> is mainly responsible\.
 
 Main web site: \fIhttps://github.com/ElementsProject/lightning\fR
 
-\" SHA256STAMP:8e1a383ed176a0b7f8b849bf2bb05f5caaaf0de4f375afd38cbc668f1f17d9a2
+\" SHA256STAMP:784f58fc76fc32b92d043b67b0b7efb88534dd29a7fabda2d705cdc0611e3c11

doc/lightning-deldatastore.7.md

@@ -4,7 +4,7 @@ lightning-deldatastore -- Command for removing (plugin) data
 SYNOPSIS
 --------
 
-**deldatastore** *key*
+**deldatastore** *key* [*generation*]
 
 DESCRIPTION
 -----------
@@ -12,7 +12,8 @@ DESCRIPTION
 The **deldatastore** RPC command allows plugins to delete data it has
 stored in the c-lightning database.
 
-The command fails if the *key* isn't present.
+The command fails if the *key* isn't present, or if *generation*
+is specified and the generation of the data does not exactly match.
 
 RETURN VALUE
 ------------
@@ -20,14 +21,15 @@ RETURN VALUE
 [comment]: # (GENERATE-FROM-SCHEMA-START)
 On success, an object is returned, containing:
 - **key** (string): The key which has been removed from the datastore
+- **generation** (u64): The number of times this has been updated
 - **hex** (hex): The hex data which has removed from the datastore
 - **string** (string, optional): The data as a string, if it's valid utf-8
 [comment]: # (GENERATE-FROM-SCHEMA-END)
 
-The main cause of failure is an non-existing entry.
-
 The following error codes may occur:
-- -32602: invalid parameters, including non-existing key.
+- 1200: the key does not exist
+- 1201: the key does exist, but the generation is wrong
+- -32602: invalid parameters
 
 AUTHOR
 ------
@@ -44,4 +46,4 @@ RESOURCES
 
 Main web site: <https://github.com/ElementsProject/lightning>
 
-[comment]: # ( SHA256STAMP:cc1dedfded4902f59879665e95a1a877c8c72c0e217a3db3de3ae8dde859e67a)
+[comment]: # ( SHA256STAMP:ca2b7b8f45b3ecd6332978599c803e38c4f80945119a777cb8ae346cbf063b10)

doc/lightning-listdatastore.7

@@ -22,6 +22,8 @@ On success, an object containing \fBdatastore\fR is returned\.  It is an array o
 .IP \[bu]
 \fBkey\fR (string): The key which from the datastore
 .IP \[bu]
+\fBgeneration\fR (u64): The number of times this has been updated
+.IP \[bu]
 \fBhex\fR (hex): The hex data from the datastore
 .IP \[bu]
 \fBstring\fR (string, optional): The data as a string, if it's valid utf-8
@@ -47,4 +49,4 @@ Rusty Russell \fI<rusty@rustcorp.com.au\fR> is mainly responsible\.
 
 Main web site: \fIhttps://github.com/ElementsProject/lightning\fR
 
-\" SHA256STAMP:bcc83095fc1695b0c81a2763109e280d711e29edbc395672314d052f9d99a72c
+\" SHA256STAMP:b4128fc60690b3161eb76295e98f042c7be0142342bffa461c4f50f223c10684

doc/lightning-listdatastore.7.md

@@ -21,6 +21,7 @@ RETURN VALUE
 [comment]: # (GENERATE-FROM-SCHEMA-START)
 On success, an object containing **datastore** is returned.  It is an array of objects, where each object contains:
 - **key** (string): The key which from the datastore
+- **generation** (u64): The number of times this has been updated
 - **hex** (hex): The hex data from the datastore
 - **string** (string, optional): The data as a string, if it's valid utf-8
 [comment]: # (GENERATE-FROM-SCHEMA-END)
@@ -43,4 +44,4 @@ RESOURCES
 
 Main web site: <https://github.com/ElementsProject/lightning>
 
-[comment]: # ( SHA256STAMP:b93d725952e9ac5134dc25711d9bfbbd8b719e8ee8592f27bd1becbb56f89971)
+[comment]: # ( SHA256STAMP:a6503e3d2da8f9a35a0d461b5b93248f3fea306371ad62f98df613efea51959d)

doc/schemas/datastore.schema.json

@@ -2,12 +2,16 @@
     "$schema": "http://json-schema.org/draft-07/schema#",
     "type": "object",
     "additionalProperties": false,
-    "required": [ "key", "hex" ],
+    "required": [ "key", "hex", "generation" ],
     "properties": {
         "key": {
             "type": "string",
             "description": "The key which has been added to the datastore"
         },
+        "generation": {
+            "type": "u64",
+            "description": "The number of times this has been updated"
+        },
         "hex": {
             "type": "hex",
             "description": "The hex data which has been added to the datastore"

doc/schemas/deldatastore.schema.json

@@ -2,12 +2,16 @@
     "$schema": "http://json-schema.org/draft-07/schema#",
     "type": "object",
     "additionalProperties": false,
-    "required": [ "key", "hex" ],
+    "required": [ "key", "hex", "generation" ],
     "properties": {
         "key": {
             "type": "string",
             "description": "The key which has been removed from the datastore"
         },
+        "generation": {
+            "type": "u64",
+            "description": "The number of times this has been updated"
+        },
         "hex": {
             "type": "hex",
             "description": "The hex data which has removed from the datastore"

doc/schemas/listdatastore.schema.json

@@ -9,12 +9,16 @@
 	    "items": {
 		"type": "object",
 		"additionalProperties": false,
-		"required": [ "key", "hex" ],
+		"required": [ "key", "hex", "generation" ],
 		"properties": {
 		    "key": {
 			"type": "string",
 			"description": "The key which from the datastore"
 		    },
+		    "generation": {
+			"type": "u64",
+			"description": "The number of times this has been updated"
+		    },
 		    "hex": {
 			"type": "hex",
 			"description": "The hex data from the datastore"

lightningd/datastore.c

@@ -5,10 +5,12 @@
 #include <wallet/wallet.h>
 
 static void json_add_datastore(struct json_stream *response,
-			       const char *key, const u8 *data)
+			       const char *key, const u8 *data,
+			       u64 generation)
 {
 	const char *str;
 	json_add_string(response, "key", key);
+	json_add_u64(response, "generation", generation);
 	json_add_hex(response, "hex", data, tal_bytelen(data));
 	str = utf8_str(response, data, tal_bytelen(data));
 	if (str)
@@ -58,12 +60,14 @@ static struct command_result *json_datastore(struct command *cmd,
 	const char *key, *strdata;
 	u8 *data, *prevdata;
 	enum ds_mode *mode;
+	u64 *generation, actual_gen;
 
 	if (!param(cmd, buffer, params,
 		   p_req("key", param_string, &key),
 		   p_opt("string", param_string, &strdata),
 		   p_opt("hex", param_bin_from_hex, &data),
 		   p_opt_def("mode", param_mode, &mode, DS_MUST_NOT_EXIST),
+		   p_opt("generation", param_u64, &generation),
 		   NULL))
 		return command_param_failed();
 
@@ -78,15 +82,25 @@ static struct command_result *json_datastore(struct command *cmd,
 					    "Must have either hex or string");
 	}
 
-	prevdata = wallet_datastore_fetch(cmd, cmd->ld->wallet, key);
+	if (generation && !(*mode & DS_MUST_EXIST))
-	if ((*mode & DS_MUST_NOT_EXIST) && prevdata)
 		return command_fail(cmd, JSONRPC2_INVALID_PARAMS,
+				    "generation only valid with must-replace"
+				    " or must-append");
+
+	prevdata = wallet_datastore_fetch(cmd, cmd->ld->wallet, key,
+					  &actual_gen);
+	if ((*mode & DS_MUST_NOT_EXIST) && prevdata)
+		return command_fail(cmd, DATASTORE_UPDATE_ALREADY_EXISTS,
 				    "Key already exists");
 
 	if ((*mode & DS_MUST_EXIST) && !prevdata)
-		return command_fail(cmd, JSONRPC2_INVALID_PARAMS,
+		return command_fail(cmd, DATASTORE_UPDATE_DOES_NOT_EXIST,
 				    "Key does not exist");
 
+	if (generation && actual_gen != *generation)
+		return command_fail(cmd, DATASTORE_UPDATE_WRONG_GENERATION,
+				    "generation is different");
+
 	if ((*mode & DS_APPEND) && prevdata) {
 		size_t prevlen = tal_bytelen(prevdata);
 		tal_resize(&prevdata, prevlen + tal_bytelen(data));
@@ -94,13 +108,16 @@ static struct command_result *json_datastore(struct command *cmd,
 		data = prevdata;
 	}
 
-	if (prevdata)
+	if (prevdata) {
 		wallet_datastore_update(cmd->ld->wallet, key, data);
-	else
+		actual_gen++;
+	} else {
 		wallet_datastore_create(cmd->ld->wallet, key, data);
+		actual_gen = 0;
+	}
 
 	response = json_stream_success(cmd);
-	json_add_datastore(response, key, data);
+	json_add_datastore(response, key, data, actual_gen);
 	return command_success(cmd, response);
 }
 
@@ -112,6 +129,7 @@ static struct command_result *json_listdatastore(struct command *cmd,
 	struct json_stream *response;
 	const char *key;
 	const u8 *data;
+	u64 generation;
 
 	if (!param(cmd, buffer, params,
 		   p_opt("key", param_string, &key),
@@ -121,22 +139,24 @@ static struct command_result *json_listdatastore(struct command *cmd,
 	response = json_stream_success(cmd);
 	json_array_start(response, "datastore");
 	if (key) {
-		data = wallet_datastore_fetch(cmd, cmd->ld->wallet, key);
+		data = wallet_datastore_fetch(cmd, cmd->ld->wallet, key,
+					      &generation);
 		if (data) {
 			json_object_start(response, NULL);
-			json_add_datastore(response, key, data);
+			json_add_datastore(response, key, data, generation);
 			json_object_end(response);
 		}
 	} else {
 		struct db_stmt *stmt;
 
 		for (stmt = wallet_datastore_first(cmd, cmd->ld->wallet,
-						   &key, &data);
+						   &key, &data, &generation);
 		     stmt;
 		     stmt = wallet_datastore_next(cmd, cmd->ld->wallet,
-						  stmt, &key, &data)) {
+						  stmt, &key, &data,
+						  &generation)) {
 			json_object_start(response, NULL);
-			json_add_datastore(response, key, data);
+			json_add_datastore(response, key, data, generation);
 			json_object_end(response);
 		}
 	}
@@ -152,19 +172,29 @@ static struct command_result *json_deldatastore(struct command *cmd,
 	struct json_stream *response;
 	const char *key;
 	u8 *data;
+	u64 *generation;
+	u64 actual_gen;
 
 	if (!param(cmd, buffer, params,
 		   p_req("key", param_string, &key),
+		   p_opt("generation", param_u64, &generation),
 		   NULL))
 		return command_param_failed();
 
-	data = wallet_datastore_remove(cmd, cmd->ld->wallet, key);
+	if (generation) {
+		data = wallet_datastore_fetch(cmd, cmd->ld->wallet, key,
+					      &actual_gen);
+		if (data && actual_gen != *generation)
+			return command_fail(cmd, DATASTORE_DEL_WRONG_GENERATION,
+					    "generation is different");
+	}
+	data = wallet_datastore_remove(cmd, cmd->ld->wallet, key, &actual_gen);
 	if (!data)
-		return command_fail(cmd, JSONRPC2_INVALID_PARAMS,
+		return command_fail(cmd, DATASTORE_DEL_DOES_NOT_EXIST,
 				    "Key does not exist");
 
 	response = json_stream_success(cmd);
-	json_add_datastore(response, key, data);
+	json_add_datastore(response, key, data, actual_gen);
 	return command_success(cmd, response);
 }
 

tests/test_misc.py

@@ -2636,6 +2636,7 @@ def test_datastore(node_factory):
     # Add entries.
     somedata = b'somedata'.hex()
     somedata_expect = {'key': 'somekey',
+                       'generation': 0,
                        'hex': somedata,
                        'string': 'somedata'}
     assert l1.rpc.datastore(key='somekey', hex=somedata) == somedata_expect
@@ -2660,6 +2661,10 @@ def test_datastore(node_factory):
     l1.rpc.datastore(key='somekey', hex=somedata[-2:], mode="create-or-append")
     assert only_one(l1.rpc.listdatastore('somekey')['datastore'])['hex'] == somedata
 
+    # Generation will have increased due to three ops above.
+    somedata_expect['generation'] += 3
+    assert l1.rpc.listdatastore() == {'datastore': [somedata_expect]}
+
     # Can't replace or append non-existing records if we say not to
     with pytest.raises(RpcError, match='does not exist'):
         l1.rpc.datastore(key='otherkey', hex=somedata, mode="must-replace")
@@ -2669,6 +2674,7 @@ def test_datastore(node_factory):
 
     otherdata = b'otherdata'.hex()
     otherdata_expect = {'key': 'otherkey',
+                        'generation': 0,
                         'hex': otherdata,
                         'string': 'otherdata'}
     assert l1.rpc.datastore(key='otherkey', string='otherdata', mode="create-or-append") == otherdata_expect
@@ -2691,6 +2697,7 @@ def test_datastore(node_factory):
 
     # if it's not a string, won't print
     badstring_expect = {'key': 'badstring',
+                        'generation': 0,
                         'hex': '00'}
     assert l1.rpc.datastore(key='badstring', hex='00') == badstring_expect
     assert l1.rpc.listdatastore('badstring') == {'datastore': [badstring_expect]}
@@ -2700,3 +2707,27 @@ def test_datastore(node_factory):
     l1.restart()
 
     assert l1.rpc.listdatastore() == {'datastore': [otherdata_expect]}
+
+    # We can insist generation match on update.
+    with pytest.raises(RpcError, match='generation is different'):
+        l1.rpc.datastore(key='otherkey', hex='00', mode='must-replace',
+                         generation=otherdata_expect['generation'] + 1)
+
+    otherdata_expect['generation'] += 1
+    otherdata_expect['string'] += 'a'
+    otherdata_expect['hex'] += '61'
+    assert (l1.rpc.datastore(key='otherkey', string='otherdataa',
+                             mode='must-replace',
+                             generation=otherdata_expect['generation'] - 1)
+            == otherdata_expect)
+    assert l1.rpc.listdatastore() == {'datastore': [otherdata_expect]}
+
+    # We can insist generation match on delete.
+    with pytest.raises(RpcError, match='generation is different'):
+        l1.rpc.deldatastore(key='otherkey',
+                            generation=otherdata_expect['generation'] + 1)
+
+    assert (l1.rpc.deldatastore(key='otherkey',
+                                generation=otherdata_expect['generation'])
+            == otherdata_expect)
+    assert l1.rpc.listdatastore() == {'datastore': []}

wallet/db.c

@@ -752,6 +752,7 @@ static struct migration dbmigrations[] = {
     {SQL("CREATE TABLE datastore ("
 	 "  key TEXT,"
 	 "  data BLOB,"
+	 "  generation BIGINT,"
 	 "  PRIMARY KEY (key)"
 	 ");"),
      NULL},

wallet/db_postgres_sqlgen.c

@@ -1023,8 +1023,8 @@ struct db_query db_postgres_queries[] = {
          .readonly = false,
     },
     {
-         .name = "CREATE TABLE datastore (  key TEXT,  data BLOB,  PRIMARY KEY (key));",
+         .name = "CREATE TABLE datastore (  key TEXT,  data BLOB,  generation BIGINT,  PRIMARY KEY (key));",
-         .query = "CREATE TABLE datastore (  key TEXT,  data BYTEA,  PRIMARY KEY (key));",
+         .query = "CREATE TABLE datastore (  key TEXT,  data BYTEA,  generation BIGINT,  PRIMARY KEY (key));",
          .placeholders = 0,
          .readonly = false,
     },
@@ -2007,14 +2007,14 @@ struct db_query db_postgres_queries[] = {
          .readonly = true,
     },
     {
-         .name = "UPDATE datastore SET data=? WHERE key=?;",
+         .name = "UPDATE datastore SET data=?, generation=generation+1 WHERE key=?;",
-         .query = "UPDATE datastore SET data=$1 WHERE key=$2;",
+         .query = "UPDATE datastore SET data=$1, generation=generation+1 WHERE key=$2;",
          .placeholders = 2,
          .readonly = false,
     },
     {
-         .name = "INSERT INTO datastore VALUES (?, ?);",
+         .name = "INSERT INTO datastore VALUES (?, ?, 0);",
-         .query = "INSERT INTO datastore VALUES ($1, $2);",
+         .query = "INSERT INTO datastore VALUES ($1, $2, 0);",
          .placeholders = 2,
          .readonly = false,
     },
@@ -2025,14 +2025,14 @@ struct db_query db_postgres_queries[] = {
          .readonly = false,
     },
     {
-         .name = "SELECT data  FROM datastore WHERE key = ?;",
+         .name = "SELECT data, generation  FROM datastore WHERE key = ?;",
-         .query = "SELECT data  FROM datastore WHERE key = $1;",
+         .query = "SELECT data, generation  FROM datastore WHERE key = $1;",
          .placeholders = 1,
          .readonly = true,
     },
     {
-         .name = "SELECT key, data FROM datastore;",
+         .name = "SELECT key, data, generation FROM datastore;",
-         .query = "SELECT key, data FROM datastore;",
+         .query = "SELECT key, data, generation FROM datastore;",
          .placeholders = 0,
          .readonly = true,
     },
@@ -2068,4 +2068,4 @@ struct db_query db_postgres_queries[] = {
 
 #endif /* LIGHTNINGD_WALLET_GEN_DB_POSTGRES */
 
-// SHA256STAMP:5b2863e970ee24f2552e45224fed3f248badc9fd87ee3ab3ff0458fe3bcc0a83
+// SHA256STAMP:247577c68e8e536ee1736ddce425efb58ff609c1d3d4fbb1148b5c65b8922741

wallet/db_sqlite3_sqlgen.c

@@ -1023,8 +1023,8 @@ struct db_query db_sqlite3_queries[] = {
          .readonly = false,
     },
     {
-         .name = "CREATE TABLE datastore (  key TEXT,  data BLOB,  PRIMARY KEY (key));",
+         .name = "CREATE TABLE datastore (  key TEXT,  data BLOB,  generation BIGINT,  PRIMARY KEY (key));",
-         .query = "CREATE TABLE datastore (  key TEXT,  data BLOB,  PRIMARY KEY (key));",
+         .query = "CREATE TABLE datastore (  key TEXT,  data BLOB,  generation INTEGER,  PRIMARY KEY (key));",
          .placeholders = 0,
          .readonly = false,
     },
@@ -2007,14 +2007,14 @@ struct db_query db_sqlite3_queries[] = {
          .readonly = true,
     },
     {
-         .name = "UPDATE datastore SET data=? WHERE key=?;",
+         .name = "UPDATE datastore SET data=?, generation=generation+1 WHERE key=?;",
-         .query = "UPDATE datastore SET data=? WHERE key=?;",
+         .query = "UPDATE datastore SET data=?, generation=generation+1 WHERE key=?;",
          .placeholders = 2,
          .readonly = false,
     },
     {
-         .name = "INSERT INTO datastore VALUES (?, ?);",
+         .name = "INSERT INTO datastore VALUES (?, ?, 0);",
-         .query = "INSERT INTO datastore VALUES (?, ?);",
+         .query = "INSERT INTO datastore VALUES (?, ?, 0);",
          .placeholders = 2,
          .readonly = false,
     },
@@ -2025,14 +2025,14 @@ struct db_query db_sqlite3_queries[] = {
          .readonly = false,
     },
     {
-         .name = "SELECT data  FROM datastore WHERE key = ?;",
+         .name = "SELECT data, generation  FROM datastore WHERE key = ?;",
-         .query = "SELECT data  FROM datastore WHERE key = ?;",
+         .query = "SELECT data, generation  FROM datastore WHERE key = ?;",
          .placeholders = 1,
          .readonly = true,
     },
     {
-         .name = "SELECT key, data FROM datastore;",
+         .name = "SELECT key, data, generation FROM datastore;",
-         .query = "SELECT key, data FROM datastore;",
+         .query = "SELECT key, data, generation FROM datastore;",
          .placeholders = 0,
          .readonly = true,
     },
@@ -2068,4 +2068,4 @@ struct db_query db_sqlite3_queries[] = {
 
 #endif /* LIGHTNINGD_WALLET_GEN_DB_SQLITE3 */
 
-// SHA256STAMP:5b2863e970ee24f2552e45224fed3f248badc9fd87ee3ab3ff0458fe3bcc0a83
+// SHA256STAMP:247577c68e8e536ee1736ddce425efb58ff609c1d3d4fbb1148b5c65b8922741

wallet/statements_gettextgen.po

@@ -675,94 +675,94 @@ msgid "ALTER TABLE outputs ADD csv_lock INTEGER DEFAULT 1;"
 msgstr ""
 
 #: wallet/db.c:752
-msgid "CREATE TABLE datastore (  key TEXT,  data BLOB,  PRIMARY KEY (key));"
+msgid "CREATE TABLE datastore (  key TEXT,  data BLOB,  generation BIGINT,  PRIMARY KEY (key));"
 msgstr ""
 
-#: wallet/db.c:984
+#: wallet/db.c:985
 msgid "UPDATE vars SET intval = intval + 1 WHERE name = 'data_version' AND intval = ?"
 msgstr ""
 
-#: wallet/db.c:1084
+#: wallet/db.c:1085
 msgid "SELECT version FROM version LIMIT 1"
 msgstr ""
 
-#: wallet/db.c:1146
+#: wallet/db.c:1147
 msgid "UPDATE version SET version=?;"
 msgstr ""
 
-#: wallet/db.c:1154
+#: wallet/db.c:1155
 msgid "INSERT INTO db_upgrades VALUES (?, ?);"
 msgstr ""
 
-#: wallet/db.c:1166
+#: wallet/db.c:1167
 msgid "SELECT intval FROM vars WHERE name = 'data_version'"
 msgstr ""
 
-#: wallet/db.c:1193
+#: wallet/db.c:1194
 msgid "SELECT intval FROM vars WHERE name= ? LIMIT 1"
 msgstr ""
 
-#: wallet/db.c:1209
+#: wallet/db.c:1210
 msgid "UPDATE vars SET intval=? WHERE name=?;"
 msgstr ""
 
-#: wallet/db.c:1218
+#: wallet/db.c:1219
 msgid "INSERT INTO vars (name, intval) VALUES (?, ?);"
 msgstr ""
 
-#: wallet/db.c:1232
+#: wallet/db.c:1233
 msgid "UPDATE channels SET feerate_base = ?, feerate_ppm = ?;"
 msgstr ""
 
-#: wallet/db.c:1253
+#: wallet/db.c:1254
 msgid "UPDATE channels SET our_funding_satoshi = funding_satoshi WHERE funder = 0;"
 msgstr ""
 
-#: wallet/db.c:1269
+#: wallet/db.c:1270
 msgid "SELECT type, keyindex, prev_out_tx, prev_out_index, channel_id, peer_id, commitment_point FROM outputs WHERE scriptpubkey IS NULL;"
 msgstr ""
 
-#: wallet/db.c:1331
+#: wallet/db.c:1332
 msgid "UPDATE outputs SET scriptpubkey = ? WHERE prev_out_tx = ?    AND prev_out_index = ?"
 msgstr ""
 
-#: wallet/db.c:1356
+#: wallet/db.c:1357
 msgid "SELECT id, funding_tx_id, funding_tx_outnum FROM channels;"
 msgstr ""
 
-#: wallet/db.c:1375
+#: wallet/db.c:1376
 msgid "UPDATE channels SET full_channel_id = ? WHERE id = ?;"
 msgstr ""
 
-#: wallet/db.c:1396
+#: wallet/db.c:1397
 msgid "SELECT  channels.id, peers.node_id FROM  channels JOIN  peers ON (peers.id = channels.peer_id)"
 msgstr ""
 
-#: wallet/db.c:1429
+#: wallet/db.c:1430
 msgid "UPDATE channels SET  revocation_basepoint_local = ?, payment_basepoint_local = ?, htlc_basepoint_local = ?, delayed_payment_basepoint_local = ?, funding_pubkey_local = ? WHERE id = ?;"
 msgstr ""
 
-#: wallet/db.c:1462
+#: wallet/db.c:1463
 msgid "INSERT INTO channel_blockheights  (channel_id, hstate, blockheight) SELECT id, 4, 0 FROM channels WHERE funder = 0;"
 msgstr ""
 
-#: wallet/db.c:1470
+#: wallet/db.c:1471
 msgid "INSERT INTO channel_blockheights  (channel_id, hstate, blockheight) SELECT id, 14, 0 FROM channels WHERE funder = 1;"
 msgstr ""
 
-#: wallet/db.c:1482
+#: wallet/db.c:1483
 msgid "SELECT   c.id, p.node_id, c.fundingkey_remote, inflight.last_tx, inflight.last_sig, inflight.funding_satoshi, inflight.funding_tx_id FROM channels c  LEFT OUTER JOIN peers p   ON p.id = c.peer_id  LEFT OUTER JOIN   channel_funding_inflights inflight   ON c.id = inflight.channel_id WHERE inflight.last_tx IS NOT NULL;"
 msgstr ""
 
-#: wallet/db.c:1549
+#: wallet/db.c:1550
 msgid "UPDATE channel_funding_inflights SET last_tx = ? WHERE channel_id = ?   AND funding_tx_id = ?;"
 msgstr ""
 
-#: wallet/db.c:1573
+#: wallet/db.c:1574
 msgid "SELECT   c.id, p.node_id, c.last_tx, c.funding_satoshi, c.fundingkey_remote, c.last_sig FROM channels c  LEFT OUTER JOIN peers p  ON p.id = c.peer_id;"
 msgstr ""
 
-#: wallet/db.c:1640
+#: wallet/db.c:1641
 msgid "UPDATE channels SET last_tx = ? WHERE id = ?;"
 msgstr ""
 
@@ -1331,23 +1331,23 @@ msgid "SELECT status  FROM offers WHERE offer_id = ?;"
 msgstr ""
 
 #: wallet/wallet.c:4746
-msgid "UPDATE datastore SET data=? WHERE key=?;"
+msgid "UPDATE datastore SET data=?, generation=generation+1 WHERE key=?;"
 msgstr ""
 
 #: wallet/wallet.c:4757
-msgid "INSERT INTO datastore VALUES (?, ?);"
+msgid "INSERT INTO datastore VALUES (?, ?, 0);"
 msgstr ""
 
-#: wallet/wallet.c:4770
+#: wallet/wallet.c:4771
 msgid "DELETE FROM datastore WHERE key = ?"
 msgstr ""
 
-#: wallet/wallet.c:4785
+#: wallet/wallet.c:4787
-msgid "SELECT data  FROM datastore WHERE key = ?;"
+msgid "SELECT data, generation  FROM datastore WHERE key = ?;"
 msgstr ""
 
-#: wallet/wallet.c:4806
+#: wallet/wallet.c:4812
-msgid "SELECT key, data FROM datastore;"
+msgid "SELECT key, data, generation FROM datastore;"
 msgstr ""
 
 #: wallet/test/run-db.c:126
@@ -1365,4 +1365,4 @@ msgstr ""
 #: wallet/test/run-wallet.c:1753
 msgid "INSERT INTO channels (id) VALUES (1);"
 msgstr ""
-#  SHA256STAMP:33c97b729f031ff86abf939de5571a7e5a9cf3d0795bf975189d90ace76d28a1
+#  SHA256STAMP:f68886ac022d1170ef8a4e137a6f4fedea23a7cd06a735418e5f635bb4224a58

wallet/wallet.c

@@ -4743,7 +4743,7 @@ void wallet_datastore_update(struct wallet *w, const char *key, const u8 *data)
 	struct db_stmt *stmt;
 
 	stmt = db_prepare_v2(w->db,
-			     SQL("UPDATE datastore SET data=? WHERE key=?;"));
+			     SQL("UPDATE datastore SET data=?, generation=generation+1 WHERE key=?;"));
 	db_bind_talarr(stmt, 0, data);
 	db_bind_text(stmt, 1, key);
 	db_exec_prepared_v2(take(stmt));
@@ -4754,16 +4754,17 @@ void wallet_datastore_create(struct wallet *w, const char *key, const u8 *data)
 	struct db_stmt *stmt;
 
 	stmt = db_prepare_v2(w->db,
-			     SQL("INSERT INTO datastore VALUES (?, ?);"));
+			     SQL("INSERT INTO datastore VALUES (?, ?, 0);"));
 
 	db_bind_text(stmt, 0, key);
 	db_bind_talarr(stmt, 1, data);
 	db_exec_prepared_v2(take(stmt));
 }
 
-u8 *wallet_datastore_remove(const tal_t *ctx, struct wallet *w, const char *key)
+u8 *wallet_datastore_remove(const tal_t *ctx, struct wallet *w, const char *key,
+			    u64 *generation)
 {
-	u8 *data = wallet_datastore_fetch(ctx, w, key);
+	u8 *data = wallet_datastore_fetch(ctx, w, key, generation);
 	if (data) {
 		struct db_stmt *stmt;
 
@@ -4776,21 +4777,24 @@ u8 *wallet_datastore_remove(const tal_t *ctx, struct wallet *w, const char *key)
 }
 
 u8 *wallet_datastore_fetch(const tal_t *ctx,
-			   struct wallet *w, const char *key)
+			   struct wallet *w, const char *key,
+			   u64 *generation)
 {
 	struct db_stmt *stmt;
 	u8 *data;
 
 	/* Test if already exists. */
-	stmt = db_prepare_v2(w->db, SQL("SELECT data"
+	stmt = db_prepare_v2(w->db, SQL("SELECT data, generation"
 					"  FROM datastore"
 					" WHERE key = ?;"));
 	db_bind_text(stmt, 0, key);
 	db_query_prepared(stmt);
 
-	if (db_step(stmt))
+	if (db_step(stmt)) {
 		data = db_column_talarr(ctx, stmt, 0);
-	else
+		if (generation)
+			*generation = db_column_u64(stmt, 1);
+	} else
 		data = NULL;
 	tal_free(stmt);
 	return data;
@@ -4799,27 +4803,31 @@ u8 *wallet_datastore_fetch(const tal_t *ctx,
 struct db_stmt *wallet_datastore_first(const tal_t *ctx,
 				       struct wallet *w,
 				       const char **key,
-				       const u8 **data)
+				       const u8 **data,
+				       u64 *generation)
 {
 	struct db_stmt *stmt;
 
-	stmt = db_prepare_v2(w->db, SQL("SELECT key, data FROM datastore;"));
+	stmt = db_prepare_v2(w->db,
+			     SQL("SELECT key, data, generation FROM datastore;"));
 	db_query_prepared(stmt);
 
-	return wallet_datastore_next(ctx, w, stmt, key, data);
+	return wallet_datastore_next(ctx, w, stmt, key, data, generation);
 }
 
 struct db_stmt *wallet_datastore_next(const tal_t *ctx,
 				      struct wallet *w,
 				      struct db_stmt *stmt,
 				      const char **key,
-				      const u8 **data)
+				      const u8 **data,
+				      u64 *generation)
 {
 	if (!db_step(stmt))
 		return tal_free(stmt);
 
 	*key = tal_strdup(ctx, (const char *)db_column_text(stmt, 0));
 	*data = db_column_talarr(ctx, stmt, 1);
+	*generation = db_column_u64(stmt, 2);
 
 	return stmt;
 }

wallet/wallet.h

@@ -1531,7 +1531,7 @@ void wallet_offer_mark_used(struct db *db, const struct sha256 *offer_id)
 	NO_NULL_ARGS;
 
 /**
- * Add an new key/value to the datastore.
+ * Add an new key/value to the datastore (generation 0)
  * @w: the wallet
  * @key: the first key (if returns non-NULL)
  * @data: the first data (if returns non-NULL)
@@ -1551,21 +1551,25 @@ void wallet_datastore_update(struct wallet *w, const char *key, const u8 *data);
  * @ctx: the tal ctx to allocate return off
  * @w: the wallet
  * @key: the key
+ * @generation: the generation of deleted record
  *
  * Returns NULL if the key was not in the store.
  */
-u8 *wallet_datastore_remove(const tal_t *ctx, struct wallet *w, const char *key);
+u8 *wallet_datastore_remove(const tal_t *ctx, struct wallet *w, const char *key,
+			    u64 *generation);
 
 /**
- * Retreive a value from the datastore.
+ * Retrieve a value from the datastore.
  * @ctx: the tal ctx to allocate return off
  * @w: the wallet
  * @key: the first key (if returns non-NULL)
+ * @generation: the generation (if returns non-NULL), or NULL.
  *
  * Returns NULL if the key is not in the store.
  */
 u8 *wallet_datastore_fetch(const tal_t *ctx,
-			   struct wallet *w, const char *key);
+			   struct wallet *w, const char *key,
+			   u64 *generation);
 
 /**
  * Iterate through the datastore.
@@ -1573,6 +1577,7 @@ u8 *wallet_datastore_fetch(const tal_t *ctx,
  * @w: the wallet
  * @key: the first key (if returns non-NULL)
  * @data: the first data (if returns non-NULL)
+ * @generation: the first generation (if returns non-NULL)
  *
  * Returns pointer to hand as @stmt to wallet_datastore_next(), or NULL.
  * If you choose not to call wallet_datastore_next() you must free it!
@@ -1580,15 +1585,17 @@ u8 *wallet_datastore_fetch(const tal_t *ctx,
 struct db_stmt *wallet_datastore_first(const tal_t *ctx,
 				       struct wallet *w,
 				       const char **key,
-				       const u8 **data);
+				       const u8 **data,
+				       u64 *generation);
 
 /**
  * Iterate through the datastore.
  * @ctx: the tal ctx to allocate off
  * @w: the wallet
  * @stmt: the previous statement.
- * @key: the first key (if returns non-NULL)
+ * @key: the key (if returns non-NULL)
- * @data: the first data (if returns non-NULL)
+ * @data: the data (if returns non-NULL)
+ * @generation: the generation (if returns non-NULL)
  *
  * Returns pointer to hand as @stmt to wallet_datastore_next(), or NULL.
  * If you choose not to call wallet_datastore_next() you must free it!
@@ -1597,6 +1604,7 @@ struct db_stmt *wallet_datastore_next(const tal_t *ctx,
 				      struct wallet *w,
 				      struct db_stmt *stmt,
 				      const char **key,
-				      const u8 **data);
+				      const u8 **data,
+				      u64 *generation);
 
 #endif /* LIGHTNING_WALLET_WALLET_H */