getroute: change definition (and pay default) for riskfactor.

Up until now, riskfactor was useless due to implementation bugs, and
also the default setting is wrong (too low to have an effect on
reasonable payment scenarios).

Let's simplify the definition (by assuming that P(failure) of a node
is 1), to make it a simple percentage.  I examined the current network
fees to see what would work, and under this definition, a default of
10 seems reasonable (equivalent to 1000 under the old definition).

It is *this* change which finally fixes our test case!  The riskfactor
is now 40msat (1500000 * 14 * 10 / 5259600 = 39.9), comparable with
worst-case fuzz is 50msat (1001 * 0.05 = 50).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

CHANGELOG.md

@@ -17,6 +17,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 - JSON API: `listpeers` now has a `direction` field in `channels`.
 - JSON API: `listchannels` now takes a `source` option to filter by node id.
 - JSON API: New command `paystatus` gives detailed information on `pay` commands.
+- JSON API: `getroute` `riskfactor` argument is simplified; `pay` now defaults to setting it to 10.
 
 ### Changed
 
@@ -40,6 +41,7 @@ changes.
 - Fixed occasional deadlock with peers when exchanging huge amounts of gossip.
 - You can no longer make giant unpayable "wumbo" invoices.
 - CLTV of total route now correctly evaluated when finding best route.
+- `riskfactor` arguments to `pay` and `getroute` now have an effect.
 
 ### Security
 

doc/lightning-getroute.7

@@ -2,12 +2,12 @@
 .\"     Title: lightning-getroute
 .\"    Author: [see the "AUTHOR" section]
 .\" Generator: DocBook XSL Stylesheets v1.79.1 <http://docbook.sf.net/>
-.\"      Date: 01/23/2019
+.\"      Date: 02/01/2019
 .\"    Manual: \ \&
 .\"    Source: \ \&
 .\"  Language: English
 .\"
-.TH "LIGHTNING\-GETROUTE" "7" "01/23/2019" "\ \&" "\ \&"
+.TH "LIGHTNING\-GETROUTE" "7" "02/01/2019" "\ \&" "\ \&"
 .\" -----------------------------------------------------------------
 .\" * Define some portability stuff
 .\" -----------------------------------------------------------------
@@ -36,9 +36,9 @@ lightning-getroute \- Command for routing a payment (low\-level)\&.
 .sp
 The \fBgetroute\fR RPC command attempts to find the best route for the payment of \fImsatoshi\fR to lightning node \fIid\fR, such that the payment will arrive at \fIid\fR with \fIcltv\fR\-blocks to spare (default 9)\&.
 .sp
-There are two considerations for how good a route is: how low the fees are, and how long your payment will get stuck if a node goes down during the process\&. The \fIriskfactor\fR floating\-point field controls this tradeoff; it is the annual cost of your funds being stuck (as a percentage), multiplied by the percentage chance of each node failing\&.
+There are two considerations for how good a route is: how low the fees are, and how long your payment will get stuck if a node goes down during the process\&. The \fIriskfactor\fR floating\-point field controls this tradeoff; it is the annual cost of your funds being stuck (as a percentage)\&.
 .sp
-For example, if you thought there was a 1% chance that a node would fail, and it would cost you 20% per annum if that happened, \fIriskfactor\fR would be 20\&.
+For example, if you thought the inconvenience of having funds stuck was worth 20% per annum interest, \fIriskfactor\fR would be 20\&.
 .sp
 If you didn\(cqt care about risk, \fIriskfactor\fR would be zero\&.
 .sp
@@ -55,15 +55,13 @@ The formula used is the following approximation:
 .RS 4
 .\}
 .nf
-hop\-risk = num\-hops x per\-hop\-risk
+risk\-fee = amount x blocks\-timeout x per\-block\-cost
-timeout\-cost = blocks\-timeout x per\-block\-cost
-risk\-fee = amount x hop\-risk x timeout\-cost
 .fi
 .if n \{\
 .RE
 .\}
 .sp
-We are given a \fIriskfactor\fR; expressed as two multiplied percentages is the same as fractions multiplied by 10000\&. There are 52596 blocks per year, thus \fIper\-block\-cost\fR x \fIper\-hop\-risk\fR is riskfactor\*(Aq divided by 5,259,600,000\&.
+We are given a \fIriskfactor\fR expressed as a percentage\&. There are 52596 blocks per year, thus \fIper\-block\-cost\fR is \fIriskfactor\fR divided by 5,259,600\&.
 .sp
 The final result is:
 .sp
@@ -71,26 +69,26 @@ The final result is:
 .RS 4
 .\}
 .nf
-risk\-fee = amount x num\-hops x blocks\-timeout x riskfactor / 5259600000
+risk\-fee = amount x blocks\-timeout x riskfactor / 5259600
 .fi
 .if n \{\
 .RE
 .\}
 .sp
-Here are the risk fees as a percentage of the amount sent, using various parameters\&. For comparison with actual fees, we assume nodes charge 0\&.05%:
+Here are the risk fees in millisatoshis, using various parameters\&. I assume a channel charges the default of 1000 millisatoshis plus 1 part\-per\-million\&. Common delay values on the network at 14 and 144\&.
 .TS
 allbox tab(:);
 ltB ltB ltB ltB ltB.
 T{
+Amount (msat)
+T}:T{
 Riskfactor
 T}:T{
-Nodes
+Delay
 T}:T{
-Delay per node
+Risk Fee
 T}:T{
-Risk Fee %
+Route fee
-T}:T{
-Route fee %
 T}
 .T&
 lt lt lt lt lt
@@ -101,160 +99,417 @@ lt lt lt lt lt
 lt lt lt lt lt
 lt lt lt lt lt
 lt lt lt lt lt
+lt lt lt lt lt
+lt lt lt lt lt
+lt lt lt lt lt
+lt lt lt lt lt
+lt lt lt lt lt
+lt lt lt lt lt
+lt lt lt lt lt
+lt lt lt lt lt
+lt lt lt lt lt
+lt lt lt lt lt
+lt lt lt lt lt
+lt lt lt lt lt
+lt lt lt lt lt
+lt lt lt lt lt
+lt lt lt lt lt
 lt lt lt lt lt.
 T{
 .sp
-0\&.001
+10,000
 T}:T{
 .sp
-5
-T}:T{
-.sp
-6
-T}:T{
-.sp
-0
-T}:T{
-.sp
-0\&.25
-T}
-T{
-.sp
 1
 T}:T{
 .sp
-5
+14
-T}:T{
-.sp
-6
 T}:T{
 .sp
 0
 T}:T{
 .sp
-0\&.25
+1001
 T}
 T{
 .sp
-1000
+10,000
-T}:T{
-.sp
-5
-T}:T{
-.sp
-6
-T}:T{
-.sp
-0\&.0029
-T}:T{
-.sp
-0\&.25
-T}
-T{
-.sp
-0\&.001
 T}:T{
 .sp
 10
 T}:T{
 .sp
-72
+14
 T}:T{
 .sp
 0
 T}:T{
 .sp
-0\&.5
+1001
 T}
 T{
 .sp
-1
+10,000
 T}:T{
 .sp
-10
+100
 T}:T{
 .sp
-72
+14
 T}:T{
 .sp
-0\&.0001
+2
 T}:T{
 .sp
-0\&.5
+1001
 T}
 T{
 .sp
+10,000
+T}:T{
+.sp
 1000
 T}:T{
 .sp
-10
+14
 T}:T{
 .sp
-72
+26
 T}:T{
 .sp
-0\&.1369
+1001
-T}:T{
-.sp
-0\&.5
 T}
 T{
 .sp
-0\&.001
+1,000,000
 T}:T{
 .sp
-20
+1
 T}:T{
 .sp
-1008
+14
+T}:T{
+.sp
+2
+T}:T{
+.sp
+1001
+T}
+T{
+.sp
+1,000,000
+T}:T{
+.sp
+10
+T}:T{
+.sp
+14
+T}:T{
+.sp
+26
+T}:T{
+.sp
+1001
+T}
+T{
+.sp
+1,000,000
+T}:T{
+.sp
+100
+T}:T{
+.sp
+14
+T}:T{
+.sp
+266
+T}:T{
+.sp
+1001
+T}
+T{
+.sp
+1,000,000
+T}:T{
+.sp
+1000
+T}:T{
+.sp
+14
+T}:T{
+.sp
+2661
+T}:T{
+.sp
+1001
+T}
+T{
+.sp
+100,000,000
+T}:T{
+.sp
+1
+T}:T{
+.sp
+14
+T}:T{
+.sp
+266
+T}:T{
+.sp
+1100
+T}
+T{
+.sp
+100,000,000
+T}:T{
+.sp
+10
+T}:T{
+.sp
+14
+T}:T{
+.sp
+2661
+T}:T{
+.sp
+1100
+T}
+T{
+.sp
+100,000,000
+T}:T{
+.sp
+100
+T}:T{
+.sp
+14
+T}:T{
+.sp
+26617
+T}:T{
+.sp
+1100
+T}
+T{
+.sp
+100,000,000
+T}:T{
+.sp
+1000
+T}:T{
+.sp
+14
+T}:T{
+.sp
+266179
+T}:T{
+.sp
+1100
+T}
+T{
+.sp
+10,000
+T}:T{
+.sp
+1
+T}:T{
+.sp
+144
 T}:T{
 .sp
 0
 T}:T{
 .sp
-1\&.0
+1001
 T}
 T{
 .sp
-1
+10,000
 T}:T{
 .sp
-20
+10
 T}:T{
 .sp
-1008
+144
 T}:T{
 .sp
-0\&.0077
+2
 T}:T{
 .sp
-1\&.0
+1001
 T}
 T{
 .sp
+10,000
+T}:T{
+.sp
+100
+T}:T{
+.sp
+144
+T}:T{
+.sp
+27
+T}:T{
+.sp
+1001
+T}
+T{
+.sp
+10,000
+T}:T{
+.sp
 1000
 T}:T{
 .sp
-20
+144
 T}:T{
 .sp
-1008
+273
 T}:T{
 .sp
-7\&.6660
+1001
+T}
+T{
+.sp
+1,000,000
 T}:T{
 .sp
-1\&.0
+1
+T}:T{
+.sp
+144
+T}:T{
+.sp
+27
+T}:T{
+.sp
+1001
+T}
+T{
+.sp
+1,000,000
+T}:T{
+.sp
+10
+T}:T{
+.sp
+144
+T}:T{
+.sp
+273
+T}:T{
+.sp
+1001
+T}
+T{
+.sp
+1,000,000
+T}:T{
+.sp
+100
+T}:T{
+.sp
+144
+T}:T{
+.sp
+2737
+T}:T{
+.sp
+1001
+T}
+T{
+.sp
+1,000,000
+T}:T{
+.sp
+1000
+T}:T{
+.sp
+144
+T}:T{
+.sp
+27378
+T}:T{
+.sp
+1001
+T}
+T{
+.sp
+100,000,000
+T}:T{
+.sp
+1
+T}:T{
+.sp
+144
+T}:T{
+.sp
+2737
+T}:T{
+.sp
+1100
+T}
+T{
+.sp
+100,000,000
+T}:T{
+.sp
+10
+T}:T{
+.sp
+144
+T}:T{
+.sp
+27378
+T}:T{
+.sp
+1100
+T}
+T{
+.sp
+100,000,000
+T}:T{
+.sp
+100
+T}:T{
+.sp
+144
+T}:T{
+.sp
+273785
+T}:T{
+.sp
+1100
+T}
+T{
+.sp
+100,000,000
+T}:T{
+.sp
+1000
+T}:T{
+.sp
+144
+T}:T{
+.sp
+2737850
+T}:T{
+.sp
+1100
 T}
 .TE
 .sp 1
 .SH "RECOMMENDED RISKFACTOR VALUES"
 .sp
-0\&.001 is a value for tie\-breaking in favor of shorter routes, but not really costing in any risk\&.
+The default \fIfuzz\fR factor is 5%, so as you can see from the table above, that tends to overwhelm the effect of \fIriskfactor\fR less than about 5\&.
 .sp
 1 is a conservative value for a stable lightning network with very few failures\&.
 .sp
 1000 is an aggressive value for trying to minimize timeouts at all costs\&.
+.sp
+The default for lightning\-pay(7) is 10, which starts to become a major factor for larger amounts, and is basically ignored for tiny ones\&.
 .SH "RETURN VALUE"
 .sp
 On success, a "route" array is returned\&. Each array element contains \fIid\fR (the node being routed through), \fImsatoshi\fR (the millisatoshis sent), and \fIdelay\fR (the number of blocks to timeout at this node)\&.

doc/lightning-getroute.7.txt

@@ -21,11 +21,10 @@ There are two considerations for how good a route is: how low the
 fees are, and how long your payment will get stuck if a node goes down
 during the process.  The 'riskfactor' floating-point field controls
 this tradeoff; it is the annual cost of your funds being stuck (as a
-percentage), multiplied by the percentage chance of each node failing.
+percentage).
 
-For example, if you thought there was a 1% chance that a node would
+For example, if you thought the inconvenience of having funds stuck was
-fail, and it would cost you 20% per annum if that happened,
+worth 20% per annum interest, 'riskfactor' would be 20.
-'riskfactor' would be 20.
 
 If you didn't care about risk, 'riskfactor' would be zero.
 
@@ -46,45 +45,59 @@ for the purposes of comparing routes.
 
 The formula used is the following approximation:
 ----
-hop-risk = num-hops x per-hop-risk
+risk-fee = amount x blocks-timeout x per-block-cost
-timeout-cost = blocks-timeout x per-block-cost
-risk-fee = amount x hop-risk x timeout-cost
 ----
 
-We are given a 'riskfactor'; expressed as two multiplied percentages
+We are given a 'riskfactor' expressed as a percentage.  There are 52596 blocks
-is the same as fractions multiplied by 10000.  There are 52596 blocks
+per year, thus 'per-block-cost' is 'riskfactor' divided by 5,259,600.
-per year, thus 'per-block-cost' x 'per-hop-risk' is riskfactor'
-divided by 5,259,600,000.
 
 The final result is:
 ----
-risk-fee = amount x num-hops x blocks-timeout x riskfactor / 5259600000
+risk-fee = amount x blocks-timeout x riskfactor / 5259600
 ----
 
-Here are the risk fees as a percentage of the amount sent, using
+Here are the risk fees in millisatoshis, using various parameters.  I
-various parameters.  For comparison with actual fees, we assume nodes
+assume a channel charges the default of 1000 millisatoshis plus 1
-charge 0.05%:
+part-per-million.  Common delay values on the network at 14 and 144.
 
 [options="header"]
 |=======================
-|Riskfactor	|Nodes | Delay per node        |Risk Fee %	|Route fee %
+|Amount (msat)	|Riskfactor	| Delay		|Risk Fee	|Route fee
-|0.001          |5	| 6			|0		|0.25
+|10,000		|1		| 14		|0		|1001
-|1		|5	| 6			|0		|0.25
+|10,000		|10		| 14		|0		|1001
-|1000	        |5	| 6			|0.0029		|0.25
+|10,000		|100		| 14		|2		|1001
+|10,000		|1000		| 14		|26		|1001
 
-|0.001		|10	| 72			|0		|0.5
+|1,000,000	|1		| 14		|2		|1001
-|1	        |10	| 72			|0.0001		|0.5
+|1,000,000	|10		| 14		|26		|1001
-|1000		|10	| 72			|0.1369		|0.5
+|1,000,000	|100		| 14		|266		|1001
+|1,000,000	|1000		| 14		|2661		|1001
 
-|0.001		|20	| 1008			|0		|1.0
+|100,000,000	|1		| 14		|266		|1100
-|1		|20	| 1008			|0.0077		|1.0
+|100,000,000	|10		| 14		|2661		|1100
-|1000		|20	| 1008			|7.6660		|1.0
+|100,000,000	|100		| 14		|26617		|1100
+|100,000,000	|1000		| 14		|266179		|1100
+
+|10,000		|1		| 144		|0		|1001
+|10,000		|10		| 144		|2		|1001
+|10,000		|100		| 144		|27		|1001
+|10,000		|1000		| 144		|273		|1001
+
+|1,000,000	|1		| 144		|27		|1001
+|1,000,000	|10		| 144		|273		|1001
+|1,000,000	|100		| 144		|2737		|1001
+|1,000,000	|1000		| 144		|27378		|1001
+
+|100,000,000	|1		| 144		|2737		|1100
+|100,000,000	|10		| 144		|27378		|1100
+|100,000,000	|100		| 144		|273785		|1100
+|100,000,000	|1000		| 144		|2737850	|1100
 |=======================
 
 RECOMMENDED RISKFACTOR VALUES
 -----------------------------
-0.001 is a value for tie-breaking in favor of shorter routes, but not really
+The default 'fuzz' factor is 5%, so as you can see from the table above,
-costing in any risk.
+that tends to overwhelm the effect of 'riskfactor' less than about 5.
 
 1 is a conservative value for a stable lightning network with very few
 failures.
@@ -92,6 +105,9 @@ failures.
 1000 is an aggressive value for trying to minimize timeouts at all
 costs.
 
+The default for lightning-pay(7) is 10, which starts to become a major
+factor for larger amounts, and is basically ignored for tiny ones.
+
 RETURN VALUE
 ------------
 

doc/lightning-pay.7

@@ -2,12 +2,12 @@
 .\"     Title: lightning-pay
 .\"    Author: [see the "AUTHOR" section]
 .\" Generator: DocBook XSL Stylesheets v1.79.1 <http://docbook.sf.net/>
-.\"      Date: 11/26/2018
+.\"      Date: 02/01/2019
 .\"    Manual: \ \&
 .\"    Source: \ \&
 .\"  Language: English
 .\"
-.TH "LIGHTNING\-PAY" "7" "11/26/2018" "\ \&" "\ \&"
+.TH "LIGHTNING\-PAY" "7" "02/01/2019" "\ \&" "\ \&"
 .\" -----------------------------------------------------------------
 .\" * Define some portability stuff
 .\" -----------------------------------------------------------------
@@ -34,7 +34,7 @@ lightning-pay \- Command for sending a payment to a BOLT11 invoice
 \fBpay\fR \fIbolt11\fR [\fImsatoshi\fR] [\fIdescription\fR] [\fIriskfactor\fR] [\fImaxfeepercent\fR] [\fIretry_for\fR] [\fImaxdelay\fR] [\fIexemptfee\fR]
 .SH "DESCRIPTION"
 .sp
-The \fBpay\fR RPC command attempts to find a route to the given destination, and send the funds it asks for\&. If the \fIbolt11\fR does not contain an amount, \fImsatoshi\fR is required, otherwise if it is specified it must be \fInull\fR\&. If \fIbolt11\fR contains a description hash (\fIh\fR field) \fIdescription\fR is required, otherwise it is unused\&. The \fIriskfactor\fR is described in detail in lightning\-getroute(7), and defaults to 1\&.0\&. The \fImaxfeepercent\fR limits the money paid in fees, and defaults to 0\&.5\&. The maxfeepercent\*(Aq is a percentage of the amount that is to be paid\&. The `exemptfee option can be used for tiny payments which would be dominated by the fee leveraged by forwarding nodes\&. Setting exemptfee allows the maxfeepercent check to be skipped on fees that are smaller than exemptfee (default: 5000 millisatoshi)\&.
+The \fBpay\fR RPC command attempts to find a route to the given destination, and send the funds it asks for\&. If the \fIbolt11\fR does not contain an amount, \fImsatoshi\fR is required, otherwise if it is specified it must be \fInull\fR\&. If \fIbolt11\fR contains a description hash (\fIh\fR field) \fIdescription\fR is required, otherwise it is unused\&. The \fIriskfactor\fR is described in detail in lightning\-getroute(7), and defaults to 10\&. The \fImaxfeepercent\fR limits the money paid in fees, and defaults to 0\&.5\&. The maxfeepercent\*(Aq is a percentage of the amount that is to be paid\&. The `exemptfee option can be used for tiny payments which would be dominated by the fee leveraged by forwarding nodes\&. Setting exemptfee allows the maxfeepercent check to be skipped on fees that are smaller than exemptfee (default: 5000 millisatoshi)\&.
 .sp
 The response will occur when the payment fails or succeeds\&. Once a payment has succeeded, calls to \fBpay\fR with the same \fIbolt11\fR will succeed immediately\&.
 .sp

doc/lightning-pay.7.txt

@@ -18,7 +18,7 @@ and send the funds it asks for.  If the 'bolt11' does not contain an amount,
 'msatoshi' is required, otherwise if it is specified it must be 'null'.  If
 'bolt11' contains a description hash ('h' field) 'description' is
 required, otherwise it is unused.  The 'riskfactor' is described in detail
-in lightning-getroute(7), and defaults to 1.0.
+in lightning-getroute(7), and defaults to 10.
 The 'maxfeepercent' limits the money paid in fees, and defaults to 0.5.
 The `maxfeepercent' is a percentage of the amount that is to be
 paid.

gossipd/routing.c

@@ -1528,7 +1528,7 @@ struct route_hop *get_route(const tal_t *ctx, struct routing_state *rstate,
 	}
 
 	route = find_route(ctx, rstate, source, destination, msatoshi,
-			   riskfactor / BLOCKS_PER_YEAR / 10000,
+			   riskfactor / BLOCKS_PER_YEAR / 100,
 			   fuzz, &base_seed, max_hops, &fee);
 
 	/* Now restore the capacity. */

plugins/pay.c

@@ -833,7 +833,7 @@ static struct command_result *handle_pay(struct command *cmd,
 		   p_req("bolt11", param_string, &b11str),
 		   p_opt("msatoshi", param_u64, &msatoshi),
 		   p_opt("description", param_string, &pc->desc),
-		   p_opt_def("riskfactor", param_double, &riskfactor, 1.0),
+		   p_opt_def("riskfactor", param_double, &riskfactor, 10),
 		   p_opt_def("maxfeepercent", param_percent, &maxfeepercent, 0.5),
 		   p_opt_def("retry_for", param_number, &retryfor, 60),
 		   p_opt_def("maxdelay", param_number, &maxdelay,

tests/test_pay.py

@@ -1341,7 +1341,6 @@ def test_pay_routeboost(node_factory, bitcoind):
         assert [h['channel'] for h in attempts[2]['routehint']] == [r['short_channel_id'] for r in routel3l5]
 
 
-@pytest.mark.xfail(strict=True)
 def test_pay_direct(node_factory, bitcoind):
     """Check that we prefer the direct route.
     """