Commit Graph

299 Commits

Author SHA1 Message Date
lollerfirst
ff3fdd5aef feat: implement exponential backoff for paid_invoices_stream (#778)
* feat: implement exponential backoff for paid_invoices_stream across all Lightning backends

- Add exponential backoff retry logic to CLN REST, LND REST, and LND gRPC backends
- Start with 1 second delay, exponentially increase up to 5 minutes maximum
- Reset delay to 1 second on successful reconnection
- Improve error logging to include retry delay information
- Replace fixed delays with adaptive backoff to handle network issues gracefully
- Prevents system overload during persistent connection problems

Resolves issues with rapid reconnection attempts that could overwhelm Lightning nodes during network instability.

* remove unused import

* feat: extend exponential backoff to all remaining backends and invoice listener

- Implement exponential backoff in LNbits paid_invoices_stream for both SSE and WebSocket modes
- Add exponential backoff guidance comments to Blink and Strike backends (not implemented)
- Apply exponential backoff to invoice_listener in tasks.py that calls paid_invoices_stream
- Ensure consistent retry behavior across all Lightning backend integrations
- Improve system resilience during network interruptions and backend failures

All backends and the invoice listener now use the same exponential backoff strategy:
- Start with 1 second delay, exponentially increase up to 5 minutes maximum
- Reset delay to 1 second on successful reconnection
- Enhanced error logging with retry delay information

* blink + strike remove comments

* remove hardcoded values in favor of settings

* immediate first retry
2025-09-08 16:14:19 +02:00
lollerfirst
e361cff5c1 management RPC follow up (#770)
* new protobuf file

* redis

* format
2025-08-05 12:52:00 +02:00
lollerfirst
92bf6311fb [BUGFIX] SIG_ALL check returns true for other sigflags (#767)
* solved the issue

* revised approach

* fix

* fix 2
2025-08-05 12:51:37 +02:00
lollerfirst
29571287b3 Mint Management gRPC Server (#723)
* settings

* fix name settings

* management rpc

* hook up the RPC server

* working

* format

* update build script fix import error

* remove accidental commit of vscode extension data

* working ✔

* \n

* add get mint quote get melt quote

* gRPC cli update quotes commands

* update mint melt quotes from cli

* comment under get cli command group

* keyset rotation not yet implemented

* try fix

* change back contact info default to be empty list

* fix import

* add server mTLS

* ll

* script for generating certificates

* rename settings

* move generation script

* do not save TTL expiry into Cache object, rather always load from settings.

* update lightning fees

* update auth limits

* auth rate limit cli

* optional arguemnts

* better error messages

* tests for db update mint/melt quotes

* start mint rpc tests

* add tos_url field to get-info grpc response

* format checks

* add types to click groups where it's needed

* tests on updating quotes

* fix tests

* skip updating mint quote state if on regtest

* test edge case

* unified test_add_remove_contact

* mark pytest-asyncio

* fix missing db argument

* hopefully no more silly errors

* fix test_db_update_mint_quote_state

* pass in the quote id string.

* add keyset rotation

* test for keyset rotation through gRPC command

* fix logger warning

* remove rotation test because it breaks other tests

* use different bolt11 invoices

* assert returned melt quote has quote

* is_postgres

* try different things

* skip if deprecated api

* format checks

* update .gitignore

* default location for certificates
2025-06-25 12:35:53 +02:00
callebtc
48823d673d Mint: adjust mint melt settings (#764)
* adjust mint melt settings

* .env.example

* ge=0, closes #756
2025-06-01 12:24:37 +02:00
callebtc
3e32dc40e3 add msat support for strike backend (#755) 2025-05-18 15:48:15 +02:00
callebtc
c6f7ab1e79 ledger watchdog: disable the abort queue for now (#754) 2025-05-14 13:04:28 +02:00
callebtc
fc0e3fe663 Mint: watchdog balance log and killswitch (#705)
* wip store balance

* store balances in watchdog worker

* move mint_auth_database setting

* auth db

* balances returned as Amount (instead of int)

* add test for balance change on invoice receive

* fix 1 test

* cancel tasks on shutdown

* watchdog can now abort

* remove wallet api server

* fix lndgrpc

* fix lnbits balance

* disable watchdog

* balance lnbits msat

* test db watcher with its own database connection

* init superclass only once

* wip: log balance in keysets table

* check max balance using new keyset balance

* fix test

* fix another test

* store fees in keysets

* format

* cleanup

* shorter

* add keyset migration to auth server

* fix fakewallet

* fix db tests

* fix postgres problems during migration 26 (mint)

* fix cln

* ledger

* working with pending

* super fast watchdog, errors

* test new pipeline

* delete walletapi

* delete unneeded files

* revert workflows
2025-05-11 20:29:13 +02:00
lollerfirst
619d06f0ab [PATCH] LND use_mission_control + exclude failing channels (#738)
* lnd_grpc multinut patch

* lndrest multinut patch

* mypy fixes

* fixes non escaped double quotes in error messages formats

* fix

* fix debug log with correct hops number

* correctly escape "hops"

* remove `ignored_pairs` constraint

* Apply suggestions from code review

change some error logs to debug

* add tests and some cleanup

---------

Co-authored-by: callebtc <93376500+callebtc@users.noreply.github.com>
2025-05-10 15:45:15 +02:00
callebtc
2135b96706 Testing: check if pipeline gets stuck because of startup task (#745)
* lets check if this gets stuck with postgres or not

* check if it works if I add sleeps

* replace startup_ledger with await ledger._check_pending_proofs_and_melt_quotes() in tests

* fix typo to trigger tests again
2025-05-10 15:04:56 +02:00
lollerfirst
213d5b1eee Mint Keyset Rotation (#734)
* move keysets methods into newly created `LedgerKeysets` + `maybe_update_derivation_path` on init

* keyset rotation

* fix error

* make format

* format

* test keyset rotation + fix bug

* format

* fix for multiple specified derivation paths

* specify return value
2025-04-25 11:37:49 +02:00
callebtc
7abfc68cfa SIG_ALL signature flag for P2PK (#735)
* n_sigs_refund working, tests added

* update requirements

* wip sigall

* wip

* sigall works

* add signatures for refund

* add mint p2pk tests

* add more p2pk tests

* fix tests

* sign htlc pubkeys as well

* fix htlc and add new test

* fix regtest

* fix new tests with deprecated

* remove asserts

* comments

* new wallet p2pk tests

* getting there

* add more tests

* fixes

* refactor htlc and p2pk validation

* reduce code

* melt with sigall

* fix htlcs

* fix deprecated api tests

* Update cashu/mint/conditions.py

Co-authored-by: lollerfirst <43107113+lollerfirst@users.noreply.github.com>

* refactor sigall validation

---------

Co-authored-by: lollerfirst <43107113+lollerfirst@users.noreply.github.com>
2025-04-25 11:37:19 +02:00
callebtc
e1220d2329 Cjbeery24/mint copy: Invoice checker background tasks (#722)
* #616: Removed blocking call from ledger startup. Instead added it to a background task that repeats every hour.

* make task interval configurable, remember task and cancel it on shutdown

* comments

* add sleep to tests because the background task is async

---------

Co-authored-by: Caleb Beery <cjbeery@gmail.com>
2025-04-10 21:25:18 +07:00
callebtc
76e2601efd Add terms of service URL to MintInformation class (#715)
* add tos_url according to https://github.com/cashubtc/nuts/pull/205

* add tos_url everywhere
2025-03-28 16:11:04 +01:00
callebtc
f783493e1a Fix mint migrations: Fill in missing keyset IDs in proofs and promises (#729)
* mint migrations: fill in missing keyset ids in proofs and promises

* do not throw exception

* add error where proofs are missing an id but no keyset was found

* fix balance view initialization
2025-03-27 11:33:20 +01:00
callebtc
bae4855915 NUT-04/05: add amount, unit, request to melt and mint quote responses (#719)
* add amount, unit, request to melt and mint responses

* make new fields optional to not break compat with old mints

* make new flags optional for backwards compat
2025-03-08 18:23:34 +00:00
callebtc
9305905a85 Add LightningPaymentFailedError exception (#706)
* add error code

* fix test
2025-03-06 00:04:37 +01:00
lollerfirst
f72a3f260f [FIX] NUT-15 mpp amount in millisats (#703)
* fix lndrest

* fix clnrest

* fix clnrest and lndrest

* lnd grpc fix

* wallet

* convert amount to millisats in CLI pay invoice

* fix tests

* format

* fix kw arg in regtest test

* fix payment quote validation check

* clean comment

* avoid overwriting variable

* deprecated response with amount

---------

Co-authored-by: callebtc <93376500+callebtc@users.noreply.github.com>
2025-03-05 23:47:03 +01:00
lollerfirst
0c40bbbc08 Balance Views Grouped By Keyset (#652)
* balance view per keyset + relative changes.
    Still lacking: settings changes

* import unit

* fix 0 balance

* settings

---------

Co-authored-by: callebtc <93376500+callebtc@users.noreply.github.com>
2025-02-17 12:55:28 +01:00
callebtc
23a706d873 Set correct blink min fee (#701)
* min fee

* fix tests

* amount_split test
2025-02-13 15:53:26 +01:00
lollerfirst
ec3db81cee Update Error Codes (#702)
* add new error codes

* use the new errors in the code

* fix unsorted import

* fix test wallet
2025-02-13 15:35:06 +01:00
callebtc
a0ef44dba0 Blind authentication (#675)
* auth server

* cleaning up

* auth ledger class

* class variables -> instance variables

* annotations

* add models and api route

* custom amount and api prefix

* add auth db

* blind auth token working

* jwt working

* clean up

* JWT works

* using openid connect server

* use oauth server with password flow

* new realm

* add keycloak docker

* hopefully not garbage

* auth works

* auth kinda working

* fix cli

* auth works for send and receive

* pass auth_db to Wallet

* auth in info

* refactor

* fix supported

* cache mint info

* fix settings and endpoints

* add description to .env.example

* track changes for openid connect client

* store mint in db

* store credentials

* clean up v1_api.py

* load mint info into auth wallet

* fix first login

* authenticate if refresh token fails

* clear auth also middleware

* use regex

* add cli command

* pw works

* persist keyset amounts

* add errors.py

* do not start auth server if disabled in config

* upadte poetry

* disvoery url

* fix test

* support device code flow

* adopt latest spec changes

* fix code flow

* mint max bat dynamic

* mypy ignore

* fix test

* do not serialize amount in authproof

* all auth flows working

* fix tests

* submodule

* refactor

* test

* dont sleep

* test

* add wallet auth tests

* test differently

* test only keycloak for now

* fix creds

* daemon

* fix test

* install everything

* install jinja

* delete wallet for every test

* auth: use global rate limiter

* test auth rate limit

* keycloak hostname

* move keycloak test data

* reactivate all tests

* add readme

* load proofs

* remove unused code

* remove unused code

* implement change suggestions by ok300

* add error codes

* test errors
2025-01-29 22:48:51 -06:00
lollerfirst
e9952e59ed [FIX] Reject Internal MPP Melt Quote Requests (#697)
* fix

* fix the fix

* make sure the mpp is supported before testing

* adjust message

---------

Co-authored-by: callebtc <93376500+callebtc@users.noreply.github.com>
2025-01-29 14:00:35 -06:00
lollerfirst
ea96fab9e3 [FIX] Specs conformant error codes (#693)
* error codes must follow spec

* make format
2025-01-21 17:36:22 -06:00
callebtc
4e7917fb20 Fix Redis teardown logic based on usage (#696)
* fix: tear down redis only if used

* fix: close redis only if its used
2025-01-21 17:35:24 -06:00
callebtc
ad7c6b8e0b Issue NUT-08 overpaid Lightning fees for melt quote checks on startup (#688)
* startup: do not rollback unknown melt quote states

* fix: provide overpaid fees on startup

* fix: check if outputs in db

* fix test: expect melt quote pending if payment state is unknown

* fix up comment
2025-01-21 17:28:41 -06:00
lollerfirst
d98d166df1 Support NUT-XX (signatures on quotes) for mint and wallet side (#670)
* nut-19 sign mint quote

* ephemeral key for quote

* `mint` adjustments + crypto/nut19.py

* wip: mint side working

* fix import

* post-merge fixups

* more fixes

* make format

* move nut19 to nuts directory

* `key` -> `privkey` and `pubkey`

* make format

* mint_info method for nut-19 support

* fix tests imports

* fix signature missing positional argument + fix db migration format not correctly escaped + pass in NUT-19 keypair to `request_mint` `request_mint_with_callback`

* make format

* fix `get_invoice_status`

* rename to xx

* nutxx -> nut20

* mypy

* remove `mint_quote_signature_required` as per spec

* wip edits

* clean up

* fix tests

* fix deprecated api tests

* fix redis tests

* fix cache tests

* fix regtest mint external

* fix mint regtest

* add test without signature

* test pubkeys in quotes

* wip

* add compat

---------

Co-authored-by: callebtc <93376500+callebtc@users.noreply.github.com>
2024-12-15 00:39:53 +01:00
lollerfirst
399c201552 NUT-19: Cached Requests and Responses (#624)
* fast-api-cache setup

* testing the cache

* fix

* still not working

* asynccontextmanager

* move test

* use redis & custom caching setup (like CDK)

* make format

* poetry lock

* fix format string + log when a cached response is found

* log when a cahced response is found

* fix tests

* poetry lock

* try tests on github

* use docker compose

* maybe we dont need docker

* fix types

* create_task instead of run

* how about we start postgres

* mint features

* format

* remove deprecated setex call

* use global expiry for all cached routes

* refactor feature map and set default to 1 week

* refactor feature construction

* Cache NUT-19

---------

Co-authored-by: callebtc <93376500+callebtc@users.noreply.github.com>
2024-12-03 16:03:01 +03:00
ok300
ee90d840ab Add HTTP compression middleware (#676)
* Add HTTP compression middleware

* Apply fixes from `make format`
2024-11-25 13:08:46 +01:00
callebtc
2b233fd67e fix: NUT-15 setting remove mpp boolean (#673)
* fix: NUT-15 setting remove mpp boolean

* remove mpp flag
2024-11-24 16:55:17 +01:00
callebtc
0a230a1aa3 catch websocket disconenct errors (#674) 2024-11-24 16:48:39 +01:00
callebtc
901b16713c add methods key to info endpoitn a la https://github.com/cashubtc/nuts/pull/190 (#672) 2024-11-15 16:21:54 +01:00
Pavol Rusnak
0de574e006 NUT-06: add urls field (#638) 2024-11-05 15:03:23 +01:00
callebtc
602687b215 refactor pubkey extraction and set n_sigs=1 for refund spend path (#644) 2024-11-05 15:02:54 +01:00
elnosh
9cdfba52a3 add nut-14 as supported in info endpoint (#647) 2024-11-05 13:52:09 +01:00
callebtc
05547e3710 Mint: Turn off locking by default (#662)
* turn off locking by default

* fix test
2024-11-04 10:52:33 +01:00
callebtc
e9d4689b4b Chore: update dev dependencies (#658)
* update dev dependencies

* test

* revert tests

* fix event loop scope

* ruff format

* add old test

* remove custom event loop

* default loop scope

* skip earlier

* trigger

* trigger again
2024-11-01 15:07:56 +01:00
callebtc
92627399a5 Wallet: store quotes (#657)
* wallet_quotes_wip

* fix quote in db

* fix subscription test

* clean up api

* fix api tests

* fix balance check
2024-11-01 13:27:27 +01:00
elnosh
fc9ed7c1fb include preimage and change in response from melt quote state check (#656) 2024-11-01 10:20:51 +01:00
Damian
b8dd43deaa Mint: settle mint-melt on same mint with different units externally (#651)
* WIP settle different units externally

* mint melt externally different units

* deprecated route return only sat

* comment

---------

Co-authored-by: callebtc <93376500+callebtc@users.noreply.github.com>
2024-10-30 21:59:57 +01:00
callebtc
09d007ec88 Refactor conditions and fix HTLC multisig (#643)
* refactor conditions and fix htlc multisig

* restore db/write.py

* safer check for P2PK secrets for SIG_ALL

* comment cleanup
2024-10-22 14:02:45 +02:00
callebtc
cd39e18916 Bump dependencies to latest versions (#636)
* bump dependencies

* ignore type for validation handler
2024-10-08 09:20:20 +02:00
callebtc
307d2d7a98 Halt melt on exception (#635)
* halt melt for payment status exception

* refactor

* less log

* add fakewallet test throwing exceptions
2024-10-06 17:51:54 +02:00
lollerfirst
c5ccf65e4d Bump SQLAlchemy to 2.0 (#626)
* `SQLALCHEMY_WARN_20=1` fixed all removed warnings.

* fix some mypy errors

* fix fetchone

* make format

* ignore annotations

* let's try like this?

* remove

* make format

* Update pyproject.toml

Co-authored-by: Pavol Rusnak <pavol@rusnak.io>

* extract _mapping in fetchone() and fetchall() + fix poetry lock

* fix

* make format

* fix integer indexing of row fields

* Update cashu/mint/crud.py

---------

Co-authored-by: Pavol Rusnak <pavol@rusnak.io>
Co-authored-by: callebtc <93376500+callebtc@users.noreply.github.com>
2024-10-05 13:32:32 +02:00
callebtc
7fdca3b1a1 Fix: ledger requires deprecated paid flags in db + wallet disable base64 keysets by default (#634)
* wallet: deprecate base64 keysets by default

* readd deprecated paid to melt quotes as well

* readd paid flag removed in #622 before
2024-10-04 15:32:36 +02:00
Pavol Rusnak
75ffa39807 chore: modernize f-string usage (#627) 2024-10-04 14:11:00 +02:00
callebtc
f8f061f810 Fix: Unset pending melt quote by quote id (#629)
* unser melt quote based on quote id

* fixes for lnbits
2024-10-02 15:13:17 +02:00
callebtc
d8d3037cc5 WIP: New melt flow (#622)
* `PaymentResult`

* ledger: rely on PaymentResult instead of paid flag. Double check for payments marked pending.

* `None` is `PENDING`

* make format

* reflected changes API tests where `PaymentStatus` is used + reflected changes in lnbits

* reflect changes in blink backend and tests

* fix lnbits get_payment_status

* remove paid flag

* fix mypy

* remove more paid flags

* fix strike mypy

* green

* shorten all state checks

* fix

* fix some tests

* gimme 

* fix............

* fix lnbits

* fix error

* lightning refactor

* add more regtest tests

* add tests for pending state and failure

* shorten checks

* use match case for startup check - and remember modified checking_id from pay_invoice

* fix strike pending return

* new tests?

* refactor startup routine into get_melt_quote

* test with purge

* refactor blink

* cleanup responses

* blink: return checking_id on failure

* fix lndgrpc try except

* add more testing for melt branches

* speed things up a bit

* remove comments

* remove comments

* block pending melt quotes

* remove comments

---------

Co-authored-by: lollerfirst <lollerfirst@gmail.com>
2024-09-24 14:55:35 +02:00
Pavol Rusnak
25f0763f94 chore: run pyupgrade (#623)
- use `{...}` instead of `set([...])`
- do not use `class Foo(object):`, just use `class Foo:`
- do not specify default flags (`"r"`) for `open()`
2024-09-24 13:53:35 +02:00
Pavol Rusnak
870d75b205 feat: untangle MintMeltMethodSetting into MintMethodSetting and MeltMethodSetting (#617)
+ add description to MintMethodSetting
2024-09-22 15:57:17 +02:00