feat: implement exponential backoff for paid_invoices_stream (#778)

* feat: implement exponential backoff for paid_invoices_stream across all Lightning backends

- Add exponential backoff retry logic to CLN REST, LND REST, and LND gRPC backends
- Start with 1 second delay, exponentially increase up to 5 minutes maximum
- Reset delay to 1 second on successful reconnection
- Improve error logging to include retry delay information
- Replace fixed delays with adaptive backoff to handle network issues gracefully
- Prevents system overload during persistent connection problems

Resolves issues with rapid reconnection attempts that could overwhelm Lightning nodes during network instability.

* remove unused import

* feat: extend exponential backoff to all remaining backends and invoice listener

- Implement exponential backoff in LNbits paid_invoices_stream for both SSE and WebSocket modes
- Add exponential backoff guidance comments to Blink and Strike backends (not implemented)
- Apply exponential backoff to invoice_listener in tasks.py that calls paid_invoices_stream
- Ensure consistent retry behavior across all Lightning backend integrations
- Improve system resilience during network interruptions and backend failures

All backends and the invoice listener now use the same exponential backoff strategy:
- Start with 1 second delay, exponentially increase up to 5 minutes maximum
- Reset delay to 1 second on successful reconnection
- Enhanced error logging with retry delay information

* blink + strike remove comments

* remove hardcoded values in favor of settings

* immediate first retry
This commit is contained in:
lollerfirst
2025-09-08 16:14:19 +02:00
committed by GitHub
parent ffaac6a5ea
commit ff3fdd5aef
6 changed files with 125 additions and 75 deletions

View File

@@ -4,6 +4,7 @@ from typing import List
from loguru import logger
from ..core.base import MintQuoteState
from ..core.settings import settings
from ..lightning.base import LightningBackend
from .protocols import SupportsBackends, SupportsDb, SupportsEvents
@@ -21,14 +22,22 @@ class LedgerTasks(SupportsDb, SupportsBackends, SupportsEvents):
async def invoice_listener(self, backend: LightningBackend) -> None:
if backend.supports_incoming_payment_stream:
retry_delay = settings.mint_retry_exponential_backoff_base_delay
max_retry_delay = settings.mint_retry_exponential_backoff_max_delay
while True:
try:
# Reset retry delay on successful connection to backend stream
retry_delay = settings.mint_retry_exponential_backoff_base_delay
async for checking_id in backend.paid_invoices_stream():
await self.invoice_callback_dispatcher(checking_id)
except Exception as e:
logger.error(f"Error in invoice listener: {e}")
logger.info("Restarting invoice listener...")
await asyncio.sleep(1)
logger.info(f"Restarting invoice listener in {retry_delay} seconds...")
await asyncio.sleep(retry_delay)
# Exponential backoff
retry_delay = min(retry_delay * 2, max_retry_delay)
async def invoice_callback_dispatcher(self, checking_id: str) -> None:
logger.debug(f"Invoice callback dispatcher: {checking_id}")