Commit Graph

248 Commits

Author SHA1 Message Date
Sergi Delgado Segura
07aa9a73ee Adds debug logging of all request to track possible bugs (stored in file only) 2020-02-17 12:21:53 +01:00
Sergi Delgado Segura
5b32befade Removes logging txids list from every block
The list of transactions can be queried afterwards and it is currently flooding the log files. I may consider putting it back once we have a log parser, but it seems pretty unnecessary.
2020-02-17 10:41:02 +01:00
Sergi Delgado Segura
fc24a4e8c9 Fixes start_time range log 2020-02-16 17:42:10 +01:00
Sergi Delgado Segura
4406beb5cd Limits the encrypted_blob size to 2Kib for the alpha 2020-02-16 17:41:02 +01:00
Sergi Delgado Segura
6e737b735c Logs real IP if API is behind a reverse proxy 2020-02-15 19:24:20 +01:00
Sergi Delgado Segura
ec8418294b Changes to_self_delay restrictions to fit the nLockTime field 2020-02-14 14:39:51 +01:00
Sergi Delgado Segura
99a9b2f33e Improve return messages from 83df910d5954bf1943e504b84e3aa5e0eb8b21d7 2020-02-14 14:39:50 +01:00
Sergi Delgado Segura
4c9ddd0827 Sets limits for star/end time and to_self_delay for the alpha 2020-02-14 14:39:43 +01:00
Sergi Delgado Segura
3ba5db533d Fixes json parsing of None when no json was postesd to the API
h/t @yahgwai for finding this.
2020-02-13 18:42:00 +01:00
Sergi Delgado Segura
3b5ff2e92c Removes get_block_count endpoint
get_block_count was only for dummy testing and it's not used anymore
2020-02-13 16:59:06 +01:00
Sergi Delgado Segura
7e29824452 Improves API log and adds logs for get_appointment 2020-02-13 16:51:44 +01:00
Sergi Delgado Segura
02e159faac Moves EncryptedBlob to common 2020-02-13 16:32:51 +01:00
Sergi Delgado Segura
d447debe54 Temporal fix for Cryptographer logs. Close #91 2020-02-11 20:22:51 +01:00
Sergi Delgado Segura
22ded55990 Renames pisa_cli to wt_cli 2020-02-11 18:37:12 +01:00
Sergi Delgado Segura
ab21cbfc8f Moves load_key_file to Cryptographer and updates pisad to use it 2020-02-10 16:31:48 +01:00
Sergi Delgado Segura
a4f7548804 Removes sleep flag and reorders code for redability 2020-02-10 16:21:05 +01:00
Sergi Delgado Segura
aa12fa2cf8 Removes sleep flag from Watcher and Responder
The sleep flag was used to avoid doing useless work when no data was hold by the tower. However, from the implementation of the data persistence on, the Watcher and Responder should at least keep track of the last known block. The current apporach was making this harder.
2020-02-10 16:19:22 +01:00
Sergi Delgado Segura
4f000298fa Removes zmq_subscriber
The ZMQSubscriber was merged into the ChainMonitor but never deleted
2020-01-31 13:11:59 +01:00
Sergi Delgado Segura
32ff13a495 Fixes loading data from disk when both Watcher and Responder need to be brough up to date
The previous approach was not correct, since both actors need to be brought up to date at the same time.
2020-01-31 13:02:32 +01:00
Sergi Delgado Segura
02bc88ed84 Updates Watcher to use db batch updates and avoid multiple decryptions of the same data 2020-01-31 13:00:55 +01:00
Sergi Delgado Segura
d9ce265c00 Updates Responder to minimize data replication operations and properly load data from db
- Uses an instance of the Carrier so it can benefit from issued_receipts and avoid resending multiple copies of the same triggered appointment
- Defines last_known_block to properly load data from db
- Uses task_done from Queue to properly signal task completion when boostraping from db
- Creates a checked_txs dict in get_completed_trackers to avoid querying bitcoind for the same transaction over and over
- Redefines completed_trackers as dict instead of tuple
2020-01-31 12:55:21 +01:00
Sergi Delgado Segura
4848b9a058 Updates cleaner to use db batch updates 2020-01-31 12:53:38 +01:00
Sergi Delgado Segura
5bf8dbd9de Improves load_watcher_appointments and adds methods for batch updating the db
Dumping data into the db became a bottleneck mainly due to not using batch aggregation when dealing with huge amounts of info.
2020-01-31 12:52:12 +01:00
Sergi Delgado Segura
0c4b8eaf4a Adds a dict of sent receipts to act as a cache and avoid sending the same data more than once
The dict is supposed to be reset periodically so it does not grow unbounded
2020-01-31 12:48:11 +01:00
Sergi Delgado Segura
caab7a8cdd Adds a method to update the states of the Watcher and Responder if both have missed blocks
The Watcher and Responder were not properly bootstrapped from db data if both were missing blocks to process. Since some appointments may need to pass from the Watcher to the Responder during this process, they need to be brought up to date at the same time, block after block.
2020-01-31 12:46:04 +01:00
Sergi Delgado Segura
89181e6a7e Fixes handling data from db
Data obtained from the database was not properly handled in some cases:

- If some appointments are accepted and no block is received and the tower is restarted, the data was not loaded. The tower was checking that the Watcher/Responder had a last known block, which may not have been the case. Now best_tip is set as last_known_block on a fresh bootstrap.
- The Watcher/ Responder were not being awaken when loading data from the database.
- The block queues were not properly populated
2020-01-24 13:31:57 +01:00
Sergi Delgado Segura
7c4d4d0aad Moves awaking / sleeping functionality to their own methods 2020-01-24 13:24:12 +01:00
Sergi Delgado Segura
ba5aa9f651 Updates builder with new minimal in-memory data from #83
- The builder was never modified when the in-memory data was reduced, so it was still trying to build data based on the past approach.
- Renames create_block_queue to poplate_block_queue and repurposes the method

When creating a block queue, a new Queue was created and populated. That was breaking the link between the Watcher/Responder and the ChainMonitor since the Queue is defined beforehand.
2020-01-24 13:21:36 +01:00
Sergi Delgado Segura
5c75b1f40d Removes Logger dependency from setup_data_folder
Logger was only used to log when a new folder was created, and was making that the setup needed to be done on the main of pisad and cli instead of __init__, which seems a better fit
2020-01-23 20:12:21 +01:00
Sergi Delgado Segura
00a989e1b2 Updates pisad and __init__ to put together the log and config fixes
The setup of the logs and the parsing of the config file are closely related. The former need info from the later to be created, and needs to be setup only once
per pisa instance. In the same way, the later need to only be loaded and validated once per pisa intance and contains info to setup the logs.

Intead of setting up the logs in init and loading the config file in pisad, now both are dealt with in __init__
2020-01-23 18:27:04 +01:00
Sergi Delgado Segura
6884db9f58 Updates logger codebase-wise to match the Logger updates
Logger instances now specify the logger prefix so the logger can be properly loaded, fixing the issues with the cli file logger
2020-01-23 18:16:25 +01:00
Sergi Delgado Segura
418b7b49ab Creates main function and updates config to add data folder
pisad can now be run easier from other files (simplifies e2e testing)
2020-01-23 16:27:18 +01:00
Sergi Delgado Segura
245d5b49f3 Updates config files
Removes unused values from pisa config file and adds cli config file
2020-01-23 16:23:07 +01:00
Sergi Delgado Segura
03f0a270d8 Modifies when the flag is cleared
Now the only time a triggered flag has to be cleared is when a tracker is removed, otherwise the flag is never created.
2020-01-22 16:24:32 +01:00
Sergi Delgado Segura
272e61922d Move triggered flag clearing to the Watcher when a triggered appointment cannot make it to the mempool
When an appointment was triggered a flag was set in the Watcher, and removed later on in the Responder if the transaction ended up being rejected. That's pretty annoying. Since we have information about whether a transaction has made it to the mempool or not via the Carrier's receipt, this can be all done in the Watcher, which makes more sense and reduces the interaction with the db (1 write if succeeds, 0 otherwise instead of 1 write if succeeds, 2 otherwise).
2020-01-22 16:20:08 +01:00
Sergi Delgado Segura
fd6c85ced2 Moves logs to be consistent with the rest of db logs. Adds missing docs and adds an exception if the db is alredy being used 2020-01-22 15:21:55 +01:00
Sergi Delgado Segura
74a9dad071 Updates Watcher to use new Cleaner functions and reformats filter_valid_breaches to return a list of valid and invalid breaches. 2020-01-21 19:11:15 +01:00
Sergi Delgado Segura
33f2ab35d8 Updates responder to use new Cleaner functions 2020-01-21 19:11:02 +01:00
Sergi Delgado Segura
a95e301713 Renames some DBManager methods and adds new functionality
- Refactors store_update_locator_map name to create_append_locator_map, which is a better fit.
- Adds update_locator_map method to update maps by removing some uuids (store_update_locator_map was not covering this, what was misleading).
- Adds missing docs.
2020-01-21 19:09:39 +01:00
Sergi Delgado Segura
5ead43163c Refactors Cleaner to improve its modularity and fixes some small bugs
- Improves code modularity.
- Separates flag_triggered from delete_appointment, which was pretty misleading (and causing some minor bugs) and make them work with lists intead of single objects.
- Removes unused import
2020-01-21 19:06:51 +01:00
Sergi Delgado Segura
0f887feb4d Deletes appointment from Watcher's db if a tracker cannot be added
There was a bug in the Responder where, if a tracker could not be added (e.g. malformed tx) the data was dropped but never removed from the Watcher's db. This is due to flagging appointments as triggered instead of deleting them from the Watcher's db straight away (in order to deal with reorgs in the future).

If that approach is not followed, and appointments are removed from the Watcher's db once they are passed to the Responder, this should be removed since it would be redundant.
2020-01-17 17:47:45 +01:00
Sergi Delgado Segura
dad70eb780 Moves update_delete_locator_map functionality to its own method. 2020-01-17 17:47:05 +01:00
Sergi Delgado Segura
07c9b7d19e Updates error message for RPC_VERIFY_REJECTED rpc error. 2020-01-17 17:46:14 +01:00
Sergi Delgado Segura
b32a867283 Moves db_manager instantiation before checking the connection with bitcoin
As it was until now, if the connection with bitcoind failed, the  command would have also failed, since the db_manager was not yet defined.
2020-01-17 15:32:30 +01:00
Sergi Delgado Segura
90e1245a84 Fixes some bugs based on E2E testing 2020-01-16 17:08:44 +01:00
Sergi Delgado Segura
3dc1fbcb86 Specifies minimum version for bitcoind_mock 2020-01-15 16:34:17 +01:00
Sergi Delgado Segura
0ddace6256 Updates requirements and adds dev requirements 2020-01-15 15:48:30 +01:00
Sergi Delgado Segura
a49f86dd88 Merge branch 'master' into 64-data-to-disk 2020-01-15 12:44:29 +01:00
Sergi Delgado Segura
42a957f555 Merge branch 'master' into chainmonitor 2020-01-15 11:38:32 +01:00
Sergi Delgado Segura
6bf2cb5379 Merge branch 'master' into isolate-config 2020-01-15 10:15:48 +01:00