Commit Graph

607 Commits

Author SHA1 Message Date
Sergi Delgado Segura
02bc88ed84 Updates Watcher to use db batch updates and avoid multiple decryptions of the same data 2020-01-31 13:00:55 +01:00
Sergi Delgado Segura
a8f2407df9 Updates tests for Responder.check_completed_trackers 2020-01-31 13:00:04 +01:00
Sergi Delgado Segura
d9ce265c00 Updates Responder to minimize data replication operations and properly load data from db
- Uses an instance of the Carrier so it can benefit from issued_receipts and avoid resending multiple copies of the same triggered appointment
- Defines last_known_block to properly load data from db
- Uses task_done from Queue to properly signal task completion when boostraping from db
- Creates a checked_txs dict in get_completed_trackers to avoid querying bitcoind for the same transaction over and over
- Redefines completed_trackers as dict instead of tuple
2020-01-31 12:55:21 +01:00
Sergi Delgado Segura
4848b9a058 Updates cleaner to use db batch updates 2020-01-31 12:53:38 +01:00
Sergi Delgado Segura
5bf8dbd9de Improves load_watcher_appointments and adds methods for batch updating the db
Dumping data into the db became a bottleneck mainly due to not using batch aggregation when dealing with huge amounts of info.
2020-01-31 12:52:12 +01:00
Sergi Delgado Segura
0c4b8eaf4a Adds a dict of sent receipts to act as a cache and avoid sending the same data more than once
The dict is supposed to be reset periodically so it does not grow unbounded
2020-01-31 12:48:11 +01:00
Sergi Delgado Segura
b31e24f655 Adds unit tests for Builder.update_states 2020-01-31 12:46:12 +01:00
Sergi Delgado Segura
caab7a8cdd Adds a method to update the states of the Watcher and Responder if both have missed blocks
The Watcher and Responder were not properly bootstrapped from db data if both were missing blocks to process. Since some appointments may need to pass from the Watcher to the Responder during this process, they need to be brought up to date at the same time, block after block.
2020-01-31 12:46:04 +01:00
Sergi Delgado Segura
084295b889 Adds test for triggering appointment with pissad offline 2020-01-24 14:33:04 +01:00
Sergi Delgado Segura
5d068f5ecc Adds first test of dealing with pisa with a reboot in the middle 2020-01-24 13:40:08 +01:00
Sergi Delgado Segura
d29568e2d7 Merge pull request #87 from sr-gi/fix-build-from-database
Fix build from database
2020-01-24 13:36:13 +01:00
Sergi Delgado Segura
a718a5c6ae Updates Builder unit tests 2020-01-24 13:32:20 +01:00
Sergi Delgado Segura
89181e6a7e Fixes handling data from db
Data obtained from the database was not properly handled in some cases:

- If some appointments are accepted and no block is received and the tower is restarted, the data was not loaded. The tower was checking that the Watcher/Responder had a last known block, which may not have been the case. Now best_tip is set as last_known_block on a fresh bootstrap.
- The Watcher/ Responder were not being awaken when loading data from the database.
- The block queues were not properly populated
2020-01-24 13:31:57 +01:00
Sergi Delgado Segura
7c4d4d0aad Moves awaking / sleeping functionality to their own methods 2020-01-24 13:24:12 +01:00
Sergi Delgado Segura
ba5aa9f651 Updates builder with new minimal in-memory data from #83
- The builder was never modified when the in-memory data was reduced, so it was still trying to build data based on the past approach.
- Renames create_block_queue to poplate_block_queue and repurposes the method

When creating a block queue, a new Queue was created and populated. That was breaking the link between the Watcher/Responder and the ChainMonitor since the Queue is defined beforehand.
2020-01-24 13:21:36 +01:00
Sergi Delgado Segura
26d61ca784 Merge pull request #86 from sr-gi/improve_config
Improves config and init files
2020-01-23 21:18:24 +01:00
Sergi Delgado Segura
deb182edda Updates circle-ci to store keys in data folder 2020-01-23 21:09:39 +01:00
Sergi Delgado Segura
a84d753c4a Updates generate_key to accept output directory. It stores the keys in the current dir by default. 2020-01-23 20:56:42 +01:00
Sergi Delgado Segura
ebea93c103 Updates circle-ci to create cli conf for unit test 2020-01-23 20:16:39 +01:00
Sergi Delgado Segura
3d13cfbe39 Removes pisad from the commands to be run by circle-ci before e2e tests 2020-01-23 20:16:39 +01:00
Sergi Delgado Segura
3fd84a8d7f Modifies e2e tests so pisad can be run and stop from there 2020-01-23 20:16:39 +01:00
Sergi Delgado Segura
14724ceda9 Adds missing common/tools unit tests and removes pisad tests
`test_pisad.py` was only covering config parsing tests, that are now part of `common/tools.py`.
2020-01-23 20:16:32 +01:00
Sergi Delgado Segura
5c75b1f40d Removes Logger dependency from setup_data_folder
Logger was only used to log when a new folder was created, and was making that the setup needed to be done on the main of pisad and cli instead of __init__, which seems a better fit
2020-01-23 20:12:21 +01:00
Sergi Delgado Segura
dce7b4d39e Adds missing tools docs 2020-01-23 19:00:55 +01:00
Sergi Delgado Segura
96ab0039e7 Updates config file params for testing 2020-01-23 18:36:18 +01:00
Sergi Delgado Segura
9be2c2475f Removes logger instance from test_appointment 2020-01-23 18:35:32 +01:00
Sergi Delgado Segura
3185ae124d Fixes paths on cli tests 2020-01-23 18:34:53 +01:00
Sergi Delgado Segura
836048c54d Same as 00a989e1b2 but for the cli 2020-01-23 18:31:47 +01:00
Sergi Delgado Segura
00a989e1b2 Updates pisad and __init__ to put together the log and config fixes
The setup of the logs and the parsing of the config file are closely related. The former need info from the later to be created, and needs to be setup only once
per pisa instance. In the same way, the later need to only be loaded and validated once per pisa intance and contains info to setup the logs.

Intead of setting up the logs in init and loading the config file in pisad, now both are dealt with in __init__
2020-01-23 18:27:04 +01:00
Sergi Delgado Segura
0fe5969346 Merge pull request #85 from sr-gi/improve_logging
Fixes logging issues
2020-01-23 18:23:44 +01:00
Sergi Delgado Segura
6884db9f58 Updates logger codebase-wise to match the Logger updates
Logger instances now specify the logger prefix so the logger can be properly loaded, fixing the issues with the cli file logger
2020-01-23 18:16:25 +01:00
Sergi Delgado Segura
1f46e6eb26 Moves logging setup to common
The setup it's identical for cli and pisad
2020-01-23 18:14:34 +01:00
Sergi Delgado Segura
fddf2e6968 Fixes logger not properly working for cli
The Logger was set to use c_logger and f_logger from pisad, so the cli file logs were never created
2020-01-23 18:13:30 +01:00
Sergi Delgado Segura
a3f2d20499 Moves/adds methods to deal with config to common 2020-01-23 18:05:44 +01:00
Sergi Delgado Segura
c1ad1a4924 Updates cli to run with config file 2020-01-23 16:30:09 +01:00
Sergi Delgado Segura
418b7b49ab Creates main function and updates config to add data folder
pisad can now be run easier from other files (simplifies e2e testing)
2020-01-23 16:27:18 +01:00
Sergi Delgado Segura
245d5b49f3 Updates config files
Removes unused values from pisa config file and adds cli config file
2020-01-23 16:23:07 +01:00
Sergi Delgado Segura
852368a2ad Adds tests for sending two different appointments with the same locator to the tower 2020-01-22 16:25:52 +01:00
Sergi Delgado Segura
03f0a270d8 Modifies when the flag is cleared
Now the only time a triggered flag has to be cleared is when a tracker is removed, otherwise the flag is never created.
2020-01-22 16:24:32 +01:00
Sergi Delgado Segura
272e61922d Move triggered flag clearing to the Watcher when a triggered appointment cannot make it to the mempool
When an appointment was triggered a flag was set in the Watcher, and removed later on in the Responder if the transaction ended up being rejected. That's pretty annoying. Since we have information about whether a transaction has made it to the mempool or not via the Carrier's receipt, this can be all done in the Watcher, which makes more sense and reduces the interaction with the db (1 write if succeeds, 0 otherwise instead of 1 write if succeeds, 2 otherwise).
2020-01-22 16:20:08 +01:00
Sergi Delgado Segura
fd6c85ced2 Moves logs to be consistent with the rest of db logs. Adds missing docs and adds an exception if the db is alredy being used 2020-01-22 15:21:55 +01:00
Sergi Delgado Segura
1027f48611 Splits create_txs in two simpler functions and refactors fixtures so setup_node is not called every test 2020-01-22 14:46:59 +01:00
Sergi Delgado Segura
93cb003360 Setup seed for cli and common unit tests to random values are replicable 2020-01-22 11:12:01 +01:00
Sergi Delgado Segura
14503dcebd Adds e2e tests encrypted blobs with wrong key 2020-01-21 19:12:01 +01:00
Sergi Delgado Segura
0f45e8fe11 Adds methods to get seeded random values and fixes create_txs 2020-01-21 19:11:50 +01:00
Sergi Delgado Segura
7bf65c40f5 Updates unit tests, adds tests to cover new methods and removes unused imports 2020-01-21 19:11:33 +01:00
Sergi Delgado Segura
74a9dad071 Updates Watcher to use new Cleaner functions and reformats filter_valid_breaches to return a list of valid and invalid breaches. 2020-01-21 19:11:15 +01:00
Sergi Delgado Segura
33f2ab35d8 Updates responder to use new Cleaner functions 2020-01-21 19:11:02 +01:00
Sergi Delgado Segura
a95e301713 Renames some DBManager methods and adds new functionality
- Refactors store_update_locator_map name to create_append_locator_map, which is a better fit.
- Adds update_locator_map method to update maps by removing some uuids (store_update_locator_map was not covering this, what was misleading).
- Adds missing docs.
2020-01-21 19:09:39 +01:00
Sergi Delgado Segura
5ead43163c Refactors Cleaner to improve its modularity and fixes some small bugs
- Improves code modularity.
- Separates flag_triggered from delete_appointment, which was pretty misleading (and causing some minor bugs) and make them work with lists intead of single objects.
- Removes unused import
2020-01-21 19:06:51 +01:00