Overhauled the tracker system (#172)

* Overhauled the tracker system
Separated the logging and saving capabilities
Changed creation to be consistent and initializing behavior to be defined by a class initializer instead of in the training script
Added class separation between different types of loaders and savers to make the system more verbose

* Changed the saver system to only save the checkpoint once

* Added better error handling for saving checkpoints

* Fixed an error where wandb would error when passed arbitrary kwargs

* Fixed variable naming issues for improved saver
Added more logging during long pauses

* Fixed which methods need to be dummy to immediatly return
Added the ability to set whether you find unused parameters

* Added more logging for when a wandb loader fails
This commit is contained in:
Aidan Dempster
2022-07-01 12:39:40 -04:00
committed by GitHub
parent 7b0edf9e42
commit 27b0f7ca0d
7 changed files with 662 additions and 212 deletions

View File

@@ -80,20 +80,32 @@
}
},
"tracker": {
"tracker_type": "console",
"data_path": "./models",
"overwrite_data_path": true,
"wandb_entity": "",
"wandb_project": "",
"log": {
"log_type": "wandb",
"verbose": false
},
"load": {
"source": null,
"wandb_entity": "your_wandb",
"wandb_project": "your_project",
"run_path": "",
"file_path": "",
"verbose": true
},
"resume": false
"load": {
"load_from": null
},
"save": [{
"save_to": "wandb"
}, {
"save_to": "huggingface",
"huggingface_repo": "Veldrovive/test_model",
"save_all": true,
"save_latest": true,
"save_best": true,
"save_type": "model"
}]
}
}