but then how will you keep track of which parameters worked well? I've been essentially storing my kwargs in json and not felt a need to control anything directly from the CLI.
Good question, for me I programatically generate a folder with the hyperparameter key - values in the name and store the checkpoints under it. As you can imagine, it can get out of control quickly if not managed well, but it works for 1-3 person projects. For anything more large scale or organized, I would recommend looking into Comet ML which lets you query and filter your experiments by hyperparameter ranges instead of manually looking at folder names.
I did that until I hit the Linux directory name length limit, lol. Now I hash the hyperparameters dict to get the directory name, and store a json file within. Totally ad hoc and I'm sure a better solution exists.
