by DevAbdul on 2/5/24, 3:05 PM with 1 comments
Share your experiences and insights on what functionalities or features you feel are missing in your current workflow.
by PaulHoule on 2/5/24, 4:01 PM
I just did a maven build a moment ago that added up to 13 seconds but I have at times had build processes that took upwards of 40 minutes. Sometimes for bad reasons, but sometimes this is intrinsic to the kind of work I am doing.
For instance I have a machine learning model that I train automatically every day, it takes 3 minutes and in that 3 minutes it is training multiple models, testing them, choosing one, and calibrating the model. There are four steps of retrieving items from an RSS feed and incorporating them in a database, training a classification model, running another classification, and gathering more data from the Hacker news API. I could probably squash them down a single Python script but running them all every day doesn't feel like a burden.
I have another model that takes around 45 minutes to train a single model. I might have to train 10 of those to get one which is production quality. Possibly I could fully automate the process and parallelize it, just kick it off and have a model in 2.5 hours. I am not there yet, but trying to get there I will have to start the trainer, go do something else that doesn't interfere with the trainer, get a notification that the trainer completed and it needs my attention, etc.
That is, I'd like tools to help manage long-running processes like that.