from Hacker News

Auto-unloading models using __init_subclass__ (Python)

by matroid on 10/3/23, 6:47 PM with 1 comments

• by matroid on 10/3/23, 6:47 PM

  I wanted functionality where GPU VRAM isn't constantly hogged while I'm serving a PyTorch model, so that I could simultaneously train stuff.

  I wanted a solution which was agnostic to the type of the model, with respect to loading and inferring.

  So I made this AutoUnloadModel class that unloads the model if it hasn't been used for some period. I used __init_subclass__ to ensure that all the details regarding timers, locks etc are hidden from the subclass.

  I found __init_subclass__ very cool for this job, which is the reason I'm sharing this. Thanks!