Here's a secret: Studies show that you can maintain your muscles by lifting just once a week for about an hour. Of course, you should push yourself in that hour.
But the point is that it actually takes very little time in the gym to build and maintain muscles.