My situation is much the same.
Work starts at 07:00 for me and ends 16:00
add hour for commute (each way) and the day is
pretty much shot. The traffic around my area isn't
really conducive to riding seriously after work
(its ok for noodling around but not serious road miles).
04:00 rides do not thrill me so, at that hour
(0darkhundred) I'm on the rollers for an hour.
Weekends for me are better, both the wife
and I try to ride, or I go out alone.
Things are different during late spring to early fall
I can get a more "serious" ride in after work.
due to longer days.

Marty