How long have you been riding? Sometimes you just need to take a break and slowing down and/or becoming sluggish is how your body tells you this. 45 miles r/t is pretty long. If you are really pushing yourself in the mornings its only normal for your muscles to feel the resulting pain/fatigue as it recuperates in the afternoon.

Also, take a look at your route. Going to work, my route has about 250 ft of climbing. Going home my route only has 120 ft of climbing. That results in a 2-3 minute faster time and 2 mph faster average speed for my commute home.