Your biggest problem is right there. A 3 mile ride is very short. You are essentially sprinting every day, which is a lot of short term hard effort and then you stop. Endurance requires more build up with less effort. You wouldn't ride a century the same way you can ride a 3 mile ride. You have to conserve your energy to ensure that you can finish the long ride.

I'd suggest you go for longer rides on the weekend...and even on the weekdays. It sounds like you are taking the most direct route to work and home. Take the short route in and plan a longer route to go home. PM me if you need route suggestions.

Another issue may be clothing. If you are riding in street clothes...jeans, in particular...they can be restricting and energy burning. Lord knows I don't look good in Lycra but I don't wear it for fashion. I wear it because it moves like a second skin and doesn't restrict my legs or make me work harder.

To do a century, by the way, usually takes about 12 weeks of prep time with a progressively longer ride schedule. It's not something that most people decide to do spur of the moment.