i rode 60miles fixed the other day and was surprised that i wasn't sore at the end of the day or the next, i ran a brake so obviously that helped, especially on the downhills, there was someone who commented on a hill that he didn't envy us(there were 4 of us riding fixed, but when we ran into him at a coffee shop later he basically said "it got you here as well as my bike got me, if i were 20years younger..."

just make sure your bike is the right fit, find a gearing that works and make sure to balance your load(i had a 15# mess bag with water and tools and tubes on my back) and you'll be fine, i should have put panniers on my bike rather than the bag but otherwise no regrets