I agree with antokelly on this -- the descents will be the toughest part. For the climbs you can always get off and walk. I suppose you could for steep descents as well, but, ugh!

Do you have front and rear brakes on your fixie? If so, perhaps you could at least put on a freewheel and do the tour single speed rather than fixed.

That said, I'm sure you can do it either way -- just listen to your body and rest / set your pace as needed. As many have pointed out in previous threads, you don't need to follow conventional wisdom: people have toured successfully on just about everything.