Tell that to the ss and fg riders who've done 1200k brevets. I know people who have ridden PBP, GRR, Cascade, and raced The Cannonball! and the S2S on fixed gear bikes. Heck, Deanna Adams finished the Great Divide Race on a fixed CX bike this year and Kent Peterson is racing it on a fixed 29er in 2010.
I put in about 25% of my yearly mileage on my ss/fg rig in the not-flat-as-a-pancake Cascade foothills. The key is to get your gearing comfortable, and realize that you're never going to max out your speed on the downhills, and yes, you may struggle on some of the longer, steeper climbs. I use a 44/18 combo (65.4gi) which at a comfortable 90rpm cadence puts me at 17.5mph.

That all said and done, I can't imagine spending 30 minute cleaning the drivetrain on any of my bikes. Even my brevet bike (2x9) is a 10 minute process: Pull the chain and repeat rinse it with mineral spirits until it rinses clean. Spritz the cassette with mineral spirits and scrub with a parts brush or floss it with the collar from an old t-shirt. Same for the derailleurs. Wipe the rings with an old t-shirt and some mineral spirits. Slap the chain back on and give it a drop of Finish Line per link.