I ride mostly on paved rural roads and bike paths with crushed limestone or hard-packed dirt, and I have 28mm tires. The only time I've had a real problem was a few days ago when I tried going up a hill with rough gravel. I ended up getting off and walking the bike up, mainly out of concern for my tires. But that is by far the exception, and really it's avoidable -- I could've taken a different path. I'll probably eventually try a set of 32mm tires just to see the difference they make on semi-rough paths, but honestly the 28s are just fine. Like you, I was worried about that before I bought the bike, but it's turned out to be a non-issue for me.

As for your concern about costing you speed, the guy I ride with rides a cyclocross with 35mm bumpy tires, and he can run circles around me. That's not to say wider tires are faster -- just that when it comes to speed, strength and conditioning are much bigger factors than tires. So if curbs and potholes are a regular issue where you ride and you need the 35s for comfort, I don't think you'll have regrets simply due to speed.