If you feel better on the flats, ride with the flats. If there were really disadvantages to using them, would you have to ask the question? No one should tell you how to ride.
I've ridden 225 miles this weekend on Raceface Chesters, I wasn't racing, but I felt "serious" enough .