Every time I see a roadie with a Camelback on my first thought is "Cool, that person must be going on a VERY LONG ride!". Point being, people do it, if it works for you, go for it! I have gone on 'fast' rides with panniers mounted on my road bike and they did not detract from my enjoyment of the ride.
If you're trying to squeeze every bit of speed and power, then a camelback/panniers/etc. may not be optimal.