I started off commuting to school with a hybrid and I couldn't help but move onto road bikes for the positioning. I have a few harsh areas with gravel and potholes but i normally just stand up and tough em out. If you manage to sit on the saddle the whole time, good luck. Although my commute speed hasn't gotten all that faster, about 20 minutes for 5 miles (with stop lights), I just feel better riding a bike that is lighter and more aerodynamic.

So in short, speed might minimally increase, but your enjoyment might skyrocket. Get a "low end" roadie to start out with that can take some punishment.