I ride around 10 miles each way here in Denver in my work clothes... mostly jeans or khaki pants and decent shirt... Nice thing about riding a bike is that if you start to sweat you can slow down and take it easier.

Your biggest enemy will likely be goathead thorns... I'd recommend kevlar belted tires and thorn resistant or slime tubes... With only a three mile commute it won't be the end of the world if you get a flat, but you may want a spare tube, pump, and patch kit anyway.