If you don't have to push hard, anything under an hour or so is definitely doable with street clothes. I ride 3.5 miles to work every day in my office clothes. I also keep an extra clean shirt at work in the summer, and an extra pair of socks and pants as well in the winter. Just in case I get splashed/rained on while riding, or in the event that I get too sweaty and smelly on the way to work (traffic detours in hot weather, whatever).
The only time that I wear cycling shorts and a jersey, is when I'm doing a 35+ mile training ride added onto my commute (work on college campus, with showers available just a couple of blocks away from my office). Bring and/or keep some backup clothing in your office, and you should be fine.