I also commute on my road bike in everyday clothes, no problems whatsoever. My clothes are pretty loose and unrestrictive, and I've never relied on chamois padding to keep me comfortable, so no problems there
Cycling specific clothes make a difference for me once the ride becomes long enough and hard enough for sweat to become a problem.