So am I the minority wearing pants and long sleeve shirt? Will I get much faster with the usual cycling clothes?
You are in the minority in the US, but millions of Europeans bike to work in dress clothes all the time -- slowly.
Will you get faster with athletic clothes? If your street clothes were binding, then probably you will get somewhat faster, as well as be more comfortable. Otherwise, you'll only get faster if you were already riding so hard that your loose clothing made significant aerodynamic drag.