Hi JonnySays420-
I think you think you're more comfortable in your LowBudgetOutfit for riding without having tried the alternative. Obtain a UnderArmour winter compression shirt, full-length BlackBottoms bib tights, an Assos jacket, Salsa lobsterclaw gloves, and neoprene stretch booties over clipless riding shoes and tell me who is more comfortable on a bicycle. The sole exception to the booties would be if one is riding with full-size boots appropriate for Alaska-type conditions coupled with platform pedals.
I can guarantee you it isn't the guy with "loose-fitting denim jeans with cotton sweatpants" underneath. A popular phrase among hardcore outdoor enthusiasts is "cotton kills" when it comes to garments for inclement weather.
~ Blue Jays ~