Except for the uncontested fact that bike-specific clothing is more comfortable and functional than non-specific bike clothing, what you wear doesn't make a difference if you're riding alone.

Initially, when people getting into cycling, they don't want to dress in the "uniform" because they think they'll look funny (and they will, to non-cyclists).

If you think you want to ride with other cyclists who, like you, OP, have nice bikes, then you're really going to feel out of place if you wear running shorts and a t-shirt. Part of the reason for wearing bike shorts and a jersey when others do is tied up with tradition, and part is the respect you give to your fellow cyclists and for the sport of cycling itself. It's the same if your interested about any other sport, whether you enter a marathon, play golf or tennis, beach volleyball, etc.

You want to be an individual, but you also want to connect with the group, to be part of something bigger than yourself. So if it's just you riding by yourself, wear what you like. Ride with others? Tradition!