... I was a touring cyclist as well. I was wearing a pair of runners, street shorts, a running t-shirt and ball cap.
I really enjoy riding my bike covert style without a cycling uniform and haven't seen any reason to go back for most of my riding.
My case is exactly the same, with only one difference: I always have an helmet instead of baseball cap
I suggest any reader experiment some days forgetting cycling wear, cycling shoes and straps/hooks in pedals too. Enjoy freedom!!