I ride in normal clothes most times when I take my 14-mo daughter on a spin around the neighborhood.
On my beater mountain bike with a baby seat, riser bars, and platform pedals. Sometimes with a dog on a purple leash tied to the head tube.

The jersey & shorts just aren't necessary. It probably looks ridiculous enough in a t-shirt and cargo shorts.