I dont have any problem with bike shorts and jumper. I grew up with wool shorts and jersey, wool tights in the cool months, so no self consciousness there. Although I would get a lot more inquisitive looks back in the '70's.

That being said I ride quite often in cargo shorts over bike shorts, and a tee shirt. Oh, and get this - no helmet! I mean road cycling can be just a recreational ride, you don't have to be blasting down the road all the time.