Haha, great answers everyone. I happen to have a really light build, and it isn't a problem for plainclothes, but I can't seem to find an inexpensive extra small jersey. I've noticed that large and extra large sizes for clothing tend to be much more common. But sewing it might help.

By the way, would something like an Underarmour compression shirt that doesn't flap around be fine to wear?