Here is a simple rule of thumb: If you weight more than 180 pounds, you should not wear spandex...ever. Obviously, this number is lower if you are female.
I wear compression shorts under normal shorts when I ride. It's the only way my boys are comfortable and I don't chaffe. If my ride is short, then I don't worry about it.
I'm assuming the OP is referring to the full body spandex outfits. I agree with the person who said only racers should be wearing this costume.