I tuck mine into my sock, usually the left one...
Snuggish shorts are a big help. When I first started mountain biking, 20 years or so ago, I did a downhill in loose, floppy surfer shorts. At one point the boys swung down just as the bike came up and I nearly paralyzed myself.
I don't like Lycra anymore, but a decent pair of mountain bike shorts ($45 or so on sale at places like REI) provides support without putting you on display. Or you can buy cycling underwear (Andiamo and Pearlizumi are two good brands) to wear under street shorts.
Finally, make sure your saddle is level or nearly so. Occasionally I see people riding with the nose pointing up (?). That will smack your johnson for sure.