it may cause problems with the boys. don't just brush it off after a ride and think it'll go away, or reason that the increased cardio will even out any damage you do.
i'm not saying you need a brooks, or a funky saddle, but you do need to look out for your taint. it's sensitive, and just because the pain goes away, doesn't mean the problem is gone. you could be squishing your urethera, your nerves, your blood vessels, or your prostate — all of which are necessary for a healthy sex life. if you put too much pressure on these, strictures could result, and that means problems with the flow fluids. check with your urologist (if you've got one), do some reading, and above all, listen to what your body is telling you.