I can't see my hands without contacts or glasses.
I've been swimming for several months now (in a pool) with goggles + contacts and not had any problems.

I'm not sure why this would even be a problem, unless your goggles don't fit correctly. For what it's worth, I'm just using some cheapo Speedo goggles.