That feeling of dizzyness is quite disconcerting. I get it occasionally when I stand up. The world turns grey. I try and get something to hold on to.
You're making the right choice to withdraw. My son is going through that right now because 3 weeks ago he crashed his bike. Other than road rash, he only broke two ribs. He had his whole summer of triathlons all mapped out and now he can't lift his hand over his head. Hasn't even tried swimming.
Let the ear heal and don't add any complications by being too anxious to get back on your bike.