Good that you're following up with the doctor.
I've had the fuzzy-brained issues before, and ended up in the ER from it because I was too stubborn to listen to my body.
That said and done, taking time off the bike isn't a sin. The bicycle gods won't smite you with brokes spokes and rotted sidewalls for taking a week off from riding. I should know, because I do it a few times a year. I just got back on the bike (commuting) this week after a 1 week hiatus from riding at all, and a second week where all I did was bring my track bike to work and stop by the velodrome for some easy laps after work.
Sometimes, you just need some time off.