Get back on that baby! After a crash, that caused me to land facedown, stop breathing for about 80 seconds until my friend rolled me over and unconscious for 3.5 minutes (documented by video), I got back on my bike as soon as a neurologist said I was OK. That was about 10 days later. I would have gotten back sooner, but my wife was terrified I wasn't well enough to start again. Now that I look back, it took me about a week to recover, so I'm glad she held me back. But get back with calm rides and work your way back up to where you were.
Good luck!