I got a concussion after a race crash. Mine was probably more mild as my memory was fully functional (apart from the time between the crash and the ambulance ride). The guy manning the corner where I crashed said I got up and was worrying about the bike. I paniced when the lovely paramedic said I had repeated something for the 4th or 5th time...

Once the swelling goes down your memory should get better. There are some things you wouldn't remember even if you didn't have the concussion. The reason they tell you to lay off the bike is not that you can't operate a bike (I was able to the next day) but that if you crash again while your brain is still bruised you can damage it permanently. If the ER doc cleared you (and you must have had a MRI - I got one), then you should be fine after plenty of rest and proper nutrition.

edit* Of course if its been six days and you're still having memory problems, go see someone.