I've experienced the same thing you describe and so have others I know including a Master's racer friend. It's more pronounced if I haven't been getting enough sleep.
If I fight the urge to nap I am fine and can stay awake the rest of the day but many times I just take a nap of 30-40 minutes. If I do a hard ride with lots of climbing I am more inclined to want that nap. I'll probably have one later today after my ride.