i'm still not at this point with cycling but i can say from experience with other types of training that if you are feeling drained you DEFINITELY need at least a full day off, two if you can do it

i know it's hard because you feel like you are sacrificing something but to be honest with some rest you will be better than ever when you get back, rest is a necessary part of training for any sport