I had fears of losing the saddle time when my son was born.
He is now at every race with me - wife in tow as well. They are loving the team/family atmosphere and when I get looks about doing a ride I remind them of how they felt having to tell everyone why daddy started to fall back during the race.
Works pretty well.
Remember though...at some point I am sure you started riding because it was fun. Do what you need to do but rememebr that cycling can still be fun again...even at only a few hours a week.