skinnyguy, I know people that do it, I don't know how they do it but they do it so you can too if you want. One of the secrets is to get up really early while everyone is still tucked away and get out on the road.
Also if you have ever tried cyclocross and you like it then you may want to consider that as it is very family friendly events that don't take up as much time.