It gets easier! On family days out my wife and I will split driving and cycling and meet at the swimming pool, boot sale, beach or whatever. (One drives, the other cycles and then maybe swap over for the return). We also have a tandem for fast family rides and the children both ride. Have you read "Long Distance Cycling" by Burke and Pavelka? Very good and an excellent section of fitting in rides with family life.
Good luck and see you in six years! (By then I hope to have done 300 miles in 24hrs on my fixie)