Cycling isn't the most important thing in the world. You just need to be sure that the things you are doing instead of cycling are truly more important (i.e. spending time with kids, yes. Laying on couch watching TV, no)
Cycling takes time which is the most valuable commodity we have. We all have 24 hours in a day, so whatever you are doing instead of riding is getting a higher priority. That may be OK, or it may not be depending on what those things are.