Once you get more milage and speedwork in, it does become like an obsession. You want to improve, or at least maintain, that higher level of fitness. That occurs through more riding. So work, weather, and other commitments eat into that time which causes frustration.
It's often a fine line between enjoyment and obsession.