I get cranky when i am not able to meet my workout targets as well, sometimes. If I am carrying over fatigue from before, that's one thing and i dont mind. But if it is something else which ruins my workout, then for sure I get a bit annoyed/irritated.
OP - just ride more. Also, if you can, ride with others. It's a great way to push yourself more. Lastly, depending on what the limitation is, consider doing weights. I am 47, and i find my overall health and fitness to be much, much better now that i am back to doing weights, than it was when i cycled only. At our age, we are starting to lose both muscle and bone density, and weight training helps there. Doing more squats and such has also helped my time to exhaustion (ie, my ability to hold wattage for a longer period of time without fatiguing).