I'm too scared to do those rides when I think I'm not fit enough.

Way, way back, before I did my first ever track race, I spent a whole track season hiding in the bushes at the velodrome with a stopwatch, then spent the following off-season making sure I could get close to the required times. I was too scared to go down there and be embarrassed