Also keep things in perspective. Would you feel frustrated if you were out jogging and couldn't keep up with other runners? Face it, people that are less fit, are slower (I include myself here). If you get your muscles and aerobics up to par, fat should in theory make very little difference on the flat, but going uphill you are at a significant disadvantage; to maintain the same speed uphill, you have to work proportionally harder than a lighter person.
On the plus side, if we Clydes have enough balance and confidence to not have to brake, we can beat any skinny guys down any slope!