AF,
No I would not feel discouraged at all. In the summer when I MTB I can be seen in plain shorts, and hiking boots. In winter it is sweats and some type of wind breaker. I do have some polypropylene under garments but it has not gotten that cold here except once. I mainly had this stuff when I lived in Va. I would not even be concerned about getting taken on a hill or anywhere else. From what you described you are doing just fine. Keep it up.