It sounds like you have what is called late response vs. early response exercise induced asthma (EIA). (Specifically a subcatagory called refractory period EIA.) It is significant since over the counter bronchodilators when used before excercise only helps the early response EIA.

Early response EIA occurs within the first 6 to 8 minutes of exercise where you have reached at least 80% of your max heart rate. (therefore low intensity walking will not trigger it) The theory behind early response EIA is not inflammatory related and is due to changes in humidity, osmolarity gradient etc. in the bronchial tubes as a person goes from nose breathing to mouth breathing, thus bypassing the criticlal airway passages that humidify and moisten the air before it reaches the bronchiol tubes where the spasms occur. The late response EIA seems to be inflammatory mediated in much the same way people with regular asthma are affected. Therefore, things like sensitivity to the pollutants, pollen, dust etc. may be playing a role since after 2-3 hours of riding, you have exposed your lungs to a greater "allergen load" and thus triggered the late response EIA. Only prescription drugs with a long half life such as Singulair(montelukast) or a long acting beta agonist will help. The short acting ones such as some forms of albuterol will not be kicking in at the time you specified that you are having trouble.

See a doc, preferabbly one that is active enough to not pose the question "So what are you worried about if you can exercise at least 3 to four hours?" Leave if that is the response you get and seek out one who understands your plight.