Don't worry.
Be Happy!
Seriously, the matrix expands and contracts ALOT when:
-heated/cooled
-moisture absoprtion/desorption
while the fibers don't (as much)
So you can image there will be some stresses building up at the fiber/matrix interface evrytime its therm/hydro cycled. Overtime interface debonding may occur...its true....BUT (having manually placed CF samples into moisture chambers and ovens for accelerated testing) its a SMALL effect and it takes A LONG time.
It's not a new problem so matrix materials have been designed to be more and more resistant to this effect.
As for UV, yup it will degrade the CF, but it will degrade the Matrix much more. But the amount of damage vs the amount of exposure time is ridiculously in your favor (unless you park you bike in direct sunlight for years at a time)....
Practicaly speaking:
Don't worry.
Be Happy!
If your CF component begins to suffer from UV/moisture/thermal degredation. You should be seeing signs of this in the form of discoloration and warpage.
Just don't race or fly a CF car/bike/plane thats warped up like a potato chip and has matrix cracks and chips all over the place :-)