I think you'll be fine- especially as you're
travelling light! Aluminium is extremely strong, and frame design takes into
consideration the inherent qualities of this material (including the frailties/weaknesses).
I think you just have to be sensible with the kind of riding/purpose you're doing and be aware of any changes
in responsiveness in the frame if you're really concerned etc.

Aluminium frames are very well constructed these days in 6061
and 7005 Alu. I have never encountered aluminium failure myself.

What bike are you using?