everyone is different, but my legs would get cold in the set-up you describe.

for about 30F-20F I have some goretex snow pants I wear over my boxers.

below 20F I wear the snow pants over my jeans I wear at work all day and just jeans for temps over 30.

The snow pants break the wind, so any kind of wind blocker for your legs you can get would work with the layers you already have. It is amazing the kind of stuff you can get at a thrift store if you are not going for style points.

under armor pants I am unfamiliar with, maybe they are woven tight enough to block wind?