Of course your suspicion would be correct. And it would be totally natural and normal for the men to find the attractiveness of the woman in question to be very important information to enquire about and consider when pondering the scenario. It's just the way we are. The women may be more interested in other things. That's totally natural too. Neither are situations to get upset about.