I was in the Marines for 7 years, and I have no issue with people wearing the EGA out of respect.
If you were to wear a uniform, it is a potential issue, but that is not the case here.
You will probably be asked where you served etc, but if you do, just be honest that you wear it out of respect for those who it represents, then all should be well.
You might run into a jerk or two that won't like it... but you'll get that no matter what you wear.
