I felt the same about wearing lycra shorts.. Once you realize how much more comfortable it is you wont use anything else.. Someone mentioned about taking extra shorts with you if you dont want to wear them around, thats something to consider..

Don't worry about what other people think.. Do whats best for you.. (sounds selfish but I think you get the idea)