Everything you said, plus street baggies can wrinkle between you and the saddle and cause sores on a long ride. Really, though, I think a huge portion of it is style. I've been riding for nearly 40 years, and in just the last five or six, the sport's been taken over by people who place a lot of importance on dressing up like cartoon superheroes before they go out and circle the park. A friend of mine, actually a pretty good rider, won't go half a mile to the grocery store without putting on his lycra AND filling his water bottles.
Personally, I wear mountain bike shorts for nearly everything, including all-day rides. They work fine for me, and I can go into 7-Eleven or Starbuck's without looking like a dork. I usually wear a T-shirt instead of a jersey, too, because, come on, about 10 bucks vs. $75 for something that's no more comfortable? I have half a dozen real jerseys, including two expensive wool ones, but they live in the drawer.