Last year I bought 3 pair of the cheapest shorts sold by PricePoint, and took the pads out. I was very happy with them. Bike shorts are cut more ergonomically than plain old lycra running shorts or compression shorts - and I never bought into the 'compression' argument. IMHO anything that constricts is bad and may cause hotfoot.
I think having two layers in front is just about a necessity, though; so if you remove pads, definitely wear something underneath! I have coolmax underwear and some lycra ones. Just don't wear cotton for long, hot, sweaty rides!