The biggest downside is you have to secure those things when you park the bike. I've had skewers stolen, which is really a drag (especially since I rode off without noticing and crashed shortly thereafter when I tried to jump a curb).

I just bought a set of skewers and seatpost clamp that fasten with an allen bolt. I figure if I'm carrying a pump, tube and tire irons I can carry an allen wrench, and if I'm not it doesn't matter if I can get the wheel off without tools.

Some riders adjust their seat height frequently, but I don't.