They are expensive, but rapha touring shorts fit the bill for me:

http://www.competitivecyclist.com/ro...paign=Datafeed

I'm pretty thin, 5' 11", 140lbs, and they fit me great. They don't flap around and they are not baggy at all. I wear lycra shorts under them.