The only problems I've ever had with cold on a bicycle have been on long mountain descents after a long sweaty, rainy, or even snowy climb. That's when I'm sure to have an extra dry layer. Otherwise it's fairly easy to stay warm on a bike with minimal packed clothing. In summer, that's a mid-weight long-sleeved jersey (8-10 oz) and a wind jacket (4 oz). No need for a fleece, which, as you noted, can weigh up to two pounds and is just not worth its weight and packed bulk in my experience. Shorts are always fine in summer for me. Rain protection can be minimal as well, but that depends if you're afraid of getting wet or not. I go with the "wet and warm is okay" school. I steam under my 4 oz. non-breathable jacket and keep my warm jersey dry for camping. My total extra clothing for short summer trips, including rain gear, can come in as low as one pound.