Because I'm prone to riding stupid long mileage on the weekends, my view on things might be a little different than others, but here's my take on cycling shorts.

I also ride Brooks saddles on all my bikes. I can do a short ride (up to 40 miles) in street clothes with little to no discomfort. As you stated, the Brooks allows you to put your weight where it belongs; on the sit bones.

The discomfort creeps in when the temperature starts to rise, and I sweat. A lot. Sweat leads to damp clothing, damp clothing leads to chafing, which is painful and problematic.
If you look at old school bike shorts with a real leather chamois, there wasn't any padding in there. It was a semi form-cut slab of hide from Rupicapra rupicapra, which absorbed sweat and wicked it away from the skin.
Moving forward, technology advanced and people didn't want a leather chamois. They're a pain to wash and maintain; so the chamois was replaced with padding and technical fabrics. Wicking cloth with antimicrobial fibers now perform the same task as a chamois slathered with some conditioning creme containing witch hazel and menthol.
Levels of padding and associated comfort are totally personalized, much like chosing a saddle. And to make things even tougher, what's comfortable for one distance may not be comfortable for longer or shorter distances, or on a different bike, or a different saddle. For example: I used to wear Performance Century Gel bibs. They were comfortable until I started riding more than 30 or 40 miles, at which point they were bulky and non-breathable. I switched to Performance Ultras, which are a much higher compression fabric and thinner padding; very breathable and fantastic up to 200k. However, the padding doesn't evaporate sweat fast enough so it stays damp and ruins the viscosity of most chamois cremes. Bummer. I tried PI Ultrasensors, which have a thicker padding but claims of higher breathability and evaporation rates. They aren't kidding, and the Ultrasensors are great... until you wash them 5 or 6 times and the padding cover fabric takes on a microtexturing which is very unpleasent without large amounts of chamois creme. Also a bummer since the padding was just enough to be comfortable for a 300 - 400k ride without feeling like wearing a diaper (which is my personal point of contention with Castelli shorts).
So, my short and mid-distance shorts are now Performance Ultras, and I've stepped up to a pair of Assos FI.Mille S5 bibs for long distances. The difference between those and the Ultras is significant. The compression from the fabric is just as strong without the roughness of the lower priced bibs. The cut is designed specifically for a forward-leaning posture on the bike, so there is no bagginess or binding. Even the padding is cut specifically for on-the-bike posture. The padding is loftier at the sit bones because they are specifically an ultradistance short, but the wicking capacity is that of a much thinner pad. The covering fabric on the pad is very comfortable, even without reapplication of any creme after 60 - 70 miles. The downsides? I had to save up for 3 months to buy a pair, and they are known to be more "fragile" than other shorts... handwash only, special tech-wash from the vendor, don't hang by the straps to dry, etc. Some people also claim the stitching and fabric in general is not as robust as less expensive shorts, so they're my "tuxedo" of bibs and I won't put them on for anything under 300k.

Different shorts, just like different saddles, will have varied shapes and levels of padding which can make things more or less comfortable, depending on the rider, bike, saddle, etc.