Sure, getting more aero on the bike matters. But you can improve aerodynamics without sacrificing comfort. Instead of dropping the bar into ergonomic hazard territory, wear a more snug fitting jersey, zip it up, and work on the core to maintain a more aero position. (When I'm tired my elbows flail around like Froome's, but unlike him I'm not a human-pterodactyl hybrid with freakish power and endurance to compensate for looking like something that flew out of mythology, stole a bike and went racing instead of flaming villages to the ground.)
I split my rides about equally between a hybrid, an old school steel road bike (Ironman) and a now-old-school carbon road bike (1993 Trek 5900). The aero differences matter more than bike weight. I set up each bike so I can ride 20-30 miles without a break at a spirited pace (for me). That's just my personal loosely defined benchmark. With a bum neck, back and shoulder from multiple injuries, that's a reasonable guide for me and happens to fit the group rides I occasionally join.
I average about 12-14 mph on the hybrid, depending on how much effort I feel like putting in, and how much crap is on the bike. It probably weighs 30 lbs without U-lock, handlebar bag, etc. Definitely slower with the lock and bag, but those are for casual rides. It's an early '90s Univega, the frame slightly too large for me and a bit stretched out, but I've compensated with albatross bars set upright at saddle height. Very comfy for long rides, reasonably aero. I've tried flipping the bar, path racer style, but prefer the upright hand grip position. I usually wear casual clothes on that bike so the flappy baggy shorts and shirts add some drag.
Platform pedals and casual shoes on the hybrid; clipless on both road bikes. Not a huge difference, really. But my pedaling technique is kinda sloppy.
I average 16 mph on both road bikes. On a good day, closer to 18-20 mph including yesterday, but those are exceptions -- depends on wind, terrain and whether my personal trainer-shaman's incantations and potions are working.
The Trek is 5 lbs lighter but I only notice the difference on climbs. Our climbs aren't long enough for the weight advantage to matter much. It'd matter more on a 50-100 mile or longer ride, especially with lots of short, punchy climbs or long continuous climbs. On flats and downhills, nah, I don't notice any difference.
Both road bikes are from that era with less radical drop between handlebar and saddle, and a more stretched out riding position. I'd guesstimate about 2.5" drop from saddle to handlebar.
The 58cm Ironman now wears a shorter stem (80 or 90mm vs the original 120-130mm), which offsets the longish top tube. The 56cm Trek has the original long Ibis stem, probably 130mm. Riding position is approximately the same with both, although there are some differences between the Ironman's old school aero brake hoods (and downtube shifters), and the Trek's brifters (I replaced the original Dura Ace downtube shifters). The longer brifter hoods add a bit to the stretched out feel. Rather than swapping to a shorter stem I'm doing physical therapy now to improve my core (and for the shoulder/neck injuries from being hit by a car).
I work on using the drops but it's less aero than riding the hoods, bending the elbows to get the forearms as close as possible to parallel with the ground. Mostly I use the drops for a change of position when the neck cramps, for maybe 1-3 minutes at a time before riding the hoods or tops again. Also depends on whether my thigh adductor muscles are cooperating. That's the main difference I notice between bar positions. I tend to spin so it matters more how efficiently I can lift my legs for the next downstroke.