Physics doesn't care if it's bike weight or body weight, except for minor issues such as the height of the center of mass. There is one significant exception: reducing rotating wheel mass can deliver performance benefits greater than just dead weight savings, as reduced effort is needed to accelerate the wheel rotation. A Schwinn Varsity with racing wheels & sewups can sometimes be faster than an elite-level frame with steel rims and heavy tires.

My primary bike, the Nashbar Flashback "Alumicruiser", could weigh as little as 27 lbs, like its faster near-twin the Alumispeedy. But I have found front & rear racks, fenders, a front basket, and a 2-leg kickstand to be remarkably useful, and so its base unladen weight is around 36 lbs. My older Ross cruiser is about 2 lbs heavier, almost all of it in the frame (light aluminum vs. old swing sets). I like them both, and have over 10,000 miles on the Nashbar and over 21,000 on the Ross.

But for a typical ride I'm leading, I add:
2-4 water bottles at 2 lbs each
my pannier with full tool and first aid kit + long cable lock
my side basket with my fabric cooler in it for leftovers
...plus me, a good 50 lbs over my Cat IV racing weight.

On either the laden Nashbar or the Ross, I can ride comfortably at 11-13 mph all day long. Fortunately, my group is very happy with that pace.

If I switch to the Alumispeedy (same basic frame & wheels as the alumicruiser, light rear rack & racktop bag), I can ride at 14-16 mph at the same effort level, and climb faster. And if I switch over to my Raleigh Pro at 24 lbs full-up with spares & tools, I can ride at 17-19 mph without too much exertion. Spending most of my time on heavier bikes makes the light ones faster when I switch over due to the extra training.

On the other end of the scale (literally), my Chicago-forged chrome Schwinn cruiser with Wald front & rear baskets and steel fenders weighs just under 60 lbs before you put anything else on it. That bike provides a "quality per-mile workout" at 10-12 mph. But it has a 7-speed hub and aluminum rims which help a bit.

Set your bike up for what you need it to do. If you want a faster ride, evaluate if you need to keep items on it, look into wheel & tire upgrades, or better yet obtain a second bike focused on speed. But remember that weight-based speed changes tend to be evolutionary, not revolutionary, unless you splurge for a 15-lb wonderbike. Then, wheeee.

But your bike(s) is/are your bike(s), and unless you're racing or Stravaing, ride the way you want to without comparing yourself to others. And have fun!