That's a good question, and I don't have an answer, either.

I recently replaced the spindle on my old Klein. It uses pressed in sealed bearings. I can confirm that new sealed bearings have some drag because of the seals. I think the drag is supposed to decrease after the break in period.

I replaced the bottom bracket bearings when I changed the spindles for two reasons. First, the force required to press the bearings out of the shell and remove the spindle is substantial. So there is a chance of damage. Second, the bearings were old and I don't really expect to do this again anytime soon. I didn't inspect the old bearings, but even after removal, they were smooth. BTW I bought Phil Wood bearings, which are supposed to be very high quality, for a little over $10 each and shipping.

Should you replace them? I don't think it's likely you'll experience catastrophic bearing failure. You may not even notice a difference with the new bearings. But if you decide to replace them, you'll be set for years.