The small difference is largely due to how power needed to overcome air resistance is proportional to speed cubed, and at speed most power goes into overcoming air resistance. Because of that, doubling your power only increases your speed to the cube root of 2 or by about 26% on level ground, and halving your power only slows you by about 20% there.

It's a bit different going up a steep hill, however, as there most energy goes into overcoming gravity rather than air resistance. There, if you lose six watts and drop from 100W to 94W, the difference will be approximately linear, so you'll be about 6% slower -- so it'll be more significant there, but still relatively small.

The 50% efficiency guess is probably fairly close to accurate.