I think you will be surprised at how strong you become and how you will increase speed even with a loaded heavy bike.
Right now, you might be huffing and puffing and feel a little bit like granny gritting your loaded bike uphills, etc. However, you will get stronger and soon find yourself hopping on the heavy machine and quickly getting it up to impressive speeds.
I have had friends drive by me in their cars and call me later saying it was quite a sight to see a guy going fast with a loaded bike like it was just the way it was meant to be done.