Unless it's pulling /squishing the brake/shifter cables, that shouldn't be an issue.

As for being the slow rider - you are aware that you're powering the bike as well as steering it, right?
If you want to go faster, you have to pay for it - in sweat. But if your regular daily routine B.C. (before cycle) didn't contain much physical activity it will get easier after awhile.

Apart from that, listen to Drew. Going easy on the pedals while shifting will make the gears go in ever so much easier.