Did you use the same shimano PTFE coated cables?

The routing for 105 and really all the under-the-tape shifters, is very dependent on low friction cables. If you used non-PTFE cable there could definitely be additional drag in down shifting.