Just installed a caliper brake (shimano 105). One question, even though I have tightened it pretty hard the brake can still move a bit from side to side. is that normal? The problem with that is that the brake centering will be affected...

You should have had 4 pivot bolts included when you bought the brake (these come with the retail packaged brakes), try a shorter one, as the one you have is too long, as you can see by the gap between the spacer nut and the fork https://techdocs.shimano.com/media/te...9830714256.pdf

Ok, will try with that!

The brake should be centerable with a moderate amount of force, by hand by moving the caliper. However it shouldn't be loose enough that it doesn't maintain the positon it is put in easily...

Yes, but there a 0.5-1mm gap visible in this example, where there should be none, this indicates that the pivot bolt is being used is too long, and a shorter one should be used.

A shorter bolt was used which made it better, but there's still a small gap because it doesn't sit really tight against the fork..but that's more on how the fork looks at that specific area.