in practice this a huge difference. think about climbing a set of stairs that were 5mm higher with each step. over 10K's of revolutions, it makes a difference. I recently went from 175mm to 172.5mm and that was good for me at 5-11" with 32.25" inseam. I use 165-170mm for fixed gear for more clearance.
Not true -- I doubt you will notice any difference at all. 5mm is less than 1/5 inch. Stick with what you've got.