No that'll work just fine. You may actually end up with even tyre wear front & rear.
However, I just don't like cornering with 20mm tyres on the front though. Due to the high pressure you have to use to prevent pinched-flats and to get any kind of rolling-efficiency, they skip and bounce over bumps in the corners badly. Not good for doing downhills or crits with confidence.