weird handling problem
 

Today i swap wheels and i replace stem on my road bike. The stem is was 120mm i put stem 100. Now when i ride the bike i feel the fork is not stable like in speed 25 miles. I had the wheels in my other road bike and no do that problem. Is can do that because the stem is shorter.? Plus i notice the fork is more tight now i lose the stem but the fork stay tight. Is turn left and right but when the wheel is straight is stay like that even if i pickup the wheel from the ground. How i can loose the fork.

If the fork on the right way round?

More information is needed and add photos. Threaded or threadless? My guess is you have the preload to tight on the headset. Go to this article on the Park Tool Website and it will tell you how to adjust your headset.

Stem length doesn't make much difference, especially only 20 mm.

As suggested, your headset is too tight. Pictures ? Or search youtube for your type of headset.