I didn't know there was a time limit on removing spacers. Spacers are added or taken out to adjust and fine tune the fit of the bike to you.
If you want to slam your stem down so you can look like the cool kids your body may have other ideas.