Look at the missing tooth (teeth) as a percentage of the chain ring. 1 out of 53 is less than a 2% difference - any difference that you note will probably be more psychological than physical. Now going from a 12 to 11-tooth cog, however... yeah, that's a big (<8%) difference.