i'm in a very similar situation. 42-15's my favorite, and i'm rockin 42-17 right now on the commuter.
of course it depends on where you ride, but i think your speed will suffer a lot less than you think. you'll still go fast when you want to, you'll just force yourself to spin faster.
the idea that toughness is proportional to gear inches is pretty ********. any p*ssy can ride in a big gear. i used to have a bike that was geared 49 15 and rode it around in traffic all the time, no problem. i may have saved afew seconds gettin to the bar or to work, but it wasn't really improving my fitness.
the ability to get decent power at high cadences is something that actually takes skill. roadies work on it all the tme, it makes you faster in real life situations and it gives you a better aerobic workout. in the winter this is especially important to me because i don't have as much daylight to ride in.
so go for it. and don't think of it as gearing down, think of it as training to spin faster.