I was pretty out of shape when I got my track bike(it was my first bike in probably 10 years, and I had pretty weak legs/core). It came with a 15t(x48chainring) and I found that too hard to peddle to be honest. It was alright on the flats, but accelerating, hills(even mild ones) and headwinds really blew. I switched to an 18t cog and it made a world of difference. MUCH more comfortable and accelerating feels great. And spinning downhill doesn't feel sketchy until about the mid to upper 20's , and that's probably because I still have bad technique.

So if you are at average/poor shape in regards to leg strength/endurance - I would recommend going with the 16t(hell, I would say even higher for "commute/joy ride".) Switching my cog from 15t to 18t was the best thing I've done to my bike by far.