There is a difference, but it's subtle. As you said, the wider tires are generally more comfortable to ride on, but the thinner tires are generally a little bit faster (on pavement).

If you intend to ride off-road fairly regularly, you might be better off with the 700x37 size tires. If you want to ride off-road only occasionally, in that case it might make more sense to go with the 700x32 size.