I think it's reasonable to say you'll notice the difference between a 32 and 38 tire. As mentioned, any of those sizes will be fine. Personally I'd go with the 32 or maybe 35, as you'll be faster without (IMO) losing a lot in the durability or comfort areas. If the dirt areas you're talking about are somewhat rougher, the 38 may be the better choice.
Whatever size you choose, request a slick tire. Don't let them put something on like this:
http://www.bikesomewhere.com/images/...2D7BBE08A6.jpg
A tire with a tread pattern like that will be noisy and slow without providing any useful grip on or off the pavement.