There's no single right answer. Do you want one bike that can do everything, or will you think about getting more than one? If it's just going to be one, a hardtail or rigid mountain bike would be pretty versatile...slick tires will make it reasonably fast on the road, (but not as fast as a road bike) and you can switch to knobbies and go way offroad. Others may recommend a hybrid bike. A cyclocross bike is set up like a road bike with dropped handlbars, but it is sturdy and can fit somewhat fatter tires for some off-road riding. However, it wouldn't be as agile as a mountain bike in really rugged terrain.

In terms of cost savings, buying second hand is the best way to save, but you can expect to spend some money on rehabilitating what you buy.

As I got back into cycling in around age 40, I bought both a cheap road bike and later a cheap mountain bike (both of them were pretty beat up and cost around $200) and had my LBS fix what needed to be fixed.