My suggestions:

1. Determine what you goals are, weight loss, exercise, site seeing or maybe all of these
2. What type of terrain will you be riding, flat roads, bike paths, hilly areas
3. If you are brand new to riding and unsure of whether you will stick with it purchase a bike in the $500-$1000 range based on your budget. If you need to spend less look at a used bike.
4. Depending on #2 determine if a road bike, mountain bike, hybrid or comfort bike is best. Odds are a road bike may not be you best choice if you are brand new to riding.
5. Start out easy, say 5mi every other day and build up your time. Note I said time, as it is more important to be riding for longer periods of time rather than faster speeds, more hills etc.
6. If you want to really get into riding then look at picking up Friel's Cycling Over 50
7. If you find yourself doing >1000 mi per year be prepared to spend more money as you start upgrading