I would start simple. Go to a bike shop, get yourself a cheap helmet (but one that fits), some good shorts, and perhaps a jersey. Oh, and gloves and glasses are an excellent idea, and it's nice to have a cheap computer to track distances and such.

The first big upgrade that most people make - and that for me draws the line between somebody who just rides a bit and a cyclist - is clipless pedals and cycling shoes. What you choose there depends on your aspirations - there are road-specific setups, and off-road ones that can also be used on the road.