In approximate order, I'd start with:
(1) Saddle bag -- look for the biggest thing your bike shop has for under $30 or so.
(2) Spare tube -- ask shop for one that fits.
(3) Tube repair kit, for when you have two flats.
(4) Pump, Topeak Road Morph
(5) Tire irons, if you need them.
(6) Bottle cage and bottle. I like the Polar insulated bottles, filled with ice and topped with water before leaving.
(7) Cyclocomputer. Much cheaper than a GPS, and the batteries are much, much more likely to have a charge.
(8) I assume you already have a cell phone.
(9) Rear blinky. Check your LBS and pick one. Or two.
(10) Pick up a multi-tool if you want. Park's don't rust as fast as some others.
(11) OK, get yourself a helmet if you want one.
(12) and a cheap blinking front light. If you're going to ride at dusk, dawn, or at night, a big honking light.
Of course, I'd actually start with item (0), cycling gloves. Almost forgot these, since I have a few. They help absorb road vibration and shock, provide a nifty face wipe, and if you sweat too much, help you hold onto the bars anyways.