The initial costs might be pretty high, but you will definitely save over the long term.

I've driven my car less than 500 miles in the past three months. Before I started riding, I would have easily put 2000 miles on over the same period. Those 1500 miles pay for the cost of the bike if I use the IRS's mileage rate.

I've also found that a lot of bike stuff is "nice-to-have". I don't have fancy cycling clothes, I use regular platform pedals, and I got my panniers for $40 at a big box store. Frame pump, lights, patch kit were Christmas gifts. I've also somehow avoided any flat tires in the year I've been commuting. My maintenance supplies (lube, tools, spare tubes, etc.) were all given to me by my wife's father since he wasn't riding anymore.

Oh, and I've dropped my auto insurance down to "recreational" and saved about $300 a year.