Singlespeed & Fixed Gear - How important is exact chainline with a single speed bike?

I'm about 3-5mm off...my freewheel is at the far left where it's supposed to be, and my crank sits as close as my chainring allows it to, just barely hitting the chainstay! I'm a few millimeters off...any sort of disaster I should expect with this setup in the long run?

When eyeballing it, the chain looks pretty straight but it obviously isn't...

(It's been converted from a 12-speed by the way).

Here's some good info
http://sheldonbrown.com/chainline.html

it will be noisy, cause wear, and maybe a slight chance of dropping the chain.

The risk is dropping the chain at an inopportune moment. You can minimize this by using track cogs and rings with full-profile teeth (road parts usually have truncated teeth to help shifting).

The crankset is a track/single speed crankset, and the freewheel is a pretty tough spikey BMX freewheel...I'm a little worried my aluminum chainring will wear out, though. I might just go with a freehub wheel...

If it isn't making a lot of noise, don't worry about it. Sheldonbrown.com has suggestions for adjusting and calculating your chainline. Road chains are more flexible and tolerate a slightly off chainline better than a track chain.