It would make a great fixed gear or single speed, or a three speed. Do you know anyone who rides a fixed gear? I was never very comfortable with a fixed gear until I went riding with someone who knows how it's done. That somehow changed it for me.
I'd guess you can find a better touch up paint, cover over the orange bits on the chain stays, and it will look pretty great. Being a little 'scruffy' you can get away with components that show a little patina. I'd definitely go with an English cottered crank, just because nothing else will look quite right. Similarly, straight pin seat post (brand doesn't matter) with a clamp on top, rather than any kind of post with integral clamp. A chromed steel 'lugged' stem, or a girder-style aluminum one. I'd try to get this crank (or similar)
http://hilarystone.com/images/sale%2...ms-C34_sml.jpg
and this stem (or similar) from Hilary:
http://hilarystone.com/images/sale%2...GB75mm_sml.jpg
Any thoughts on the 'flip-flop' option? Have a single speed freewheel on one side and a fixed gear on the other side of the rear hub? That might be the best of both worlds, although, I am not sure how difficult that would be to get used to.