It can mean the difference between washing out the rear in a turn and kissing pavement or barely scraping and scaring the crap out of you. Are you using 165mm track cranks? If not, add at least an additional 5mm in clearance reduction... maybe more since road cranks can also stick out from the frame a little more than track cranks since they normally have 2 chainrings... at an angle that reduces clearance even more.

I'm not trying to talk you out of a conversion, it's what I learned to ride fixed on almost 30 years ago, just saying you need to be more aware of corner speeds (therefore your lean angle) somewhat.

Since you're using a flip-flop, ride your bike as a SS for awhile and remember to pedal through corners to see if you have the clearance to go fixed. If you ever touch down, you at least have a better chance of not wiping out since the ability to back-pedal may reduce the force of the pedal hitting the ground thereby lifting your rear wheel.