Track frames have higher bb's than road frames; if you've got a road conversion you may want to consider going to a smaller crank arm. But if you've had no troubles, don't switch.
I had 175's on my road bike, switched to 170's and could really tell a difference in my cadence and no difference in my climbing ability. I'm 5'8" with 30" pbh, if that helps. I did have a nasty pedal strike on my fixie with 170's; launched the bike into the air as the pedal hit the ground while I was in a high speed turn. Almost lost it but survived unscathed. Not a fun thing. My next fixie build up will have 165's.