I just started riding fixie not too long ago. My first couple rides were stressful but I think this was largely because I was being more timid than anything else. I was wobbly because I was afraid of picking up speed and not being able to stop. Once I realized that it was probably more dangerous to go slow because of fear of speed than to move at a decent clip but be in more control I was a lot better off.

Might not be applicable to the OP's situation but it helped me get from completely sucking to moderately sucking.