Whether you persist in becoming comfortable at closer following distances is entirely up to you. I can share a couple thoughts on the subject though.

1. The company you keep can be everything.

My wife was very uneasy with the idea of riding close to others. We ended up joining a club and the slower groups that she joined did nothing to ease her concerns. The less fit, less experienced, less proficient riders simply weren't particularly good at maintaining a nice smooth predictable pace line. This made it difficult for her to find an comfort in the middle of such groups. When she finally resumed riding with me and wanted to train for an upcoming event I took her out for a couple of paced endurance rides where I would establish a heart rate I knew she could maintain and simply hold a rock steady cadence, shifting in order to maintain that HR. She had an epiphany. We have subsequently ridden in pacelines with both friends and strangers and she's considerably more comfortable now that she can recognize when it's not her and simply a poor paceline.

2. Most of us can't react quick enough to a wheel we are following by 4" if we are focused on that wheel.

Learning to look, up the road, through the other rider, is key to anticipating their future movements. With experience you learn to judge distance while looking at their jersey intead of down at their wheel. This provides you with more forewarning about situations that may result in slowing, swerving, etc. As long as no overlap exists, minor side to side movements are not of great concern and you'll find that the slight increase in wind resistance that occurs with such can be used to fine tune following distances while continuing to pedal and without use of the brakes.

I don't know the extent of your symptoms. But, if you were joining me in a group ride, I would appreciate you telling me about your situation. The knowledge might assist me in providing a safe and enjoyable ride for both of us and anyone else on the ride.