I don't race everyone, but I keep a decent pace. If it's not safe, or convenient to pass/be passed at that spot, I adjust my speed.... even if it's several MPH up.
I generally pass most of the roadies in my area, except the select few keeping a 20ish mph pace or higher. MOST of the time, if I can't keep my pace, I simply stop instead of going slow.

Although, when riding with my girl... she is pretty insecure, and new to biking.... the pace is considerably slower. She goes out for leisurely rides, not fitness/fast/fun rides Atleast she'll do 10miles without complaining... toooooo much.