I don't think of it as a race, but I do typically set a pretty fast pace and will pass people (when safe) if they are slower then my pace. I have learned sometimes I will pass someone only to get passed back, either as they are faster climbers or a part where I tend to be slower. I have learned to try not to pass people in these areas where I think they may pass me back quickly. I prefer to challenge myself, such as improving my speed/time on on of the climbs on my route, and I do like to get to/from work quickly most days. Other days or parts of the ride I might decide to take it easy.
When I do get passed, I will sometimes try to see if I can hold pace with them for a bit (but not pass them). Its more of a challenge to see if I can hang.
The only times I am really annoyed being passed are 1) when its by someone with electronic assist, and they buzz uphill without breaking a sweat as I am working my tail off. 2) when someone passes me only to slow down below my pace right after passing me