I have this exact problem and I NEVER use a bike computer. However, going hard doesn't ruin the ride for me. It makes me like it more. The problem comes up when I know I should be going easy on the legs.
I agree with the folks who say it comes down to bike choice. When I am in jeans on my fixie I don't give one single crap who passes me. When I'm all johnny-racered up I feel like I have to make a showing.