If anything, I enjoy my non-commute rides even more after commuting the last decade or so.
First, I lose the panniers. The bike feels much sportier without them.
Also, I've got a pretty good idea of the safest place to be on a road, given the traffic load and speed, sightlines, visibility, etc. That commuting experience lets me enjoy the other rides, knowing that I'm riding as safely as possible.
I knew going in that bad things could happen to bicyclists. That really hasn't changed. Bad things happen to people doing other things, too. All I can do is minimize the chances of something happening to me, which I'm doing.