I ride with a lot of locals who race in crits and road races. Invariably they are involved in some sort of crash from the movements of other riders. One of my friends is off the bike for 4 weeks due to cracked ribs and a collapsed lung. I enjoy riding hard with my racing buds but the groups are smaller and their moves are very predictable and safe. Sometimes I'll "win" a KOM or a sprint to a County Line sign but most often I'm dumb enough to have done the hard pulls to get them there.
Red Rider makes a great point. Time Trials are a terrific way to "race" in a more controlled and what I think is a safer element. It is also an excellent way to compare yourself not only with yourself but with others in the same age group or riding condition. I help put on the Time Trial series at Lowes Motor Speedway in Charlotte which is open to riders of all ability levels ages 10 and above. Our oldest regular participant is 81 and usually finishes in the top 2/3 of the field. I suspect there are similar events around your area as well.