Yeah, it's dangerous. In my first M35+ race this year, there was a crash where two guys went down and both broke bones (one was a hip!). In another race (P/1/2) two of our stronger local riders (Sol Frost and Robbie Robinette) both crashed in the same wreck and broke bones (one collar and one wrist). They're really skilled Cat 1 and Cat 2 riders. There's no avoiding people who crash.
I agree that you can avoid people who take crazy risks, and masters may be the best place to do that. The problem is that bike racing itself is a crazy risk, so everyone in a race is automatically someone you want to avoid
I've been very lucky in that I've only crashed in a race once. It was my fault (I had soloed off, and when a guy bridged, I didn't have the concentration in my oxygen debt to prevent a wheel overlap). That crash was after 6 years of racing.
Other than that, I've had crashes happen in front of me, beside me, and behind me. In nearly every race I've done. I've had guys nail me from the side and go down, and guys overlap my wheel and I could feel them going down. I've ridden over the top of a bike and leg of a downed rider (no chance to do otherwise), and got 3rd in that race. I've caught pedals, slid on manhole covers, ridden straight into ditches, but somehow I stay up.
That said, I'm very afraid of getting hurt. Those guys who crashed in the M35+ pack really shook me up. I swear I could hear a bone break -- it just didn't sound like a bike part. In my little training spill 2 weeks ago, I'm amazed at how hurt I got just falling over at 6mph.
Like you, I'm not sure what my point is, but yeah, it's scary. The adrenaline is probably part of why we do it in spite of the fear.