I like being able to participate in a race. That means more than just sitting in, that means actually racing my bike and trying to get a result. Because I'm not particularly talented, that requires training. I can't show up to a Cat 3 race with no training time under my wheels and expect to do anything. With more structured training, I have better races because I have the specific ability to do what I need to do. If I don't take training seriously, I'm going to get dropped in the harder races I do, and that's just not fun. If I burn out on training, I race less. I guess my reasoning is that if I'm going to put in the time to train, I'm not going to train enough to hang in on the wheels in a Cat 4 race, I might as well try to get as fast as I possibly can.
Thing is, I'm not particularly "hardcore" about racing, but I do spend somewhere around 8-10 hours a week on my bike, sometimes a little more, sometimes a little less. I don't think the amount of time I put into this sport is particularly intense. Your mileage may vary. Keep in mind that the intensity you see in the guys who are more into the training and doing lots of racing may be more a product of your perspective than theirs. They might not be as focused as you think they are, they just might be more focused than you. Nothing wrong with that. If you're having fun, that's what matters.