Part of the reason CX bikes are more expensive than hybrids is the fact that they are marketed as a disciplined racing bike which is raced in sanctioned events. In these events, you have restrictions in what components you can use really and the geometry and tube construction. Hybrid is not a race discipline and so, it's not subjected to any of these restrictions per se and anybody can bolt on parts and tube design to keep the cost down. Typically, CX bikes have nicer components. Bike makers put them on so these bikes can be raced. Nobody is buying a boat anchor to race in the local circuit, though I've seen people riding boat anchors beating people out with nicer bikes! But really, the people with the fatter wallets will want a nicer and lighter race bike. A CX bike is that.

However with anything in life, there are exceptions to the rule. There's nothing wrong with riding a hybrid. Riding fast is basically a function of the rider being fit and how that's accomplished is to ride lots and rack up some serious mileage. That's how you're going to go fast. I've personally seen people trying to shortcut the fitness building process by buying into the advertisement that if you ride this bike you'll ride faster. Perhaps. But what usually ends up is that, the person does ride faster because of the more aggressive positioning, but because it's less comfortable the person will be riding it less not more. Unless you have a high mileage bike to complement your race bike which most racers do, then what you'll end up is you'll ride fast for a short distance and the trail behind afterwards because the saddle hurts, shoulder hurts and neck hurts etc plus the back which are not conducive to faster riding anyhow. So you can't build up the mileage, then your performance suffers. A little later on, your nice race bike finds homage in the garage and will then get neglected.

If you want a faster and nicer ride, consider upgrading yourself to a performance hybrid and better and lighter components help make the bike feel nice and speedy while durable enough to withstand trail abuse.