Maybe it's your bike, but in my case I own a Trek FX (hybrid, 700c x32 tires) and a Cannondale CAAD10 (road, 700c x23 tires), and between the two the CAAD10 handles a lot better. I had been commuting on my FX for a while, but since buying the CAAD10 I've enjoyed riding it so much that I have been riding it to my office (I keep it inside). I throw the CAAD10 into turns and it just goes, the harder I push it the more it takes, and the ride to work is a real blast. Today the weather wasn't looking too hot so I took the Trek, and I guess I have gotten too used to the cornering speed on the CAAD10, because taking turns on the FX I was going into them too fast and the bike was not responsive at all. I kept saying to myself, "jeez, I have to really manhandle this thing into turns," whereas the CAAD10 just goes. I've also ridden my CAAD10 down dirt trails, on gravel paths, and across fields. Mind you, not for extended lengths, but for 100-200m at a time. Maybe I'm just ignorant, but the CAAD10 doesn't feel horrible on this terrain, even with x23 tires. I know it's not ideal, while I do have to slow down for rough terrain, but I never feel particularly afraid if a rough section is coming up. I just try not to make it a habit, even though I think a few scratches and chips add character to a bike
... In no way do I think my FX is a bad bike, in fact it is ideal for commuting and fitness riding. I love my CAAD10, and it is definitely a lot more bike than I am capable of pushing to the limit, but I think that is part of why I enjoy riding it so much. After riding the hybrid when I get on the road bike it just feels like it wants to go faster, smoother, and farther. If you don't feel this way when switching between a hybrid to a road bike then something is seriously wrong. That road bike should be capable of infusing confidence, enthusiasm, and joy every time you get on it.