Reading this thread with interest. My experience went in the opposite direction.

I was interested in a bike for general riding & fitness & was not riding my 1980's steel MTN bike because it was heavy & uncomfortable (it was actually way too small for me). I researched hybrids & ended up buying a very slightly used Cannondale Quick Speed 2 on EBay. The sizing worked out pretty well for me & I spent about half what I would have paid for a new one. The Quick is very "road-bike" oriented, with 28mm tires. The Quick is perfectly comfortable for me, the handle bars allow for a pretty upright position & a great sense of control when used on a bike path that has twists & turns& dips.

I found myself pushing hard with the Quick & soon started to think about getting a proper road bike. I ended up buying a Wilier Triestina Imperiale carbon bike on Ebay. Again, I got a great price, but the fitting issues with a bike like the Imperiale (with an ISP) are definitely a bit more complex than with a hybrid. I am experiencing some neck pain on rides, but this seems to be gradually diminishing as I become more accustomed to the riding position & my core strength improves. In general the Wilier is surprisingly comfortable to ride - the multiple hand positions are helpful with longer rides, but the bars on the Quick are perfectly comfortable for rides up to an hour.

I would suggest keeping the Felt for the time being, getting a hybrid (hopefully at a good price) & seeing how you get on with the two of them. Incidentally, I also bought a Sirrus Elite (for my wife). It's a very nice bike (I assume it's pretty similar to the Sport) - feels more mountain-bikey than the Quick, with 32 mm tires & a geometry that feels less like a road bike. I guess the Sirrus would be more of a contrast with the Felt. As others have suggested, I think riding the hybrid might provide a good re-entry into biking, at which point you could re-evaluate the Felt.