I test rode a lot of bikes last year for my N+1. Although I was looking at lower budget bikes I could not resist trying a couple of the top end ones. Found a great deal of difference on "Feel" of the bikes so I can understand your problem.

I ride aluminium because I prefer it. I also have a CF Giant TCR-C that is good but set up for the use of hills and long distance. When I first got that bike I felt I had made a mistake. Handling was poor- it jarred the body greatly but it did go up hills. That one redeeming feature has been worked on and although it took time- this bike now works- is comfortable and still goes up hills.

So you now have a decision to make. Only way is test ride both the bike and the LBS and from your comments it looks as though you have made your decision. Not saying it is right or wrong but have another ride on the Felt after getting it better tuned in to you.That may persuade you better.