People think I'm crazy but I can feel a difference on steel bikes. I feel like the aluminum bikes have high frequency vibrations that steel doesn't pass along. I have noticed it in my car too, going from the steel version to the newer all aluminum version of the car.
I have an aluminum bike on its way but with a steel fork. I'm curious to see how it feels. Its a touring frame so I'm hoping for more relaxed position and wider tires. I think it should be comfortable.
You should ride any bike you want to.