Frame materials can be and are manipulated to take advantage of their strengths, and minimise their weaknesses. Most quality Aluminum frames don't live up to the (brittle and harsh) stereotype, and most high end steel doesn't live up to the (heavy) stereotype. If you can ride them you are probably better off closing your eyes to the material, and deciding by the seat of your pants.

No one has mentioned Cannondale track frames, and I think those look really nice. I hear the older ones ride great too. I'd like to find out for myself sometime.