Everyone is different in their tolerance to cold. I, too, hate cold ears. I use three levels of protection. Below 55 degrees F. I wear a microfiber headband that stops wind. Below 45 degrees I use a Gore Bike beanie that keeps my ears and head warm. Below freezing I wear one of two balaclavas.
Do what you need to do to stay warm and you'll enjoy your rides. Once I discovered how to dress warmly, I began to ride all year round and I love it.