No, you aren't crazy for considering winter riding. I ride all year long and have for a very long time. There are adjustments to be made however. (For lots more comprehensive information look at the Winter forum)
Clothing: You will need warm clothes. It's a fact. How much you need depends on how cold it is and how long you are going to be out. Be aware that you will not need as many extra layers as you might think you need. If you are warm when you step out of the house, you will be too warm when you get going. Start cold, end warm.
The clothing that you need is different from what I, or anyone else, needs. The deal ender for me is when my feet get cold. I have a friend who can't ride if his hands get cold. Some people can't stand having a cold head. It all depends on the individual.
Temperature is also a factor. I can comfortably ride down to around 15F. At that point it just becomes too painful. There are lots of other crazy people out there who don't stop at -15F. Maybe they have lungs of steel or maybe it's that their air has more moisture in it (Colorado is pretty dry and dry air is damned cold!)
Wet: Depending on where you live, you may be riding in wet conditions. Don't ride trails when they are wet! Especially if there is a chance they might freeze because ruts are forever! Where I live, I'll still be kicking up a dust cloud even in January so I don't have to worry as much about it as other people do but I still stay off the trails if it has snowed and started to melt.
Snow and ice are slippery. Be careful about riding on them unless you like to crash. Snow and ice also clog your drivetrain. Cables get balky and derailers won't shift. Leaving your bike in a cold garage rather than a warm house can help because the bike is the same temperature as outside.
Also, if you are wet, you risk hypothermia. If you ride in the wet use fenders and clothing to keep from freezing.
Light: The sun goes down
early in the winter. Really early. Pay attention to time and be off the bike by dark or get good lights...not some cheap little flashlight ... but good lights! And keep them charged! I ride both mornings and evenings (commuting to work) and I've observed a little known fact - it gets dark when the sun goes down
Go. Have fun. Be warm (but not too warm). Watch the clock. Stay dry.