I've had the same idea. Although I don't consider myself a "cook," I do enjoy doing my own cooking. Even at home I usually cook one pot meals and always use cast iron. However, there is a 7 lb. difference between camping pot and my small cast iron. That's a lot of extra weight as my normal camping pack is around 35-40 lbs. That being said, I still might try it sometime, so, no, I don't think you're crazy, you just have your own perspective. Also, I hope I camp with you sometime -- sounds like you eat well!!
John