Should it take 4 hours to build a bike?! I think I slapped my last one together in about 45 minutes.
It depends on your experience level and how much work you want to put into setting everything up.
Yes, you can "slap" something together pretty quickly, but for someone building a bike from the frame up for the first time, and wanting to do everything correctly, 4-5 hours isn't unreasonable, especially if you get into truing the wheels and chasing/facing the bottom bracket, adjusting the headset bearings for a threaded fork / cutting the fork for a threadless headset, taking the time to properly grease everything beforehand, getting everything fitted/adjusted properly, etc., etc., etc.