Only one night of stealth camping under my belt (just did first tour). It didn't really cross my mind to camp later in the evening. I was camping through heavily wooded areas, so once I pulled into the woods 10 feet, on a hill above the road, no one could see me and I wasn't concerned about someone wondering into my camp. I was setup several hours before dark.
I would hate to setup camp in the dark and would really be reluctant to do it stealth camping. I've never camped in a really audacious setting, though. I think it all depends on the how visible your campsite is in the light.