From past experience of digging in HELLA wooded backyards (haha mude...) Once you get like 2-3 feet down...theres nothign but clay and niice dirt...
But the neighbors dont like 5-6 foot holes that they can see from their windows, but then again nor would they like the "piles" of dirt or "motocross" track you would build there either....
I guess if you have the money its not soo much of a waste to get a truckload of NICE not-so-rocky dirt...