The main difference between the "cheap" folders and more expensive ones are...

Weight: The cheap folders tend to be significantly heavier than the more expensive bikes.

Components: The components on cheap folders tend to be a step below those on more expensive bikes. For example, a bike might advertise Shimano components, but it will be from Shimano's "cheapest" line.

Another, less tangible, difference is ride quality. This is mostly subjective, but there can be design differences that compromise the ride quality on cheap folders... and honestly on cheap bikes as well.

All of that said... find yourself a cheap folder to start with. Or, if you can, get a used one from craigslist or eBay. I doubt you'll be totally disappointed with a cheap or used folder. If you have a question about a specific one that you see on craigslist or ebay feel free to post about it here. The folks here will be more than happy to help you.

I bought my Giant Halfway used on eBay for $226 shipped. It normally goes for $500. Since it was used, I'm learning how to fix the things that need fixing and I'm learning what I do and don't like about the bike. I have a severe case of upgrad-itis... and I know before the summer is over I'll likely have a Bike Friday Tikit (saving my pennies as we speak).

--sam