I'd be willing to bet though, that the average car in Sweden is not a 5000lbs+ SUV that gets less than 20mpg on a good day.

Speaking of small cars, in Japan the government offers a lot of incentive to people to drive small cars - they even have their own government defined/controlled class, the Kei Jidoshi. The government sets the limit on engine size, and on overall exterior dimensions. The government taxes owners of these cars far less, does not require them to go through the rigorous (and extremely expensive) inspections the larger cars do, cheaper registration, they're allowed to park overnight on city streets (maybe big cars can too now? I'm not sure) and of course, they cost less initially and in upkeep.
Somehow I doubt the U.S. would ever adopt a program that makes as much sense.
I should also point out, that while the engine size of these cars has risen dramatically over the past 40 years, their fuel efficiency hasn't shown as dramatic a corresponding decrease (the engine size was allowed to be larger to compensate for emissions controls, and the exterior dimension were enlarged ever so slightly to allow for stronger, safer, bumpers,etc.)
The bottom line is though, that consumers will always opt for the most they can afford (or even what is beyond what they can afford, usually) and auto makers make a lot more per unit sold on big cars than on small cars.