I regularly ride 10+ miles without having my legs burn like they do after 5-10 minutes on the rollers I've been having to stop and rest very frequently.
Every roller/bicycle setup is different. No one can tell what is going on with your situation over the Internet. I'm very experienced and have ridden many roller setups. They usually are never setup with as much resistance as "real riding."
It could be that you are "tensing up" or over heating quickly and this is why your legs hurt prematurely. But as far as difficulty, most people can ride rollers much faster, and much more easily than "real riding."