It's not about ride difficulty. It's about ride safety. The point where you can't even hold a straight line is a good start, especially if it's gusty. A passing vehicle can easily disrupt the airflow causing you to

1) crash
2) get sucked underneath the wheels