All great responses, however you've missed the OP question.
Biking speed depends on your continuous power output to overcome the specific forces that slow you down - friction, air resistance and gravity.
Therefore the answer to
"how slow is too slow" is your variable output power that is not sufficient enough to keep your bike upright while moving in a straight line.
