People don't like "different." And they don't like change in their routine. They also may feel their manhood/womanhood being challenged if you're making good time on your commute without help from a motor vehicle, and this goes quadruple if you're making 15mph while they're stuck in the single-occupancy-vehicle lane that's crawling at 3mph.
Sorry to hear that you're getting that much harrassment, because I know that stuff really bugs me too.