As I understand it, only a handful of people can actually graduate to a level where they are making decent money in this sport. So from a strictly practical standpoint, it's folly to chase a dream like that at the expense of your education.
But as others noted, there's no reason you can't go back. And you'll probably have more focus if you do. I didn't start until I was about 22, having taken 'time off'' to hitchhike around the country and spend a few years drinking heavily in Austin and New Orleans. It was a breeze to graduate at the top of my class when I returned with a bit more maturity and purpose, whereas at 18 I'd have probably flunked out.
So why not give it a shot. But do it with a plan to keep your life moving forward if you don't make it to the big time.