I don't think you can know exactly how good you can get; you just have to dive in and see. jppe makes good points. Improving much beyond the initial jump will, indeed, take some suffering.
stapfam's assessment seems spot on. I'm in my third year of riding now, and seem to be seeing my first major improvement since that first year. I think that riding with cyclists who are faster than you helps a lot, too, and you already appear to be doing that.