Nobody will care unless you are in a paceline. As long as you stay out of the pacelines and hold a reliable line, you will be fine. But unless you are a beast, you won't be "racing" if you never draft anyone.