One has to remember that not everyone who rides these segments is trying to set a fast time, or even knows it's a segment. I am the slowest person recorded on any number of routes close to my home, but a lot of the time I'm just riding around. On the other hand, I am apparently a KOM on a nearby climb; thinking back to the ride in question, I don't think I even registered that it was a hill! So all things considered, I wouldn't worry too much about it.