i don't think you are out of line at all. It is cool to see people givin' er
With fixed gear on my usual route (very hilly) I am always in this weird situation where I have to pass people on uphills, or I lose all my steam. Then they just smoke me on the flat land and descents. Then I encounter them again at the next uphill and have to pass again
The paved loop you described sounds like a lot of fun, I would expect that kind of fun chasing if I was doing laps on such a loop