Last year, I did a couple centuries on my GT track bike (46X14) with front brake. Like you, I have to work to keep up with geared bikes. Even with my high gearing the downs are typically harder than the ups (except for the very rare extra steep climbs). One of the rides was 120 miles in about the same weather conditions you mentioned. I think my average was 19.8. The other one was 105 miles, 60 degrees, no wind or rain, averaged right around 21. I also get the laughs and grin when I show up on the fixie (my only geared bike is my 3 person tandem). Those laughs usually turn into suprise or amazement by the end of the ride when I finish with the lead group and take my turns pulling the paceline, etc.