I like to consider the laws of physics to be suggestions.
Anyway, a lot more replies than I expected! In case you're wondering, the reason I'm trying to "compare" my average speed is because I'm a college kid without a job who has all summer to train as much as I want each day (nice, right?) and it's really the only method of comparison I have to see how well I'm doing. Racing's not too popular where I live and I won't be able to get into that until I go back to school in the fall, so I can't compare myself against cat-whatever racers. And there's no way I can afford a power meter. So that pretty much leaves average speed.
Although... I just realized there is a perfectly flat 1 mile loop around a local high school. Once the school is empty for summer, I can go and time trial around it and I think that would give me the best basis for comparison.
Anyway, thanks for your all's help! It does seem like the hills can indeed make a good bit of difference. In regards to the climbing, I'll take 10-15 minutes going up a hill on the way out, and it takes me like 5 minutes on the way back in, so I do spend a lot more time at a slower speed.