I wouldn't worry about it too much. Too many times I've let stupid comments from motorists ruin perfectly good outings. Not anymore.
If you weren't breaking the law then you have nothing to worry about. Has a cop ever pulled you over for riding where you do?