I've had several situations where one car seemed confused that I didn't accept their wave through when there was at least one other lane of traffic, often more, that might have run me down, yet the wave-through driver seemed clueless. That said, if all available lanes are either free of traffic or stopped and waving me through (as in the OPs case), I do graciously accept (although, generally, I think it would be faster if people let me wait until the way is clear -- unless I already have the right of way).