I've done that a few times ... and was seen doing it by town folk ... and no one said a thing to me.
I have also tried to do that at some buildings, and have discovered that they put a little cage around the tap with a lock so I could not. I guess if they have a problem with people coming by and using a little bit of their water, that's the solution.