I'm actually a bit surprised that V-brakes are capable of pushing a rim across; this would be due entirely to the difference in return spring strength between the two arms if nothing else is wrong...

Sticky pivot maybe?