It's most likely that your rack is flexing around. Also affecting the matter is the placement of the load - putting a rucksack on top of the rack will cause a lot more instability than putting it on a pannier on the side, much lower. I would consider getting one of the rucksack/pannier combination things, so that you can get the benefits of positioning of a pannier, and still throw it on your back when you get to work. On my 700c bikes, I always carry heavy loads on the side, and it doesn't cause me noticeable problems. On my Raleigh 20, I use a rack top briefcase, but I'm perched so far above it that I suspect it works out about the same!