Some pics of the setup will help others who run a rack and panniers to see if there is anything wrong there. I've found I just don't like the way racks and panniers handle, but that's because I like aggressive bikes. Something like a Fuji hybrid should do fine. Also, is this while you're sitting down and riding, or standing up and pedaling? Anything specific that you were doing on the bike that would trigger the feeling?