Get a Topeak Super Tourist DX Tubular Rack. It'll drop your panniers by 1", which should have essentially the same effect as lowering the BB. Which, by the way, doesn't really make much of a difference.

The Jamis has a wheelbase of 40.43" and a full touring bike like a Trek 520 has a 41.3". One inch on the wheelbase also isn't going to make a whole lotta difference.

My guess is that if things feel weird, you're probably just putting too much weight exclusively on the back. I'd try moving some weight onto a handlebar bag or a small front platform bag before ditching a perfectly good bike / frame.