Sounds like too much weight up front. I have limited experience, but I doubt if I have ever had more that 5-7 pounds in each front pannier (what does a six pack of Samuel Adams weigh?) I can no hand on my fully loaded bike with about 10 pounds up front, 20 in back, down a gravel road (converted MTB with 2" tires). If the shimmy was less after eating some of the weight, that might be it. I think my top speed loaded, however, has been about 30 mph heading downhill.