For sure, do what works for you, but are you sure you have the saddle bag mounted correctly? It's a fairly common occurrence for people to just strap them on without paying attention to how exactly it should loop through the rails.
Alternatively, if the bag is too big, consider a smaller bag.
However, top tube bags are handy in the right applications. I used one on a long night ride to keep the food stuff right in front of me. Not that I knew how to properly eat for a long effort, but the food was still there.
