i've got an xl as well. when worn near empty it'll go from over my left shoulder down around my ribs on my right side.
the bigger problem, is that since it's so big, the strap has to be that much longer. when i cinch it down for a ride across town, i use nearly all the available strap - so there is a good portion fluttering off to my side.
i'm about 5'8", around 160, so i'm not necessarily small.