Look into cyclocross bikes.

With a tire switch (or second set of wheels) you could do both loaded touring and faster rides. Of course you may not be able to put large panniers on a cyclocross bike because of the shorter chain stays.