occasional few miles of "dirt road" as part of the same ride? If so, how "occasional"?

Pick and configure you bike for what you ride the most. If you only occasionally ride in dirt, You can either keep and modify another bike appropriately for that occasional ride. Or simply have two wheel sets and quickly swap out normal road tires for up to whatever the max width is your frame will take. If you are good at changing tires, you don't even need the extra wheel set unless you get super narrow rims.