Yeah, good question.
I do that a lot. Some things I consider:
- What tire size do I need (how rough is the terrain)
- What size handle bars do I need (drop bars are much narrower and less precice at low speed than drop bars)
- Is pedal strike an issue on your frame?
- Is geometry appropriate? Like with handle bar size, what I want at 5-10mph may be different than what I want at 20-30mph).
I like riding my cross bike off road because I like the challenge (and have the skills). A gravel bike may have more of a problem with pedal strike though.
I like riding mountain bike on the road, because I like the challenge (LOL). I'm using '90s mountain bikes because those are fast and extreamly versatile bikes (unlike today's more specialized bikes).