That's the safe decision. Happy wife, and all that.

IMO...She's participating in an activity that you're serious about, and when you ride the way you like to ride, you dress in "serious" attire. When she participates, and you wear casual clothes, it gives the appearance that you're dumbing things down to ride with her. I can see why she might not be thrilled about that. While you're correct that the advantages of lycra are diminished at low speeds, there's no disadvantage. If wearing the same clothing you wear when you ride by yourself helps keep her motivated to ride with you, it seems like an easy choice.