The OP's problem is not just French but typically French. I have run across lots of old French bicycles with poorly installed seat stay caps, Mercier being the worst offender.

Worry not about the problem. If you can see movement between the stay and the cap, then take the bike to a competent welder and ask him/her to either silver solder or braze the joint. Your best bet would be to visit a frame builder to implement the repair.

Or, just ride the bike and keep an eye on the stay. Chances are it has been the way it is since the bike left the factory.