I found that the key to navigating around France by bike was referring to a detailed map, such as the widely available Michelin maps. I saw other maps from other companies that contain even more detail, but I don't think I ever got lost, at least, not seriously lost!
Getting lost (or misdirected, if you prefer) is part of bicycle touring. Asking local people for directions is a good way to figure out where you are. I do it almost every day that I tour.
It can't hurt to carry a compass. I travelled with one for years, but don't recall ever using it on a bike tour.