Move about 10 feet to your left, then pass. Almost no chance of spooking someone if you give them enough room. If you pass closely, there is a good chance of spooking them no matter what you do.

Note that in some states, it is illegal to pass a bicyclist by less than 3 feet. Those laws are mostly aimed at cars passing bicyclists, but another bicycle passing closely can be just as dangerous.