Riding slowly in and of itself shouldn't hurt your performance but if you want to build up fitness to go fast you'll need to go fast, if you want to build up endurance to last longer (no innuendo here
) you'll need to ride further and longer.
When I ride with my wife I find the speed to be verging on painfully slow but it means we both get out on the bikes. Then when I'm riding on my own I get to go faster, longer, or both. If all I ever did was ride with my wife I'd imagine my fitness and endurance would slowly decline.