Listen to your body, and take the ride home at whatever pace is comfortable. Think of it as a recovery ride.

Parking is difficult by my gym, and it's close enough ( 2 or 3 mi ) that it's silly to drive anyway. I do this often, and it's never been a problem.