Wear the jersey and shorts into the shower and clean them at the same time that you clean yourself. Put them in for the spin cycle in the wash and they'll mostly dry out. They'll be damp when you put them on for the afternoon, but not stinky.

Alternatively, here's another way to look at it. The Team kit shows your sponsors. For max sponsor support, wear it for the group ride so that the more people see it. If there's only a couple of folks at the track, it's a small drop compared to a group ride on the road.