The Varsity is a special bike with it's super-relaxed frame geometry and sturdy kickstand keeping the parked bike ever so well-planted.

The easiest upgrade process might be to find a better bike having 700c wheels to act as a parts donor.

My 1964 Varsity went together in a day using discarded parts I had laying about, I only had to buy the inexpensive SRAM 8s cassette, a longer chromed seatpost and the green tires and handlebar tape. See below.
I would leave the original crankset in place and maybe just upgrade the parts that you find a need to improve upon.
700c wheels work with the original brakes and lower the bike slightly which is all for the better since these bikes have such a high bottom bracket. You save several pounds there.

When servicing the rear brake cabling be sure not to lose the cable housing ferrules, they're special.

How does your bike look? If you like the color then that might be reason enough to proceed. There's nothing like a Varsity, except for a Continental perhaps.
The Varsity also takes well to use of a swept-back upright bar, and some of them were sold that way. They handle great with such handlebars.