Can you leave a lock/chain at both pool and work? And leave the pool locker lock attached to the lock/chain?
I've packed a microfiber camping towel when I was commuting. They dry quickly and pack small, about regular bath size towel. A swim chamois can get you from dripping wet to damp.

I wonder if fresh vegs could ride in a 2nd water bottle topped off with ice water. They would need to be cut up. Carrots, celery, cauli, broc that will be OK with a wet bumpy ride.

I'm sympathetic toward the OP's desire to swim and baggage needed.
I carry fins, pull-buoy, kickboard, goggles, hand-paddles as well as shower sandals, swim cap, swimsuit, towel, minimal toiletries.
Years ago I picked up a very nasty foot infection once from wet deck or shower stalls, I no longer go barefoot. The plastic pool decking is very uncomfortable to walk on barefoot anyhow.
I use a mesh laundry bag for the wet stuff and swim backpack for the dry stuff.
I would be hard-pressed to carry all that on a bicycle commute to work.