When speaking of traditional steel frames, there are several approaches.

The simplest is the old "straddle the top tube and look for an inch of crotch clearance". This has fallen out of any favor it ever had, but for recreational cyclists it can work just fine.

Research from the late 70s/early 80s developed into a few formulas that are still useful. One is that your actual inseam (measured precisely rather than assumed via pants size) multiplied by .65 gives you your frame size center-to-center. So by the formula, you should be riding a 58 to 59 cm (23 inch) frame. That calculation is for racers, though, and it was/is common for recreational riders and tourists to go a little bigger, to get their handlebars a little higher.

These days, everything is "compact" with sloping top tubes, and everything is much, much smaller. This is doubtless why the shop worker was shocked at your 24" frame and thought you should be riding a tiny little 21" - the assumption these days seems to be that pro racers ride the smallest possible bikes, so everyone else should too. I think a 21" frame is much, much too small for you.

IMO if you are comfortable on a 24" frame then there's no reason to go smaller. The only possible concern you should have is crotch clearance: if you like to jump off the saddle and land with both feet flat on the ground, you should ensure that your naughty bits won't collide with the top tube. If you never jump off the saddle and land with both feet on the ground, then you don't have to worry about crotch clearance.

If I were you I would not go smaller than 23" with a traditional steel frame, and would probably stick with the 24". As a tourist and distance rider, using steel frames, I ride a 24.5" frame - and I am 6'1" with a slightly smaller inseam than you. I just make sure to never jump off the saddle...