I've got a 20 and a 24 also. I'm 38, 6'0" and weigh 185. I ride trails and my local bmx track just for fun. I also enjoy riding these bikes on the boardwalk and around my neighborhood. The 24 is a lot more stable. I get less fatigue riding the 24 for an afternoon than I do on the 20. It's also a more fun bike to ride. Feels substantial under me.

Worth mentioning is that most 24s come factory set up so that the bars are the exact same height as bars on a 20. What they sell as "cruiser bars" typically have 4 inches or so of rise. Not more. It was not until I put some wide 8 inch rise bars on my cruiser and opened up the cockpit that the bike really became fun and super comfortable. Now it's a very different bike than my 20.

I ride my 20 with the seat slammed and my 24 with about 8 inches of post. I can actually pedal while in the saddle on my 24.

One more thing: buy a 20 inch bike and it's yours until you give it away or throw it away. 24 inch cruisers seem to do much better on the resale market.