If you are more comfortable on the elite, you will be more likely to ride it. If you think right now that the 24 is not comfortable (and that is on short test rides) it is going to be even less comfortable on longer rides.
As Chaztrip said, fit should be your number one priority. That said, I don't think there are any major differences between the 24, sport, and elite frame, so if you fit well on one frame, you will fit well on the other.
Ockey53 makes a good point about what you want to do with your bike. When I got my road bike, I went in knowing what I wanted....a bike to allow me to do a century. What are your goals? How motivated are you about this goal. I wouldn't worry about you not being in good enough shape for your goals. When I got my road bike (Allez Comp) 2.5 months ago I was basically at zero --- I had just received clearence from my doc to start exercizing after heart surgery. Now, I'm up to doing 40 milers.
When I was test-riding bikes, I tried several bikes in the Allez series, but unfortunately, I didn't test the 24. The sport and elite are both fine bikes. I did notice a difference in "feel" of the bikes. The elite seemed to have a smoother ride, especially with shifting. The wheels are also better in the higher end bikes.
If you think the seat post is a big part of your added comfort on the elite compared to the 24, why not ask your LBS to swap the seatposts for a test ride? Then you can decide if that is the factor. If it is only the seatpost, maybe see if your LBS will upgrade the seatpost on the 24. From your comments, I am guessing you are looking at 04's. The sport has a CF seatpost, but not the one with Pave insert.
If you can do the swap, please let us know the difference. It would be interesting to see a direct comparison between AL and CF posts on the same bike with the same rider.
If the bike is in stock at Specialized, it won't take too long to get it in. My bike took about a week for the LBS to get it in and get it set up.
The Sequoia has been mentioned as an option. The price difference between the Sequoia and Allez series is modest. I think the lowest end Sequoia is about the same price as the sport. But, the sport has a better equipment group. The Sequioa does have the Pave seatpost though.
Good luck with your decision. Whatever way you go, if you make the decision based on good research and your needs, you'll have a great ride. Whichever you choose, you'll get approval from other riders.