1. Is the bike too small? Can't tell you based on what you've given us. I can tell you that I'm 5'8"/31.6" inseam/135 lbs and I ride a 52cm Felt. I ride with the 13cm of aerobar pad drop from a 72cm seat height. To compare things for you my Felt has a stack of 50cm and reach of 40.5cm; the Cervelo P3sl has a stack of 47cm and a reach of 50.5cm. So that means they have essentially the same top tube length, but the Cervelo is 3cm lower. So if I were to get a Cervelo I would fit on it, but would need some spacers, or a tall aerobar. So in a long winded response, maybe it fits. A person who is 5'8" would likely fit on a 51cm P3sl but we're not all the same.
2. Is the price reasonable? Well, if it's in good condition like you say, most likely yes. The p3sl is about as aerodynamic as the P2 carbon which costs almsot three times what this bike costs. Then again, it comes with 650c wheels which are not as desirable as 700c. 650c has it's problems, mostly not being able to swap wheels between road bike and tri bikes and it's difficult to find good tires and tubes(which can have a big effect on speed). Although, 650c is good because you can get race wheels for next to nothing if you buy second hand.
3. Is it worth riding/buying? Thats up to you.
4. Will there be a difference? Yes the position and frame will be much more aerodynamic on the Cervelo if it fits you well. Typically going from a road position to a tri position will save you around 2-3 minutes in a 40k tt. A good aerodynamic frame will save you another minute in a 40k tt. 4 pounds even on going up Alp d'Huez is not even close to those kinds of savings, not to mention you get half of that on the way down from the better aerodynamics.
In case you haven't seen it, here is the old P3sl webpage:
Lets put it this way, if you want to keep racing and you want to be competitive, a triathlon bike helps. If it's something you are going to do once or twice a year for fun, I don't know it'd be worth it to me, but thats just me. The p3sl is a bad ass bike in my opinion. The geometry is very low, but I think some of the "taller" aerobars are fairly nice so just choose your front end appropriately. 650c wheels are also something to keep in mind, shorter people tend to like them and race wheels second hand are cheap but be smart about tire and tube choice.