i tried manual labor, decided it was not for me (though i still work in a type of construction). I never did hay, which is often considered one of the most backbreaking work one can do (around my area anyway). But I did a fair bit of logging, mostly setting choker and wielding a chainsaw all day, and also ditch digging... i think ditch digging is worse. spend 8 hours just digging a trench.. blaeach backbreaking and boring.
I will tell you the worst job though: Weeding a cranberry bog. you can only do it by hand, and its not all that difficult--- but Sisyphus type of situation. You can spend a week bent over pulling weeds, every day, 8 hours a day, and look up and see that you have cleared like 20% of the bog and the first 10% now has new weeds.