I weigh the same and I'd guess the lowest I'd go with tires of that size would be around 40/45psi front/back, below that I suspect the handling would become squirrelly especially when climbing out of the saddle. But of course that's just a guess, since as Andy_K mentioned there are several variables at play here, what I suggest is that you make your decision based on sight/feel - I like to see my tires bulge out just slightly when I'm riding on a hard surface in my primary riding position, personally.
As for the harsh ride, I would guess that's the tires themselves. I haven't ridden these myself, but the specs available online show those tires have a low TPI count in the casing, and enough rubber for two tires applied in a very lumpy pattern. They're cheap for a reason.