I adjusted my saddle last night and that made a big difference. Couldn't ride to work today because of fog, but hopefully will tomorrow. If it still feels uncomfortable I'm going to put my non pursuit bars back on, and go from there.
I really don't think the frame itself is too big. I'm just under 6'4" and it's a 61cm frame. I have another 61cm road bike that seems to fit me fine. If anything I think it's an issue of the stem being too long.