Are you sure that your bike is properly fitted to your body type and riding style. You might start with trying to get you seat and handlebar stem at the same height. This will take some strain off your back, arms, wrist and neck. If you have a sore rear, that is very common. It only takes about a week of riding to get the body used to the new muscle usage.
You can check out Rivbike.com or sheldonbrown.com for some good information of bike fitting.