Depends on how much weight and how far your going. There is an argument that back packs will make you top heavy thus more prone to accidents in emergency situations...but I don't put much stock in that.

If your currently using a backpack and have experienced no problems with an aching back then why switch to panniers? Work clothes are light in weight, I don't see an issue to continue using the backpack.