Forget about the rear fork and tire for a moment.

Does the brake bridge hole, seat tube, and steering head tube "line up" visually when viewed from the rear?

If yes, then your rear tire and/or rear fork is out of alignment (or you have a slightly bent rear axle).

If not, then you may have a bent frame.