A backpack will work but eventually, (particularly come summer), you'll appreciate putting your load on a rack instead of on your back. You're shoulders will also thank you a million times over.

I highly suggest looking into a rack and a set of panniers. To get an idea of the size of panniers which are measured in cubic inches, visit Jansport's website. Their basic backpack is rated at 1900 cubic inches. I have medium-sized panniers which are rated at 1830 cubic inches and I can easily fit a pair of jeans, a shirt, and a pair of shoes in 1 pannier. I got a good deal at my local bike shop (commonly referred to around these parts a an LBS) on a set of Axiom Kootenay panniers. My point is that you don't need a particularly large bag or set of bags to transport a change of clothes.

I wouldn't consider a rack and panniers to be adding "a bunch of bulky crap" to your bike. I've got a rack and panniers on my road bike and I can take the bags off faster than you think and the rack is only 4 allen bolts and I'm done. However, I'll admit I rarely take off the rack and having the bags are nice in case I end up needing the room mid-ride.

Not sure how you feel about spandex roadie gear but as far as cycling clothing goes, a pair of padded cycling shorts and a jersey makes all the difference in warm weather. I started with just swimming trunks and a cotton undershirt and it was miserable.

Keep up the commuting and stay safe. Sorry for the novel too.