I'd be surprised if you really notice a whole lot of difference between a bike with a small trunk bag and a bike without it. Ultimately you have to choose between carrying stuff and dealing with the extra weight/handling issues/sweaty back. When you're commuting to work it's probably not an option to leave your shirt and trousers at home but maybe you could leave them in the office and only deal with the extra weight at the beginning and end of the week?

I must admit I notice very little difference between a bike with just the pannier rack and a bike with a pannier on the side of it, unless the pannier is heavily loaded. Even then the main difference I notice is I often use a slightly lower gear when climbing. If you're riding to work I don't imagine your primary focus is on speed.

It's probably worth having a small wedge bag for things like your bike tools and a separate bag for your clothes. If nothing else it keeps greasy tools away from your nice clean shirt.