After installing a rack and trunk bag on my Centurion Ironman (previous pic below), I have noticed something interesting that I honestly didn't think I would notice, until I actually got on the bike today for my commute.
Well, I noticed all the extra weight (loaded with lunch/clothes, etc) causing the whole bike to feel much more unstable. The swaying back and forth when standing and pedalling initially scared the crud out of me. I thought something was seriously wrong. I noticed that the handling characteristics of the bike have completely changed, compared to when I hauled everything in my backpack on prior commutes.

Has anyone noticed that drastic of a change? Is it normal? How long did it take to get used to all the extra weight throwing off the stability like it does?

Trunk bags destabilize by default, grab a set of panniers instead. They will make your bike more stable than without. A trunkbag is used to supplement panniers not the other way around.

I'm guessing (hoping) you have a frame mounted rack and not a seat post mounted rack.
A trunk bag put all the weight up higher and the bike will sway and be harder to control.
Panniers will put the weight much lower on the bike so you'll have a lower center of gravity and easier control of the bike.

i've been riding with a seat post mounted trunk (on mtb) for a few months and only noticed a problem when i hadn't tightened it down all the way, which allowed the bag to swing side to side when i turned. once i got that taken care of i hardly even notice it's there.

You will adapt and in a couple days it won't be an issue....

The best place to mount weight on a bike is as low and centered as possible. Ideally some bag company should design a main triangle above the bottom bracket pack - but it would have to have some internal tabs to keep it thin and could only carry items that were about 2.5 inches thick or less. Your trunk mount is about the worst place to mount anything. You have mounted the weight high and back. The front end is now unstable and steering will be difficult. The best place for a pack is mounted in the main triangle tubes - but these packs are small and slim - usually mounting below the top tube and to the front downtube. 2nd best place and for panniers the best place is balanced, low in front - since they can go over the axles in front that is the lowest possible center of gravity and should be balanced by back panniers or a oversize seatbag. 3rd best is just low fronts. 4th is a seatbag- it may be high but the weight is kept near the main down tube - just back of the center of the bike. 5th is high panniers in back. 6th high panniers in front - fast braking could result in an endo - tumble bike and all. Finally the worst - trunk mount - high and far back. Consider what gives the lowest center of gravity and pick something more practical. I personally like small low front panniers balanced by a large seatbag with my heavy bike light battery in the seatbag near the seat tube - this combo actually makes for a bike that is more aerodynamic then a racer with no racks or packs - since most of the turbulence comes out of the wheels, whirlwinds centering on the hubs and a whirlwind of feet, crank and pedals that upset the smooth flow of air over the bike and rider.

My frame has no eyelets to bolt a regular rack to, which is why I am using a seatpost rack. I suppose I could try and rig something up to make a regular rack work, and just mount the rack to the rear skewer, and bolt/strap to the seat stays somehow. A regular rack would lower the trunk bag by about 5".

Some people notice this more than others. I notice that kind of thing right away too. You get used to it. Make it as tight as possible. Or even bungee it tight to the seat rails, that way it does not move as much.

The TUBUS "Fly" rack is only 350 grams. You can attach it to the seat stays with P clamps. I think it's a lot lighter than any seatpost rack.

edit post: I stand corrected 325 grams.

https://www.lickbike.com/productpage.asp?PART_NUM_SUB='3590-00'

and it's got a pretty steep steering tube, which would make it somewhat less stable with a load. A messenger bag would give you more ability to adjust. I used to commute on a road bike, carrying a messenger bag, and it felt pretty stable, since I could just move my body to adjust when I needed to.