I share your fears.
Speed limit on one of the roads where I live is 45, and cars regularly go past 55. The other road near where I live speed limit is 35, and cars will still go 50+
Ultimately I guess it is about gaining confidence and just going for it, and maybe not riding alone..
I tried going the other day, but when a lifted truck blew past me in like a couple seconds at a close pass I got on the side walk and turned right around haha.