I'm only concerned that certain common, everyday obstacles 'terrify' you when you venture out into traffic. You need to be able to ride competently and confidently near debris, grates, gutters and even curbs. You've already sighted the lack of drver awareness. (Unfortunately, this is a problem in the great majority of U.S. cities.) And sooner or later one of these motorists is going to force you towards the side, or even off the road, and without even the uninformed verbal advice the aforementioned cyclist/motorist took the time to offer. So you should be practiced in riding through the tough stuff. It might save your life. Better to run over some debris or bunny hop a grate or pothole than get hit by a brain dead cager.
Although I take the lane when necessary, usually I try to ride as far to the right as practical and let traffic have it's way when lane width allows it. And I run 38cc wide, semi-knoby comfort tires for when I find myself running less than ideal pavement.
DanO