Ride your bike if you want. I'm another female who lives (nearly) car free. I'm 5'3" and I live in a not that great area. The only times I've been hassled have been by the local college kids. Even then it's just stupid crap from kids who apparently think they are still in HS/were never taught any better by their parents.

Funny enough, the 'scary' people leave me alone.

Generally if they ask about a car they just want to be sure you aren't going to have trouble getting to work on time.