Definitely get pedals and shoes.

The reason people are passing you is probably because they are more fit than you.

If someone in sandals passed me it would certainly make my head turn.