Don't let the rest of the monkeys get you down!
When I started cycling, I just bought cycling shorts ('cause I need the padding!) and a helmet (for obvious reasons). I rode in baggy t-shirts or sweatshirts to hide the fact I was overweight -- which is why I was riding in the first place.
I didn't spring for the shoes & pedals for quite a while, because I wanted to get used to the bike before I attempted clipless pedals.
As I got more comfortable, I added clipless pedals, which required the shoes. As I rode more and more and started going on longer rides, I realized I needed to carry things like food and energy gels with me. By then, I'd dropped a lot of weight so then -- and only then! -- did I get a jersey.
As far as the socks, I just wear white athletic socks...
Anyway, my advice is to wear whatever you want to. Are you out there for yourself, or to impress the monkeys?