Do cycling jerseys and cycling socks really make a difference?

I have riding shorts, but I do not have the jerseys with the pockets on the back. I have a saddle bag with all the essentials (CO2, Tube, Tire Levers, Gu), and when I need a Gu or a Shot Block, I stop and take a break since I am not super hardcore yet. Is a jersey just for easy storage, fashion, a sign of affluence, or all?

Currently, I wear running shirts and socks since they are cheaper and wick away sweat just as cycling gear would.

Are cycling socks really that different from running socks?

Since I am a noob.... I would like to know what dedicated cyclists think about people that do not have the jerseys, but still ride the roads.

Not that negative opinons or impressions would affect my clothing choices, I am just curious.