Right now it would seem that you are lacking more in skills than fitness. To work on skills:
Solo Stuff:
Ride at the very top of the track. Everyone is comfortable in the sprinter's lane. Few beginners are comfortable against the boards. Spend like 10 laps at a time right up against the boards. It feels different up there. Get used to it.
If the track is clear, weave around the track, up and down, left and right all at various speeds. In and out of the saddle. This will train your body to know what the track feels like.
Group Stuff:
Warmup riding 2-abreast next to each other (above the blue stayer's line, of course) while chatting. This will help you get comfortable being close to someone.
Follow-the-leader. Have the more experienced person be the leader. That person just leads the other newbs around like a mamma duck, weaving up and down the track (assuming it's clear, of course).
Pacelines. Lots of them. This will help you hold a straight line and hold on to a wheel. Maybe do some long ones 20-30 laps to work on getting in a rhythm. 1/2 lap pulls. This will get you moving around other people and the track at the same time. Also do some race-pace pacelines, too. Maybe 3-5 laps at 90%. Go sit down and rest for 15 minutes, then do more. Repeat till everyone is tired.
Do some 80-90% efforts two-by-two, like in a match sprint when someone is trying to pass. One guy in the sprinter's lane, the other guy right next to him above the red line. This will help you get comfortable being close to someone at high speeds. Don't try to pass each other, that defeats the purpose. Both have to make sure that they stay side-by-side.
I hope that helps.