To answer the question of performance, I doubt you'll notice much difference in a solo ride unless you repeat the same ride a bunch of times and calculate the average time with/without the Lycra. In a first race, you'll probably get dropped no matter what you ware. I'm not saying you aren't a strong rider, but staying with the pack will have more to do with learning how to position yourself in the pack, predict surges, get on the right wheels. Once you've been dropped, if you continue racing, you'll look to eliminate every disadvantage you can to avoid being dropped again and show up to your next race in the Lycra.

There is a pretty fast local group ride that I do where there is one guy who always shows up in basketball shorts and he keeps up OK for most of the ride. There is a hill followed by a sprint at the end, however, that tends to separate the more serious riders and racers from the recreational types. I've never seen him with the front group at the end. I'm not saying that the shorts are keeping him from being up in the front, but if you are serious about the sport, you'll eliminate disadvantages. I would say that baggy shorts are more of a symptom, rather than a cause of slowness.