I suppose the answer depends on what you want to spend, and what the alternative is. I can think of five possibilities so i will list them, and if you have another idea then tell the internet what you were planning to do so we can yell at you. (joking, or for some people maybe not, but I am not planning on it)

1. Wear a baggy T-shirt during the swim.
2. Put on some kind of something in T1
3. Wear a cycling jersey during the swim
4. Just go shirtless the whole time, or sports bra if you are a girl, and assuming the tri allows it.
5. Buy a tri top and wear it the whole time.

Options 1 and 2 are frankly terrible ideas so don't try either.
Option 3 would be fine, may slow you down a bit.
Option 4 depends on how many little kids you make cry at the beach, or referees depending on the race.
Option 5 is probably the way to go, but involves a bit of money, and may make you go a bit faster.

And as for your concern about how much a top will slow you down, do you mean a tri top or a regular top?
Tri tops are designed for swimming, They should not slow you down at all, and higher end ones should make you faster, but just by a bit.