Probably need to try it to see if you like it. I don't think there is anything inherently bad about it. I swam in high school (season was during winter months) and regularly for exercise in college (all this was a thousand years ago) and never felt there were any problems (no colds or flu, etc).

But later in life I returned to a swimming regimen only to find it irritated my back (a certain arched back position that aggravated my facet problems).