You were doing what most would to stay visible, above and beyond what's mandated by law, so the onus is on the driver to see you.
However.
Motorists, already not the best in the world where it comes to cyclist safety... or their own, get worse when it rains. Don't know why, but people take bigger chances in the rain than I think they would otherwise. Just a personal observation on my end, not to be confused with real data.
But clearly your cow-orker was concerned enough for your safety to say something. Sure, take him/her to task for his/her comment--you're absolutely right that the responsibility is on them--but for everyone like that, there's plenty of drivers who might voice the same thing if they knew you. Meaning--there's probably plenty of other drivers who also don't see you with your current visibility setup.
Get more visible with more lights. You shouldn't have to, but that cow-orker is the universe telling you that you are not noticeable enough with your current setup for your riding style.
Maybe put a bright light or SOLAS grade reflective material on the back of your helmet...
