I was inspecting the drivetrain and noticed that bolts holding the pulleys were loose.

Do I tighten them all the way or is there a risk of overtightening? Also, is it advisable to put threadlocker on them?

The bolts holding the rear derailleur pulleys are "shoulder bolts" so they stop at the right depth. Tighten them fully. Thread locker is applied by the manufacturer for the initial assembly but I've never needed it when I've removed the pulleys for cleaning and then reinstalled them. Just be sure the bolts are tight.

You can add a threadlocker if you like , it wouldn't hurt anything plus it keep the bolts from rusting in place .

I beg to differ with Hillriders, otherwise usually spot on, advise. The bolts are not what keeps the pulleys from freely rotating. The pulleys have a bushing bearing (most often) which has a sleeve running through the pulley. On either side of this sleeve, and the pulley body, are two dust covers. The sleeve is slightly wider then the pulley body so with even very high bolt torque the sleeve just is trapped within each dust cover and the pulley is free to rotate on this sleeve. The Shimano SIS upper pulley has a slight bit more of this end play. So the bolts only need to be tight enough to not come loose and no more. Andy.