Yup, I do that too.
Yesterday's tweak involved a folding handlebar riser for my recumbent. It had a set screw to adjust it's angle. Unfortunately, I'd start off on a ride just fine, but the angle would change as I was riding and I'd eventually hit my knees on my thumbs or on the shifters. My solution was to install a lock nut on the adjusting screw. I rode 38 miles today and it worked fine.
My next project is finding a way to conveniently hold my camelback mouthpiece.